NFS problems after OS updates - can't access directories
by Detlev Habicht
Hello,
i am using IPA, NFS-services and CIFS on several server running Scientific Linux 7.
This morning a lot of IPA-Updates arrived for Scientific Linux.
My clients are running Fedora 25.
Now i have a big problem:
- I am using automount and NFS4 and on my clients i see the directories i can mount.
But when i am trying to „cd“ to a directory, i see a message like can’t find file or directory.
- Replication don’t work.
- Authentication is still working, but maybe this is done by caching.
- SMB/CIFS is also still working.
- I can destroy my user ticket and i can get a new one with kinit.
I am still no expert, so i want to know, what i can do to find the error.
How can i search the error? Where i find logs for this NFS mount problem?
Thanx for any help!
Greetings
Detlev
P.S.:
---
My updates from today (only with „ipa“ in the name) on the IPA-Server:
Aug 22 08:11:14 Updated: ipa-common.noarch 4.5.0-21.sl7
Aug 22 08:11:58 Updated: ipa-client-common.noarch 4.5.0-21.sl7
Aug 22 08:12:23 Updated: libipa_hbac.x86_64 1.15.2-50.el7
Aug 22 08:12:25 Updated: python-libipa_hbac.x86_64 1.15.2-50.el7
Aug 22 08:12:50 Updated: python2-ipalib.noarch 4.5.0-21.sl7
Aug 22 08:12:50 Updated: python2-ipaclient.noarch 4.5.0-21.sl7
Aug 22 08:12:51 Updated: ipa-server-common.noarch 4.5.0-21.sl7
Aug 22 08:12:52 Updated: python2-ipaserver.noarch 4.5.0-21.sl7
Aug 22 08:12:55 Updated: sssd-ipa.x86_64 1.15.2-50.el7
Aug 22 08:12:57 Installed: ipa-client.x86_64 4.5.0-21.sl7
Aug 22 08:13:34 Updated: ipa-server.x86_64 4.5.0-21.sl7
Aug 22 08:14:37 Updated: ipa-server-dns.noarch 4.5.0-21.sl7
Aug 22 08:14:50 Updated: ipa-python-compat.noarch 4.5.0-21.sl7
Aug 22 08:15:51 Erased: ipa-admintools
---
The same with sssd on the IPA-Server:
Aug 22 08:11:14 Updated: libsss_idmap.x86_64 1.15.2-50.el7
Aug 22 08:11:54 Updated: python-sssdconfig.noarch 1.15.2-50.el7
Aug 22 08:11:55 Installed: libsss_certmap.x86_64 1.15.2-50.el7
Aug 22 08:12:24 Updated: libsss_nss_idmap.x86_64 1.15.2-50.el7
Aug 22 08:12:24 Updated: libsss_autofs.x86_64 1.15.2-50.el7
Aug 22 08:12:25 Updated: sssd-client.x86_64 1.15.2-50.el7
Aug 22 08:12:49 Updated: python-sss-murmur.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: libsss_sudo.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: sssd-common.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: sssd-krb5-common.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: sssd-common-pac.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: sssd-ipa.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: sssd-ad.x86_64 1.15.2-50.el7
Aug 22 08:12:55 Updated: sssd-ldap.x86_64 1.15.2-50.el7
Aug 22 08:12:56 Updated: sssd-krb5.x86_64 1.15.2-50.el7
Aug 22 08:12:56 Updated: sssd-proxy.x86_64 1.15.2-50.el7
Aug 22 08:12:56 Updated: sssd.x86_64 1.15.2-50.el7
Aug 22 08:12:57 Installed: sssd-dbus.x86_64 1.15.2-50.el7
---
A message from my replica problem:
Replication to sipa.ims.intern last operation 1970-01-01 00:00:00 Status: Error (18) Replication error acquiring replica: Incremental update transient error. Backing off, will retry update later. (transient error).
Well, i see the date. But why i have this date in the message?
--
Detlev | Institut fuer Mikroelektronische Systeme
Habicht | D-30167 Hannover +49 511 76219662 habicht(a)ims.uni-hannover.de
--------+-------- Handy +49 172 5415752 ---------------------------
6 years, 8 months
Samba update can't read NT Hash
by Randy Morgan
Yesterday we updated our fileserver to bring it up to the newest
kernel. At the same time it update the ipa-client and samba. After the
update was finished our ability to access the shared resources on the
fileserver disappeared. After some very careful troubleshooting we have
been able to narrow it down to a problem with Samba, but we have been
unable to find where in the configuration the problem is. I am
including several logs, config files, etc with this, we need this
restored ASAP, but can't seem to isolate the issue.
logs:
Log.192.168.105.237
[2017/08/17 07:59:38.684827,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[homes]"
[2017/08/17 07:59:38.684939,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[stockroom]"
[2017/08/17 07:59:38.685049,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[inorgstk]"
[2017/08/17 07:59:38.685144,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[netlogon]"
[2017/08/17 07:59:38.685211,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[deptchair]"
[2017/08/17 07:59:38.685333,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[deptfinance]"
[2017/08/17 07:59:38.685448,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[facultysearch]"
[2017/08/17 07:59:38.685523,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[research]"
[2017/08/17 07:59:38.685610,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[working]"
[2017/08/17 07:59:38.685713,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[csradmin]"
[2017/08/17 07:59:38.685802,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[install]"
[2017/08/17 07:59:38.685933,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[coffice]"
[2017/08/17 07:59:38.686097,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[grants]"
[2017/08/17 07:59:38.686202,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[deptoffice]"
[2017/08/17 07:59:38.686330,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[gradadmissions]"
[2017/08/17 07:59:38.686411,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[mainoffice]"
[2017/08/17 07:59:38.686525,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[busoffice]"
[2017/08/17 07:59:38.686607,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[entropy]"
[2017/08/17 07:59:38.686718,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[ltarch]"
[2017/08/17 07:59:38.686807,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[netlogon-n175]"
[2017/08/17 07:59:38.686963,3] ../source3/param/loadparm.c:1592(lp_add_ipc)
adding IPC service
[2017/08/17 07:59:38.687257,2] ../source3/lib/interface.c:345(add_interface)
added interface eth0 ip=192.168.105.99 bcast=192.168.105.99
netmask=255.255.255.255
[2017/08/17 07:59:38.687362,3] ../source3/smbd/oplock.c:1322(init_oplocks)
init_oplocks: initializing messages.
[2017/08/17 07:59:38.687511,3] ../source3/smbd/process.c:1957(process_smb)
Transaction 0 of length 159 (0 toread)
[2017/08/17 07:59:38.687557,3]
../source3/smbd/process.c:1538(switch_message)
switch message SMBnegprot (pid 22349) conn 0x0
[2017/08/17 07:59:38.688383,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [PC NETWORK PROGRAM 1.0]
[2017/08/17 07:59:38.688408,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [LANMAN1.0]
[2017/08/17 07:59:38.688418,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [Windows for Workgroups 3.1a]
[2017/08/17 07:59:38.688423,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [LM1.2X002]
[2017/08/17 07:59:38.688429,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [LANMAN2.1]
[2017/08/17 07:59:38.688434,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [NT LM 0.12]
[2017/08/17 07:59:38.688439,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [SMB 2.002]
[2017/08/17 07:59:38.688444,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [SMB 2.???]
[2017/08/17 07:59:38.688548,3]
../source3/smbd/smb2_negprot.c:290(smbd_smb2_request_process_negprot)
Selected protocol SMB2_FF
[2017/08/17 07:59:38.689133,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_spnego' registered
[2017/08/17 07:59:38.689159,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_krb5' registered
[2017/08/17 07:59:38.689171,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_krb5_sasl' registered
[2017/08/17 07:59:38.689181,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'spnego' registered
[2017/08/17 07:59:38.689191,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'schannel' registered
[2017/08/17 07:59:38.689203,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'naclrpc_as_system' registered
[2017/08/17 07:59:38.689221,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'sasl-EXTERNAL' registered
[2017/08/17 07:59:38.689249,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'ntlmssp' registered
[2017/08/17 07:59:38.689265,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'ntlmssp_resume_ccache' registered
[2017/08/17 07:59:38.689283,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'http_basic' registered
[2017/08/17 07:59:38.689334,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'http_ntlm' registered
[2017/08/17 07:59:38.690888,3] ../source3/smbd/negprot.c:730(reply_negprot)
Selected protocol SMB 2.???
[2017/08/17 07:59:38.691535,3]
../source3/smbd/smb2_negprot.c:290(smbd_smb2_request_process_negprot)
Selected protocol SMB3_11
[2017/08/17 07:59:46.501902,3]
../source3/smbd/smb2_server.c:3097(smbd_smb2_request_error_ex)
smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1]
status[NT_STATUS_LOGON_FAILURE] || at ../source3/smbd/smb2_sesssetup.c:134
[2017/08/17 07:59:46.503583,3]
../source3/smbd/server_exit.c:246(exit_server_common)
Server exit (NT_STATUS_CONNECTION_RESET)
[2017/08/17 07:59:59.462220,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[homes]"
[2017/08/17 07:59:59.462329,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[stockroom]"
[2017/08/17 07:59:59.462456,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[inorgstk]"
[2017/08/17 07:59:59.462530,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[netlogon]"
[2017/08/17 07:59:59.462577,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[deptchair]"
[2017/08/17 07:59:59.462630,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[deptfinance]"
[2017/08/17 07:59:59.462711,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[facultysearch]"
[2017/08/17 07:59:59.462761,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[research]"
[2017/08/17 07:59:59.462839,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[working]"
[2017/08/17 07:59:59.462896,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[csradmin]"
[2017/08/17 07:59:59.462962,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[install]"
[2017/08/17 07:59:59.463032,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[coffice]"
[2017/08/17 07:59:59.463098,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[grants]"
[2017/08/17 07:59:59.463161, 2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[deptoffice]"
[2017/08/17 07:59:59.463238,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[gradadmissions]"
[2017/08/17 07:59:59.463289,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[mainoffice]"
[2017/08/17 07:59:59.463355,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[busoffice]"
[2017/08/17 07:59:59.463418,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[entropy]"
[2017/08/17 07:59:59.463478,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[ltarch]"
[2017/08/17 07:59:59.463540,2]
../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[netlogon-n175]"
[2017/08/17 07:59:59.463623,3] ../source3/param/loadparm.c:1592(lp_add_ipc)
adding IPC service
[2017/08/17 07:59:59.463828,2] ../source3/lib/interface.c:345(add_interface)
added interface eth0 ip=192.168.105.99 bcast=192.168.105.99
netmask=255.255.255.255
[2017/08/17 07:59:59.463902,3] ../source3/smbd/oplock.c:1322(init_oplocks)
init_oplocks: initializing messages.
[2017/08/17 07:59:59.464003,3] ../source3/smbd/process.c:1957(process_smb)
Transaction 0 of length 159 (0 toread)
[2017/08/17 07:59:59.464038,3]
../source3/smbd/process.c:1538(switch_message)
switch message SMBnegprot (pid 22371) conn 0x0
[2017/08/17 07:59:59.464721,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [PC NETWORK PROGRAM 1.0]
[2017/08/17 07:59:59.464747,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [LANMAN1.0]
[2017/08/17 07:59:59.464760,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [Windows for Workgroups 3.1a]
[2017/08/17 07:59:59.464786,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [LM1.2X002]
[2017/08/17 07:59:59.464795,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [LANMAN2.1]
[2017/08/17 07:59:59.464817,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [NT LM 0.12]
[2017/08/17 07:59:59.464876,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [SMB 2.002]
[2017/08/17 07:59:59.464893,3] ../source3/smbd/negprot.c:603(reply_negprot)
Requested protocol [SMB 2.???]
[2017/08/17 07:59:59.465013,3]
../source3/smbd/smb2_negprot.c:290(smbd_smb2_request_process_negprot)
Selected protocol SMB2_FF
[2017/08/17 07:59:59.465821,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_spnego' registered
[2017/08/17 07:59:59.465869,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_krb5' registered
[2017/08/17 07:59:59.465879,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_krb5_sasl' registered
[2017/08/17 07:59:59.465888,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'spnego' registered
[2017/08/17 07:59:59.465910,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'schannel' registered
[2017/08/17 07:59:59.465930,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'naclrpc_as_system' registered
[2017/08/17 07:59:59.465941,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'sasl-EXTERNAL' registered
[2017/08/17 07:59:59.465949,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'ntlmssp' registered
[2017/08/17 07:59:59.465957,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'ntlmssp_resume_ccache' registered
[2017/08/17 07:59:59.465972,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'http_basic' registered
[2017/08/17 07:59:59.465982,3]
../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'http_ntlm' registered
[2017/08/17 07:59:59.467516,3] ../source3/smbd/negprot.c:730(reply_negprot)
Selected protocol SMB 2.???
[2017/08/17 07:59:59.468111,3]
../source3/smbd/smb2_negprot.c:290(smbd_smb2_request_process_negprot)
Selected protocol SMB3_11
[2017/08/17 08:00:06.151513,3]
../source3/smbd/smb2_server.c:3097(smbd_smb2_request_error_ex)
smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1]
status[NT_STATUS_LOGON_FAILURE] || at ../source3/smbd/smb2_sesssetup.c:134
[2017/08/17 08:00:06.153192,3]
../source3/smbd/server_exit.c:246(exit_server_common)
Server exit (NT_STATUS_CONNECTION_RESET)
Log.smbd
[2017/08/17 02:27:26.578214,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 18077 -- ignoring
[2017/08/17 02:42:26.580707,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 19278 -- ignoring
[2017/08/17 02:57:26.585133,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 20546 -- ignoring
[2017/08/17 03:12:26.588487,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 21704 -- ignoring
[2017/08/17 03:27:26.592306,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 22935 -- ignoring
[2017/08/17 03:42:26.594330,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 2162 -- ignoring
[2017/08/17 03:57:26.598090,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 4040 -- ignoring
[2017/08/17 04:12:26.602245,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 5113 -- ignoring
[2017/08/17 04:27:26.606161,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 6269 -- ignoring
[2017/08/17 04:42:26.610297,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 7382 -- ignoring
[2017/08/17 04:57:26.612547,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 8497 -- ignoring
[2017/08/17 05:12:26.615685,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 9614 -- ignoring
[2017/08/17 05:27:26.618609,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 11133 -- ignoring
[2017/08/17 05:42:26.621232,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 12292 -- ignoring
[2017/08/17 05:57:26.625906,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 13379 -- ignoring
[2017/08/17 06:12:26.628955,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 14452 -- ignoring
[2017/08/17 06:27:26.630512,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 15536 -- ignoring
[2017/08/17 06:42:26.634709,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 16600 -- ignoring
[2017/08/17 06:57:26.638292,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 17706 -- ignoring
[2017/08/17 07:12:26.642297,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 18790 -- ignoring
[2017/08/17 07:27:26.644817,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 19881 -- ignoring
[2017/08/17 07:42:26.649127,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 21035 -- ignoring
[2017/08/17 07:57:26.653799,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 22175 -- ignoring
[2017/08/17 08:12:26.656684,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 23303 -- ignoring
[2017/08/17 08:27:26.660355,2]
../source3/smbd/server.c:794(remove_child_pid)
Could not find child 24397 -- ignoring
smb.conf (global portion only)
[global]
#debug level = 2
debug level = 3
workgroup = RESEARCH
realm = CHEM.BYU.EDU
netbios name = CHEM
kerberos method = dedicated keytab
dedicated keytab file = FILE:/etc/samba/samba.keytab
create krb5 conf = no
log file = /var/log/samba/log.%m
security = user
passdb backend = ipasam:ldaps://ipa1.chem.byu.edu
ldapsam:trusted = yes
ldap ssl = no
ldap suffix = dc=chem,dc=byu,dc=edu
ldap user suffix = cn=users,cn=accounts
ldap group suffix = cn=groups,cn=accounts
load printers = no
cups options = raw
printcap name = /dev/null
running an ldapsearch yields correct results:
[root@fs-ipa-rhel7 samba]# ldapsearch -Y GSSAPI uid=randym ipaNTHash
SASL/GSSAPI authentication started
SASL username: randym(a)CHEM.BYU.EDU
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <dc=chem,dc=byu,dc=edu> (default) with scope subtree
# filter: uid=randym
# requesting: ipaNTHash
#
# randym, users, compat, chem.byu.edu
dn: uid=randym,cn=users,cn=compat,dc=chem,dc=byu,dc=edu
# randym, users, compat, chem.byu.edu
dn: uid=randym,cn=users,cn=compat,dc=chem,dc=byu,dc=edu
# randym, users, accounts, chem.byu.edu
dn: uid=randym,cn=users,cn=accounts,dc=chem,dc=byu,dc=edu
# search result
search: 4
result: 0 Success
# numResponses: 4
# numEntries: 3
I also tried changing my password to see if it was just an NT hash
issue, but that had not effect either.
Any help would be greatly appreciated.
Randy
--
Randy Morgan
CSR
Department of Chemistry and Biochemistry
Brigham Young University
801-422-4100
6 years, 8 months
Kvno error on validating one-way trust: "kvno: Decrypt integrity check failed while getting credentials"
by bogusmaster＠o2.pl
Hi All,
I am setting up a one-way trust from FreeIPA server to AD domain with a pre-shared key.
It seems that it was set up successfully but I cannot verify the Kerberos configuration when I follow the steps described here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/....
Although I successfuly kinit with a username from AD domain and obtain a ticket:
klist
Ticket cache: KEYRING:persistent:0:0
Default principal: testuser(a)DOMAIN.COM
Valid starting Expires Service principal
08/22/2017 09:47:41 08/22/2017 19:47:41 krbtgt/DOMAIN.COM(a)DOMAIN.COM
renew until 08/23/2017 09:47:34
I am not able to request service tickets for a service within IdM domain:
[root@idm1 ~]# KRB5_TRACE=/dev/stdout kvno -S host idm1.ipa.domain.com
[16119] 1503409696.153004: Getting credentials testuser(a)DOMAIN.COM -> host/idm1.ipa.domain.com(a)IPA.DOMAIN.COM using ccache KEYRING:persistent:0:0
[16119] 1503409696.153288: Retrieving testuser(a)DOMAIN.COM -> host/idm1.ipa.domain.com(a)IPA.DOMAIN.COM from KEYRING:persistent:0:0 with result: -1765328243/Matching credential not found
[16119] 1503409696.153422: Retrieving testuser(a)DOMAIN.COM -> krbtgt/IPA.DOMAIN.COM(a)IPA.DOMAIN.COM from KEYRING:persistent:0:0 with result: -1765328243/Matching credential not found
[16119] 1503409696.153520: Retrieving testuser(a)DOMAIN.COM -> krbtgt/DOMAIN.COM(a)DOMAIN.COM from KEYRING:persistent:0:0 with result: 0/Success
[16119] 1503409696.153536: Starting with TGT for client realm: testuser(a)DOMAIN.COM -> krbtgt/DOMAIN.COM(a)DOMAIN.COM
[16119] 1503409696.153600: Retrieving testuser(a)DOMAIN.COM -> krbtgt/IPA.DOMAIN.COM(a)IPA.DOMAIN.COM from KEYRING:persistent:0:0 with result: -1765328243/Matching credential not found
[16119] 1503409696.153609: Requesting TGT krbtgt/IPA.DOMAIN.COM(a)DOMAIN.COM using TGT krbtgt/DOMAIN.COM(a)DOMAIN.COM
[16119] 1503409696.153663: Generated subkey for TGS request: aes256-cts/A13D
[16119] 1503409696.153718: etypes requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[16119] 1503409696.153875: Encoding request body and padata into FAST request
[16119] 1503409696.153942: Sending request (1851 bytes) to DOMAIN.COM
[16119] 1503409696.154236: Resolving hostname domain.com
[16119] 1503409696.290796: Initiating TCP connection to stream 10.10.10.10:88
[16119] 1503409696.398086: Sending TCP request to stream 10.10.10.10:88
[16119] 1503409696.836098: Received answer (1551 bytes) from stream 10.10.10.10:88
[16119] 1503409696.836121: Terminating TCP connection to stream 10.10.10.10:88
[16119] 1503409696.836212: Response was from master KDC
[16119] 1503409696.836258: Decoding FAST response
[16119] 1503409696.836423: TGS reply is for testuser(a)DOMAIN.COM -> krbtgt/ipa.domain.com(a)DOMAIN.COM with session key aes256-cts/C0B1
[16119] 1503409696.836454: TGS request result: 0/Success
[16119] 1503409696.836461: Received TGT for offpath realm ipa.domain.com
[16119] 1503409696.836468: Requesting TGT krbtgt/IPA.DOMAIN.COM(a)ipa.domain.com using TGT krbtgt/ipa.domain.com(a)DOMAIN.COM
[16119] 1503409696.836486: Generated subkey for TGS request: aes256-cts/743D
[16119] 1503409696.836523: etypes requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[16119] 1503409696.836579: Encoding request body and padata into FAST request
[16119] 1503409696.836648: Sending request (1854 bytes) to ipa.domain.com
[16119] 1503409696.904352: Resolving hostname idm1.ipa.domain.com.
[16119] 1503409696.938147: Sending initial UDP request to dgram 10.10.10.11:88
[16119] 1503409696.943465: Received answer (146 bytes) from dgram 10.10.10.11:88
[16119] 1503409696.977047: Response was from master KDC
[16119] 1503409696.977102: TGS request result: -1765328353/Decrypt integrity check failed
kvno: Decrypt integrity check failed while getting credentials for host/idm1.ipa.domain.com(a)IPA.DOMAIN.COM
Can you please advise me on how to resolve this issue?
Bart
6 years, 8 months
FIPA 2FA OTP+PASSWORD
by saidireddy ranabothu
Hello all, I have enabled password+OTP authentication for a user and able
to sync tokens and SSH. While ssh to server using FIPA credentials it's
asking authentication in two steps as First Factor and Second Factor . But
i just want to give it in a single line password ,Can any one suggest how
to do it as a single line password?
6 years, 8 months
HTTPD does not start when NSS enabled
by Julian Gethmann
Hallo,
Unfortunately I don't know when this problem occurred first, but it may
have occurred after an update.
The httpd does not start and aborts with the error
[:info] [pid 15383] Using nickname Server-Cert.
[...] [:error] [pid 15383] Certificate not found: 'Server-Cert'
when I want to start FreeIPA via "systemctl start ipa" or "ipactl start"
or "systemctl start httpd"
If I turn the NSSEngine off it starts of cause.
In contrast to this message "ipa-getcert list -d /etc/httpd/alias/ -n
Server-Cert" does find a certificate, if I get the output [1] right.
ipa-server-upgrade also complained about the HTTPD not starting, so I
tried to run it with "NSSEnigne off" which made the upgrade run through,
but did not fix the problem with the HTTPd
My System:
(After running "ipa-server-upgrade" with out any failures, but with
"NSSEngine off")
# ipa --version
VERSION: 4.4.4, API_VERSION: 2.215
on Fedora Server 26
CA-Server at main IPA-Server (which is failing now)
/etc/hosts has got the fqdn in the first line
and DNS is not installed.
[1] # ipa-getcert list -d /etc/httpd/alias/ -n Server-Cert
Number of certificates and requests being tracked: 8.
Request ID '20160718102648':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa_server.example.com,O=EXAMPLE.COM
expires: 2018-03-24 14:33:00 CET
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Many thanks in advance,
Julian
6 years, 8 months
Issues after adding Let's encrypt certificate
by Sarhan Aissi
Hello,
I am using FreeIPA 4.3.1 with Ubuntu Server 16.04 and i tried to add my
Let's encrypt certificate using the "freeipa-letsencrypt" script (I replaced Fedora/RHEL commands with ubuntu equivalents):
https://github.com/freeipa/freeipa-letsencrypt
After restarting freeipa i cannot add new members to the ipa server or
connect to the REST api. The error message is related to the certificate
and " (SEC_ERROR_UNKNOWN_ISSUER) Peer's Certificate issuer is not
recognized.".
How can add the Let's encrypt issuer to the trust list or at undo what
i have done (i don't have any backup for /etc/apache2/nssdb) ?
Thank you
6 years, 8 months
annoying messages systemd: pam_sss(systemd-user:account): Access denied for user (Permission denied)
by Kees Bakker
Hi,
This is on Ubuntu 16.04 systems configured as FreeIPA clients. Logging in through ssh
is successful. But in /var/log/auth.log there are annoying messages like this:
Aug 18 15:38:02 client1 systemd: pam_sss(systemd-user:account): Access denied for user joe: 6 (Permission denied)
This only happens for non-local users (i.e. users that are known in FreeIPA).
Is this perhaps a configuration fault in Ubuntu? Or else, is there a way to suppress
this message? I'd like to keep my auth.log as clean as possible.
--
Kees Bakker
6 years, 8 months
Re: web UI - login failed after updates on server
by Stefan Uygur
Hi everyone,
I have an IPA instance installed and working for the last 6 months but after the patching yesterday the Web UI login has stopped to work.
To be clear the IPA server is fully functional at the backend, the problem is when I try to login via web UI I get the following error:
Login failed due to an unknown reason.
The server is a Red Hat Enterprise Linux Server release 7.4 (Maipo) with the IPA VERSION: 4.5.0, API_VERSION: 2.228
Furthermore, this is what I get from apache error logs while trying to login using web UI:
[Thu Aug 17 11:58:40.727456 2017] [:error] [pid 20879] ipa: INFO: *** PROCESS START ***
[Thu Aug 17 11:58:40.911349 2017] [:error] [pid 20878] ipa: INFO: *** PROCESS START ***
[Thu Aug 17 11:58:57.224594 2017] [auth_gssapi:error] [pid 20884] [client IPADDR:54323] NO AUTH DATA Client did not send any authentication headers, referer: https://aws-ipa1.firstderivatives.com/ipa/ui/
[Thu Aug 17 11:58:57.266259 2017] [auth_gssapi:error] [pid 20884] [client IPADDR:54323] GSS ERROR In Negotiate Auth: gss_accept_sec_context() failed: [An unsupported mechanism was requested (Unknown error)], referer: https://ipa1.example.com/ipa/ui/
/usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
[Thu Aug 17 11:59:03.637157 2017] [:error] [pid 20878] ipa: INFO: 404 Not Found: URL="/ipa/session/cookie", URL fragment "/session/cookie" does not have a handler
[Thu Aug 17 11:59:03.638346 2017] [:error] [pid 20879] ipa: INFO: 401 Unauthorized: No session cookie found
[Thu Aug 17 12:00:01.567042 2017] [:error] [pid 20882] SSL Library Error: -12195 Peer does not recognize and trust the CA that issued your certificate
[Thu Aug 17 12:00:01.617683 2017] [:error] [pid 21225] SSL Library Error: -12195 Peer does not recognize and trust the CA that issued your certificate
[Thu Aug 17 12:00:09.967173 2017] [auth_gssapi:error] [pid 20881] [client IPADDR:54377] NO AUTH DATA Client did not send any authentication headers, referer: https://aws-ipa1.firstderivatives.com/ipa/ui/
/usr/lib/python2.7/site-packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
[Thu Aug 17 12:00:17.495782 2017] [:error] [pid 20879] ipa: INFO: 404 Not Found: URL="/ipa/session/cookie", URL fragment "/session/cookie" does not have a handler
[Thu Aug 17 12:00:17.497067 2017] [:error] [pid 20878] ipa: INFO: 401 Unauthorized: No session cookie found
I know it is complaining about the new mod_gssapi but never seen this sort of problem before on IPA.
Any help would be greatly appreciated.
Stefan
____________________________________________________
Stefan Uygur | First Derivatives Ireland Ltd | +353 16307761 | suygur(a)firstderivatives.com<mailto:suygur@firstderivatives.com>
***********************************************************************************************
This email, its contents and any files attached are a confidential communication and are
intended only for the named addressees indicated in the message.
If you are not the named addressee or if you have received this email in error, you may not,
without the consent of First Derivatives, copy, use or rely on any information or attachments in any way.
Please notify the sender by return email and delete it from your email system.
Unless separately agreed, First Derivatives does not accept any responsibility for the accuracy or
completeness of the contents of this email or its attachments. Please note that any views,
opinion or advice contained in this communication are those of the sending individual
and not those of First Derivatives and First Derivatives shall have no liability whatsoever in relation to
this communication (or its content) unless separately agreed.
***********************************************************************************************
6 years, 8 months
Introducing FC26 into domanlevel 0 of Centos7 servers
by pgb205
so far we have pure domainlevel0 consisting of Centos7 servers. The plan is to add Fedora Server 26 which will initially also be at domanlevel0.
Are there any pitfalls that we should watch out for with these two different versions of OS?
thank you
6 years, 8 months
AD-Trust users not known
by Michael Gusek
Hi,
for testing i've installed an FreeIPA-Server with a trust to an
AD-Server. On IdM i can resolve AD-users with 'id username(a)example.com',
on IdM member client not.
AD-Domain is Server 2012R2 as 'example.com'
IdM is latest CentOS 7 with ipa-server-4.4.0-14.el7.centos.7.x86_64 as
'ipa.example.com'
IdM member client is latest CentOS 7 with
sssd-client-1.14.0-43.el7_3.18.x86_64
Here an example on an Centos 7 client:
ipa-member> id username(a)example.com
id: 'username(a)example.com': no such user
Logmessages, with log_level=10, shows:
ipa-member> tail -f /var/log/sssd/sssd_ipa.example.com.log | grep s2n
(Fri Aug 18 11:38:08 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_send] (0x0400): Executing extended operation
(Fri Aug 18 11:38:08 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 13
(Fri Aug 18 11:38:09 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_done] (0x0400): ldap_extended_operation result:
Success(0), (null).
(Fri Aug 18 11:38:09 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_send] (0x0400): Executing extended operation
(Fri Aug 18 11:38:09 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 14
(Fri Aug 18 11:38:09 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: No such
object(32), (null).
(Fri Aug 18 11:38:09 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_get_fqlist_next] (0x0040): s2n exop request failed.
(Fri Aug 18 11:38:09 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_get_fqlist_done] (0x0040): s2n get_fqlist request failed.
Running on IdM:
ipa-server> id username(a)example.com
uid=299801104(username) gid=299801104(username)
Gruppen=299801104(username),299800513(domänen-benutzer),299801109(mitarbeiter),556800008(ad_users)
Any help is welcome.
Michael
----- /etc/sssd.conf on ipa-member -----
[domain/ipa.example.com]
cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.example.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = ipa-server.ipa.example.com
chpass_provider = ipa
dyndns_update = True
ipa_server = _srv_, ipa-server.ipa.example.com
dyndns_iface = eth0
ldap_tls_cacert = /etc/ipa/ca.crt
debug_level = 10
[sssd]
debug_level = 10
services = nss, sudo, pam, ssh
domains = ipa.example.com
[nss]
debug_level = 10
homedir_substring = /home
[pam]
debug_level = 10
[sudo]
[autofs]
[ssh]
[pac]
debug_level = 10
[ifp]
----- /etc/sssd.conf on ipa-server -----
[domain/ipa.example.com]
cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = ipa.example.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = ipa-server.ipa.example.com
chpass_provider = ipa
ipa_server = ipa-server.ipa.example.com
chpass_provider = ipa
ipa_server_mode = True
ldap_tls_cacert = /etc/ipa/ca.crt
subdomain_homedir = /home/%u
shell_fallback = /bin/bash
debug_level = 10
[sssd]
services = nss, sudo, pam, ssh
domains = ipa.example.com
[nss]
memcache_timeout = 600
homedir_substring = /home
[pam]
[sudo]
[autofs]
[ssh]
[pac]
[ifp]
----- complete log messages for 'id username(a)example.com' on ipa-member
-----
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sysdb_search_user_by_upn] (0x0400): No entry with upn
[username(a)example.com] found.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending
request
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [dp_req_done]
(0x0400): DP Request [Account #5]: Request handler finished [0]: Erfolg
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [_dp_req_recv]
(0x0400): DP Request [Account #5]: Receiving request data.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_reply_list_success] (0x0400): DP Request [Account #5]: Finished.
Success.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_reply_std] (0x1000): DP Request [Account #5]: Returning
[Success]: 0,0,Success
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_table_value_destructor] (0x0400): Removing
[0:1:0x0001:1:1:U:ipa.example.com:name=username@example.com] from reply
table
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_destructor] (0x0400): DP Request [Account #5]: Request removed.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_destructor] (0x0400): Number of active DP request: 0
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_process_result] (0x2000): Trace: sh[0x7f14ec425550], connected[1],
ops[(nil)], ldap[0x7f14ec409710]
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_process_result] (0x2000): Trace: end of ldap_result list
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [sbus_dispatch]
(0x4000): dbus conn: 0x7f14ec428290
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [sbus_dispatch]
(0x4000): Dispatching.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sbus_message_handler] (0x2000): Received SBUS method
org.freedesktop.sssd.dataprovider.getAccountInfo on path
/org/freedesktop/sssd/dataprovider
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sbus_get_sender_id_send] (0x2000): Not a sysbus message, quit
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_get_account_info_handler] (0x0200): Got request for
[0x1][BE_REQ_USER][1][name=username(a)example.com]
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [dp_attach_req]
(0x0400): DP Request [Account #6]: New request. Flags [0x0001].
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [dp_attach_req]
(0x0400): Number of active DP request: 1
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_id_op_connect_step] (0x4000): reusing cached connection
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_send] (0x0400): Executing extended operation
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 12
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [sdap_op_add]
(0x2000): New operation 12 timeout 6
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_process_result] (0x2000): Trace: sh[0x7f14ec425550], connected[1],
ops[0x7f14ec40ca10], ldap[0x7f14ec409710]
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED]
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: No such
object(32), (null).
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_op_destructor] (0x2000): Operation 12 finished
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [sdap_id_op_done]
(0x4000): releasing operation connection
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [dp_req_done]
(0x0400): DP Request [Account #6]: Request handler finished [0]: Erfolg
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]] [_dp_req_recv]
(0x0400): DP Request [Account #6]: Receiving request data.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_reply_list_success] (0x0400): DP Request [Account #6]: Finished.
Success.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_reply_std] (0x1000): DP Request [Account #6]: Returning
[Success]: 0,0,Success
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_table_value_destructor] (0x0400): Removing
[0:1:0x0001:1:1:U:webtrekk.com:name=username@example.com] from reply table
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_destructor] (0x0400): DP Request [Account #6]: Request removed.
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[dp_req_destructor] (0x0400): Number of active DP request: 0
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_process_result] (0x2000): Trace: sh[0x7f14ec425550], connected[1],
ops[(nil)], ldap[0x7f14ec409710]
(Fri Aug 18 11:54:05 2017) [sssd[be[ipa.example.com]]]
[sdap_process_result] (0x2000): Trace: end of ldap_result list
--
6 years, 8 months
