|See inline comments in bold
I'm seeing a pair of event id 529 failure audits logged exactly every 8 hours (to the second) on my primary DC (2003), but I cannot figure out what is causing it (i.e. what or who is trying to logon).
The 'user name' identified by the 529 event is a domain admin (user) account. It is 'logon type:3' so it's trying to connect to a network share or IIS. The 'logon process' is "advapi" (Windows advanced API?).
'workstation name' is the name of the PDC this event was logged on. 'caller user name' is the "machine account" for the PDC (i.e. the same computer name with $ at the end). I was thinking 'workstation name' was the machine that logged the event (the target) and 'caller name' was the "source" machine, but 'source network address' in this case is the IP of a (backup) DC (also 2003). So, I guess I'm not understanding what these fields are referring to.
Workstation name is the name of the client computer. Computers can talk to themselves so sometimes workstation name will be the same as the computer where the event is logged. Source IP is the address of the client computer. DCs talk to each other so don't be suprised with source IPs of other DCs and servers - not just workstations. There's really no such thing as PDC in Win2000+ and definitely not BDCs. Don't know why workstation name would indicate one computer while source IP would indicate a different computer. Are you sure about that?
'caller process id' points to lsass.exe--assuming this process id is referring to the PDC in this case. I've observed no process with that id on on the machine identified by the source network address (the backup DC). Is 'caller process id' the process id on the target machine that the source user/machine is using? or the process id on the source machine? FYI: the 'caller process id' is always the same ("544").
Caller user name is usually not useful on event ID 529 nor is caller process id. But since you bring it up caller process ID in any event identifies the PID (same as in TaskManager and event ID 592) of the process on the local compuer (the one that logged the event) that is handling the logon request. Process IDs in security log events always identify processes on the local computer that can be correlated to event 592 - never process ids from some other computer on the network.
Two "Failure Audit" Event ID 680 events occur at the exact same time as the 529 events, every time they are logged. These events show the 'logon account' is the same as the 'user name' in the 529 events. The 'source workstation' is the same as the 'workstation name in the 529 events. The error code is 0xC000006A. I understand this means the user is successfully authenticated via NTLM instead of Kerberos, but I still can't figure out what is triggering these event log entries.
No 0xC000006A does not mean successful. 680 means NTLM attempted instead of Kerberos but can be successful or failed. 0xC000006A means username is correct but password is wrong http://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=680. sounds like you have a domain controller trying to talk to another domain controller with outdated authentication information. is replication working correctly between them?
Can you tell me where else to look?
your sample events do no correspond to what you described above
SAMPLE EVENT ID: 529
reason:unknown user name or bad password
caller user name:sundata$
caller logon id0x0,0x3e7)
caller process id:544
source network address:10.129.10.27
source port:9405 [THE PORTS APPEAR TO BE RANDOM, ALTHOUGH EACH PAIR OF EVENTS USES PORTS 2 APPART: e.g. 9405 & 9407, 56748 & 56750, etc.]
SAMPLE EVENT ID: 680
Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon account: admin_vapor
Source Workstation: VAPORDATA
Error Code: 0xC000006A