Weblogic Fails To Start With Error Unable To Obtain Lock

After killing the Weblogic server, the following errors were raised on restart:
<23-Jul-009 12:57:00 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. R
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
And then on a re-attempt:
<23-Jul-2009 12:58:06 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:44)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:107)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.Heap.open(Heap.java:166)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:85)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:350)
        at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:202)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:61)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:42)
        ... 4 more
Caused by: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.StoreDir.throwIOException(StoreDir.java:213)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:103)
        at weblogic.store.io.file.Heap.open(Heap.java:164)
        ... 9 more
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.open(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:51)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:138)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:142)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:95)
        ... 10 more

This problem sometimes occurs if you killed the bin/startWebLogic process via:
SOLUTION
There are two possible solutions:
remove the lock files and restart
kill any remaining server processes (if they exist) and restart
Solution1: Removing the lock files and restarting
1. Find -name "*.DAT" files in user_projects/domains/<your domain name> directory (as appropriate) and move or remove them, for example:
$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
NOTE: Gentle reminder that Unix is case sensitive. you do need to make sure you do NOT delete the file <CM HOME>/security/SerializedSystemIni.dat, it is the files ending with "*.DAT" not "*.dat"
Deleting .DAT files: with reference to Note 1332274.1. Please be aware of the implications of removing the .DAT files. "...There may be error messages for transactions that were already committed and finished during the WebLogic Server run. However, the transaction entries will still be available in the transaction persistent store file (.DAT file)..."
2. Remove lock files: "EmbeddedLDAP.lok" and "AdminServer.lok" (maybe some more). These files are usually under paths such as below
/opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok
opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok
3. You can use the following shell script to find out which process is listening to your port, and kill this via kill -TERM:
#!/bin/ksh
line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
if [ $# -eq 0 ]; then
   read ans?"Enter port you would like to know pid for: "
else
   ans=$1
fi
for f in $pids
do
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
   if [ $? -eq 0 ]; then
      echo $line
      echo "Port: $ans is being used by PID:\c"
      /usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
   fi
done
exit 0
4. Execute:
nohup ./startWebLogic.sh &
Solution2: kill any remaining server processes (if they exist) and restart
Using the example of a managed server
1. Find the PID of the managed server
ps -ef | grep ManagedServer
2. Kill the managed server process with a -KILL
kill -9 <pid of managed server>
3. Restart the Admin server and Managed servers
nohup ./startWebLogic.sh &
Note: the startWebLogic.sh script is usually run with nohup so the user can exit the shell leaving the process running.
For Windows based servers please see the following
1. The AdminServer.lok file default location is usually: C:\Oracle\Middleware\user_projects\domains\<domain_name>\servers\AdminServer\tmp\
2. Start startWebLogic.cmd
Note: If AdminServer.lok cannot be deleted:
i) end all java.exe processes in Task Manager on WebLogic server,
ii) delete AdminServer.lok file