Ok, we returned to work on the 4th Jan. Going through the event logs, there are huge amounts of information events for MSSQLSERVER.
The first error on the 4th though was:
Host HA-DB01: error while processing organization 0520837a-e995-471b-a8a3-80389b6c298b. Exception: System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server. (provider: Shared Memory Provider, error: 0 - No process is on the other end of the pipe.)
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.WriteSni()
at System.Data.SqlClient.TdsParserStateObject.ExecuteFlush()
at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch(String text, Int32 timeout, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.Crm.CrmDbConnection.InternalExecuteNonQuery(IDbCommand command)
at Microsoft.Crm.CrmDbConnection.ExecuteNonQuery(IDbCommand command, Boolean impersonate)
at Microsoft.Crm.CrmDbConnection.SetTransactionIsolationLevel(IsolationLevel il)
at Microsoft.Crm.CrmDbConnection.Open()
at Microsoft.Crm.Asynchronous.DataAccessBase.ExecuteSqlCommandAndProcessRecords(IDbCommand command, RecordProcessor recordProcessor)
at Microsoft.Crm.Workflow.WaitSubscriptionDataAccess.Notify(IDbCommand selectCommand, WaitNotificationService waitNotificationService)
at Microsoft.Crm.Workflow.WaitSubscriptionDataAccess.Notify(String hostId, WaitNotificationService waitNotificationService)
at Microsoft.Crm.Workflow.WorkflowHost.OnNotificationTimerEvent(Object sender, ElapsedEventArgs e)
The next one is:
CrmKeyGenerator encountered an error generating/monitoring key type System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server. (provider: Shared Memory Provider, error: 0 - No process is on the other end of the pipe.)
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.WriteSni()
at System.Data.SqlClient.TdsParserStateObject.ExecuteFlush()
at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch(String text, Int32 timeout, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.Crm.CrmDbConnection.InternalExecuteNonQuery(IDbCommand command)
at Microsoft.Crm.CrmDbConnection.ExecuteNonQuery(IDbCommand command, Boolean impersonate)
at Microsoft.Crm.CrmDbConnection.SetTransactionIsolationLevel(IsolationLevel il)
at Microsoft.Crm.CrmDbConnection.Open()
at Microsoft.Crm.SharedDatabase.DatabaseService.Retrieve(String tableName, String[] columns, PropertyBag[] conditions)
at Microsoft.Crm.SharedDatabase.DatabaseService.RetrieveById(String tableName, Object id, String[] columns)
at Microsoft.Crm.CrmKeySetting.LoadSettingsBag()
at Microsoft.Crm.CrmKeySetting.Initialize(Boolean loadSettingsFromConfigDB, Guid scaleGroupId, ConfigSku configSku)
at Microsoft.Crm.CrmKeySetting..ctor(CrmKeyType crmKeyType, Boolean loadSettingsFromConfigDB, Boolean useCachedValues, Guid scaleGroupId, ConfigSku configSku)
at Microsoft.Crm.CrmKeyGenerator.DoManageKeys(Guid scaleGroupId). Check event log to see if keys are being generated properly. KeySetting : %2. Error : %3