AOS crashes on max session Id (65535)

(1) Share

Report

Posted on by Francesco

514

The admins of my current assignment routinely monitor the highest session id and when it comes near the 60k mark they reboot the AOS so that the numbering starts from scratch. Rather inconvenient since this happens during operation hours.

The reason for this is that apparently the lights go out when the session id hits 65k: AOS crashes.

So, if I interpret SP CREATEUSERSESSIONS correctly, then it does appear to increment session ID's until it hits the magical number of 65535 (FFFF), after that it no longer returns a session ID, but returns -3 which I assume is an error-code.

All fine and dandy, but if this was the case then I surely would have encountered it before and at least half the community here would show up on Microsoft's parking lot with torches and pitch forks. So I'm sure I'm missing something here (please don't say SQL skills!) and my interpretation of the SP falls short.

Now the FFFF-doom scenario only occurs when

if (select count(*) from SYSCLIENTSESSIONS where SESSIONID IN (@first)) > 0
1. @first is the lowest session ID that is greater than @maxclientId and not the @masterid
@sessionid = -1 or @licenseType = 0

I guess I could write these parameters to a temp table, but at the moment I only have access to the PRD environment and there are limits to my level of confidence.

So my questions:

1. Does anyone know what the parameters @maxclientId and @licenseType represent?

2. Does anyone know why the AOS would crash upon session ID -3?

3. WTF? Why does it not reuse available session ID's? (two questions, but please only answer the 2nd one)

Categories:

Administration and Setup

Development / Customization / SDK

All responses (18)

Answers (0)

Georg 5 on at

Like (0)

Report
RE: AOS crashes on max session Id (65535)

Hi Francesco

Thank you for sharing this information. It lead us to do some more digging into this infamous sysClientSessions. Looks like the normal client will reuse SessionIDs, but type 5 will not.

.

My findings until now:

The connect from SharePoint (ClientType 5 - Web User), will not reuse SessionID with status 0 IF they are in between some with Status 1. I will then find the highest SessionID with status 1, and then reuse the next SessionID (or add a new if it was the highest).

I did a lot of connect/disconnect. And WebUser got ever increasing SessionIDs. Most of them change to status 0 very fast, but one of the highest SessionID stays in status 1. So the next connection go higher and ect.

Then I recycle the application pool (AX will disconnect). All (ClientType 5 - Web User) change to status 0, and I connect with a much lower SessionID at the next test connect.
Suggested answer

guk1964 10,877 on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Look at online users (System Administrator > Users > Online Users), where you may see a user with status “Ending – Blocked”. This is because AX locked the user session when there was a problem to complete processing a transaction - maybe due to time or a deadlock.

On the User Online screen see which userID and which session has status as Ending –Block

Go to the SQL server, look into SYSCLIENTSESSIONS table to see all user sessions.

SELECT TOP (200) SESSIONID, SERVERID, VERSION, LOGINDATETIME, LOGINDATETIMETZID, STATUS, USERID, SID, USERLANGUAGE, HELPLANGUAGE,

CLIENTTYPE, SESSIONTYPE, CLIENTCOMPUTER, DATAPARTITION, RECVERSION, RECID

FROM SYSCLIENTSESSIONS

WHERE (USERID = ‘Admin’)

Delete the line with the USERID and SESSIONID of that user

You may find many sessions with both inactive and blocked status.

These inactive and block session are often due to crashing of AX clients or just forgetting to log off.

These sessions may not create any problem.

However, when youy start the AOS service with many blocked or inactive sessions then, it may take too much time for the system to change status from Starting to Started. One reason for this is that AX will try to query and locate any Active session in SysClientSessions.

To reduce that time remove all session inactive from SysClientSessions table In “SysClientSessions” Status with:

Status 0 (Inactive),

Status 2 (Ending Waiting for AOS),

Status 3 (Ending –Block).

1. Stop AOS service.

2. Open SQL Server Management Studio

3. Open a new query on Dynamics Ax Database.

4. Run the following Query

where sessions with status 0 (Inactive), Status 2 (Ending Waiting for AOS), 3 (Ending –Block).

delete from SysClientSessions where status in(0,2,3)

5. Restart AOS.
Francesco 514 on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Hi Magic, thanks for chiming in. However, I fail to see how this could be a blocking issue.
guk1964 10,877 on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Try using SQL Activity Monitor to detect when extended blocking is occurring.

In SQL Server Management Studio, connect to your SQL instance, right click your instance in the object explorer and select ‘Activity Monitor’.

Once activity monitor is open, click the processes bar to view the SQL processes. It may take a few seconds for anything to load. The default refresh interval is 10 seconds and this is changed by right click the white space in the top right and selecting the refresh interval you desire.

When you a filter the processes, pause the refresh if necessary.

Finding Stuck Processes

Once the processes load, look at:

• the ‘Wait Type’ column,

• ‘Blocked By’ column,

• ‘Head Blocker’ columns.

The Session Id’s with ‘LCK’ wait types are the processes that are stuck and are waiting for another process to release a lock on a resource.

The Session Id with a 1 in the ‘Head Blocker’ column is the process that is holding a lock and preventing another process from accessing the same resource. Right click the Head Blocker and select ‘View Details’ to get an idea of the process that is running. The statement shown the current statement being executed and is not always the statement that is holding the lock.

Once you know the SQL Session Id (SPID) identify the user/process associated to the SPID.

In AX, go to Administration -> Common forms -> Online Users and use the SPID column on the form to gather a lot of useful information. Identify the AOS instance, the computer the SPID originated on, the status, session type, and login date/time.

Use this information to help identify the process causing the blocking.

With multiple AOS’s, you will only see the SPID’s for the AOS to which your current AX client session is connected. Use the ‘Host Name’ column in SQL Activity Monitor to identify on what AOS is the blocking SPID.
Francesco 514 on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

No, sorry. Bad observation. It looks like the counter for the "SharePoint sessions" was reset and it started numbering from 3 again early this morning. SysAdmins assure me there has been no rebooting of any kind (or any server) and there is no event on the sharepoint server that matches the start of the renumbering.

So to redefine the problem as correctly as possible:

A secondary application (most likely SharePoint) creates AX client sessions and calls CREATEUSERSESSIONS with parameter @maxClientId set to an ever increasing value, which over time leads to reaching the largest value that can be used to address the memory location (65535 or [tag:FFFF]) and results in the inevitable demise of the AOS service.

Since I don't have access to the SharePoint server, there is not a whole lot I can do. I could modify the stored procedure, but....let's not and say we did.

If I did have access I wouldn't know what to do other than to snoop around without a clear target. Since I have never-ever-ever heard of this before, I can't imagine it being anything other than screwy setup, but how? why? or where?....no clue.
André Arnaud de Cal... 285,583 Super User on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Hi Francesco,

When you open only the AX client, there are indeed no AX Anywhere sessions created. There will be only sessions when you use a handheld.

You mentioned that you are also using role centers. What contents is used within these role centers?
Francesco 514 on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Thanks, but as mentioned I doubt that AX Anywhere is the culprit. For starters, I'm not using any handheld :)

SharePoint is my prime target, or I should say...was. Today everything appears to be hunky dory. ALL sessions created today neatly reuse the available slots. Just shoot me!
André Arnaud de Cal... 285,583 Super User on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Hi Francesco,

I just recently joined To-Increase and I'm focusing mainly on Dynamics 365 add-on applications. I will ask internally if there are similar support cases known in relation with our product and AX 2012 max sessions.
Francesco 514 on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Okay, the plot thickens...

Today I had a chance to reboot the server and monitoring my own account I logged on and off three times.

This clearly illustrates the problem. Opening an AX client results in 6 sessions (logon 1 only shows 5, but this is because another user already consumed one of my inactive ones by the time I took the snapshot). Two are associated with the RD (I assume this is my actual AX client). One comes from the AOS (my server side process). Then three client sessions from the SharePoint server, which incidentally also runs AX Anywhere (Handheld terminal server by André's employer) and a webshop API. I assume that SharePoint is the only one creating sessions and that these have to do with my role center. However, I have the sysadmin role center and it is completely empty.

The three "SharePoint sessions" do not re-use available slots, but add additional records in sysclient. As can be expected they all are of client type 'server user'.

I've done a little "debugging" by catching the SP variables in a custom table.

As it turns out, the values for the "bad" sessions are:

@clienttype = 5 (server session)

@sessiontype = 0

@licensetype = 0

@sessionId = -1 (if a slot is found then @max_val is inserted)

@startId = 3

The problem comes however from @maxClientId, a parameter being fed to the SP. This one is set to the highest session ID instead of to the highest session ID with status = 0.
André Arnaud de Cal... 285,583 Super User on at

Like (0)

Report

RE: AOS crashes on max session Id (65535)

Hi Francesco,

Unfortunately, I don't know details of this environment. Are there ISV/partner solutions or customizations related to the webshop and warehouse terminals? Then you can probably contact the vendor(s) if they heard about similar issues.