Microsoft drops support for mstshash RDP cookies? (embrace and destroy policy?)

UPDATE: November 2015 — We are pretty sure Microsoft have quietly fixed this bug and not told anyone... But the story is quite fun so lets leave it here for a lesson in corporate stupidity.

Recently we have seen quite a few customer issues where using RDP cookies. So we decided to investigate this and find out why.

A couple of customer recently reported that when using RDP cookies, this sometimes resulted in multiple active sessions over several RDP servers as shown below (Notice the user Rob on both TS Servers).

Where do we start?

Well our first thought was to check if there were any issues with the load balancing application or its configuration. we found that everything there looked to be fine except that not every user seemed to be in the stick table and that testing the exact same environment with Windows XP or a Linux client we were unable to replicate the issue.

After some research we found the TechNet documentation on the x.224 Connection Request PDU and the following key text.

<13> Section 3.2.5.3.1: Microsoft RDP 5.1, 5.2, 6.0, 6.1, 7.0, 7.1, and 8.0 clients always include the cookie field in the X.224 Connection Request PDU if a nonempty username can be retrieved for the current user and the routingToken field is not present (the IDENTIFIER used in the cookie string is the login name of the user truncated to nine characters).

Ok so that makes sense, when making a connection request the client should send a cookie made from the users username so the loadbalancer knows who the user is. We fired up Wireshark and what do you know we found that the new client does not always send this cookie. it seems it will not send it or the incorrect cookie in the following conditions:

1. Starting the client and using the cached/saved credentials, then no hash is sent.
2. Mistyping the username and then correcting, mstsc still uses the old username as the hash and not the updated username.
3. Under certain conditions if a user entered the wrong password the cookie would default to domain/user and not the expected user@domain format that was entered in the username field.  This is an issue for users with domains over 9 characters long as the cookie is then allowed to be duplicated for many users or not match the previously used cookie.

Ok so what happens with the XP client?

Well this seems to follow the documentation and explains why that worked.

how do we fix this?
Well the TechNet Docs say this is how it works, yet the client does not do this...

Sounds like a bug so lets log it with Microsoft:

......First problem. How the hell do you log a bug? Nothing stands out from their site so instead we paid to log a support call with them (which is still ongoing 4 months later).

So where does this all lead?

Well so far Microsoft have stated this is and I quote (including spelling mistake) from them:

"The behavior that you are experiencing is a documentation bug and we are currently in the process of publishing a Knowledge Base article for it"

Err ok, so a Microsoft Product does not follow their own documentation and they say it is a Documentation bug? So their answer is if a product does not fit the design spec, just change the spec. I have now seen the new version of the documentation and it now says...

"An optional and variable-length ANSI character string terminated by a 0x0D0A two-byte sequence. This text string MUST be "Cookie: mstshash=IDENTIFIER", where IDENTIFIER is an ANSI character string (an example cookie string is shown in section 4.1.1). The length of the entire cookie string and CR+LF sequence is included in the X.224 Connection Request Length Indicator field. This field MUST NOT be present if the routingToken field is present."

So now its an optional item, and not always sent?

Well thats no good to anyone using RDP cookies!

They also say there is a perfectly valid workaround..... but failed to advise what it was, well after chasing for it we are told....

"This problem occurs only when the same MSTSC process is used to connect to a server with 2 different credentials. The scenario can easily workaround by starting a new MSTSC process before connecting to the server with different credentials."

Really?

Well to us this indicates an issue with the MSTSC client where it is unable to update its own cookies without having to restart the application, and after many emails back and forth we get the very latest response.

"Based on my internal discussions, we conclude that for load balancing terminal servers, using Session Directory is the recommended approach. This whitepaper talks more about the implementation http://download.microsoft.com/download/8/6/2/8624174c-8587-4a37-8722-00139613a5bc/TS_Session_Directory.doc. In the business impact statement that you had provided, I did see that you have mentioned that you cannot use Session Directory."

Well thats lovely for anyone with Windows 2003 Enterprise and above, what about users of 2003 Standard who do not have the Session Directory included? Hell their document even says this:

"Note: Terminal servers must be running Windows Server 2003 Enterprise Edition or Windows Server 2003 Datacenter Edition to participate in a Session Directory-enabled farm."

Well we are still waiting for an answer to this from Microsoft but it looks like the RDP cookie maybe being dropped by Microsoft and no fix in sight... Guess we will just have to wait and see what they say next, however we would love to hear from anyone else who is experiencing the same issues as us so we can advise Microsoft of other users who are being hampered by this issue.

Update...

Well it looks like Microsoft have indeed silently dropped support for mstshash cookies for load balancing as suspected.....

As detailed in the last post we had a call open with Microsoft and have just received the following response that confirms our suspicions that they have in fact dropped support for mstshash silently in favour of their Session Directory/Broker solutions.

I hope this email finds you well. I regret to state that, I’ve a negative news for you. Based on the triage of code with product group, they suggest you to devise some alternate way to achieve load balancing. According to them, the dependency you took on the optional cookie field is not recommended or supported.The protocol documentation does not specify what is passed in that field and the value can change based on the scenario. Unfortunately, there are no plans on updating the documentation as it does not call out what is being passed in this field and is already marked as optional. Please let me know if you wish to have a conference call with us on this matter. Thank you for your understanding and patience. I again, is truly regretful.

Unfortunately it seems they have decided to change the documentation to fit their product changes rather then fix their product to work as the original documentation detailed that has since been updated.

Oh well guess you win some and you lose some.