Update on Life Cycle Services, AX Update Installer

This is a short follow up on my last post.

LCS has been down for partners in Norway for a while, but it is available now (maybe a candidate for a later post titled “The downside of being dependent upon services in the Cloud”).

A short check today (October 31 2014) revealed a change in the mysterious “Updates until a given date” limitation and right now, this includes Updates until October 19 2014 as shown below:

Like people blogging about the different updates (builds) for AX, SQL Server and SharePoint (very useful information indeed and a life saver), this could also be a hot candidate for such information.

But the gap is not as big as it was last time and it seems like Microsoft has included most of the available updates for AX 2012 R3 in the current range.

Don't get me wrong - I think Life Cycle Services will be an important part of the future services around AX, and clearly something Customers (projects are initiated by Customers) and Partners (invited to participate) will benefit from in a long term relationship. Many thoughts and reflections could be tied to this, but I'll leave it for now...

Life Cycle Services, AX Update Installer

Microsoft is now investing heavily in Life Cycle Services (LCS) with monthly releases.

One of the most interesting features for Technical resources is AX Update Installer found under the Updates tile for AX 2012 R3 projects. This is the content after opening the tile:

After downloading the package, you have to extract the special build of AXUpdate.exe and then execute AXUpdate.exe to connect to the project in Life Cycle Services and get the magic started.

I will not go into details around this, since it is a quite straightforward and self-explaining process. One thing to mention is the requirement to have the System Diagnostics configured and at least have uploaded the Discovered Environment once. This uploads (push) Diagnostics information from your On Premise solution to Life Cycle Services including Metadata, which is important for several of the technical features in LCS like Cloud Power Support and Updates.

What I found compelling about the new AXUpdate approach was that it seemed to address the pain trying to keep up with the number of hotfixes released both for the application and for binaries. Therefore, I went through a test earlier in September to see how this really fitted with the description and my expectations. The expected benefit was to be able to produce solution specific update packages without the need to identify and download many individual fixes between Cumulative Updates (a collection of hotfixes and new features). This would be very efficient in a project where we are approaching GOLIVE on R3 (CU8 will be the first CU for R3, but no formal information about expected release date yet).

Everything played surprisingly well and I managed to update a default AX 2012 R3 solution (Model Store) without any modifications with approximately 500 application fixes and binaries for all server roles. The possibility to create and store an Update Package is another nice feature.

After finishing the Update process including the The Model Store has been modified Wizard, it was time for QA and I therefore opened Help > About from the AX Windows Client. As expected I now had many new Models in the SYP-layer (you get one SYP-model for each KB number), but when I checked the current Kernel Build number against the latest/current build number, I discovered that the updated binaries had a lower Build number than the most recent released Binary Build. This was indeed not the expected outcome…

After a couple of days I ran AXUpdate.exe again to check if any new hotfixes where available and the answer was zero (0) applicable updates. I then reviewed one of the first steps in the process after logging in to LCS called Select updates and this revealed the reason as shown below (performed October 06):

In the Combobox to Select a group of updates to evaluate, Most recent was explained as “Contains all updates until August 04, 2014”.

I have not checked after this, but Microsoft has for whatever reason defined a limitation on the set of available updates to include updates until a given date. If this was (is) intentionally it is hard to understand why since the power of AXUpdate.exe connected to LCS, is to address the pain of discovering available updates and downloading them periodically (or when needed) as a package.

I will check this again as soon as LCS is available (still inaccessible for me) to see what the date limit is right now – hopefully this was a onetime glitch and not an intended limitation.

Bottom line – do not be surprised if you do not get all available updates when using AXUpdateInstaller connected to LCS. I would like to add that this was a possible onetime glitch, but it could also explain why you do not end up with the updates you expect.

AX 2012 R3, Error running Cmdlet Test-AXReportServerConfiguration -The Network Path not Found

After successfully installing Reporting Extensions and modifying the Report Configuration, I wanted to test the configuration from the RS Server. The cmdlet test-axtestreportserverconfiguration is obviously the best candidate to do the job, but it threw an error stating "The Network Path was not found" and "Could not open the registry to read the EnableLUA key" like shown below (ran the AX Management Shell as Administrator):

I have experienced issues due to UAC (User Access Control), but after first trying to disable UAC completely (registry, EnableLUA key) on the RS Server, the error still was thrown.

I then noticed that the error actually was thrown when the cmdlet tried to read the EnableLUA key on the AOS Server (it's a two-phased test) and after checking for some references, I found an article on TechNet titled "Configure Remote Management in Server Manager" (Windows Server 2012 & R2) and the section "To configure MMC or other tool remote management over DCOM", seemed to point in the right direction.

After enabling the following rules (and Enabling EnableLUA again), the cmdlet worked as expected.

• COM+ Network Access (DCOM-In)*
• Remote Event Log Management (NP-In)
• Remote Event Log Management (RPC)
• Remote Event Log Management (RPC-EPMAP)

*It's probably the DCOM-In rule that's most important (I did not check combinations of the rules since I was working on building a Production solution for a customer).

Verification

Windows Firewall will have different rules enabled according to the customers policy and you should pay attention to the Firewall rules when experiencing issues during installation of AX Components or when a cmdlet throws errors. I have described several issues due to Windows Firewall in other posts related to installation issues and this was the first one related to PowerShell cmdlets.

AX 2012 R3 Data Import Export Framework (DIXF), Exception at Microsoft.Dynamics.AX.Framework.Tools.DMF.ServiceProxy.DmfEntityProxy.DoWork[T](Func`1 work)

A preliminary warning about a potential issue for Data Import Export Framework released for AX 2012 R3 based on experience from 

• Scenario 1: Single-Server install (AOS and SSDS + SSIS with all DIXF Components) 
• Scenario 2: Multi-Server install (2x AOS with DIXF AOS and Client Component, SSDS + SSIS with DIXF Framework Service Component)

In scenario 1, the Service Account for the AOS Service was utilized for the DIXF Framework Service while in scenario 2 a separate Integration User Account was utilized for the DIXF Framework Service.

Symptom:

In scenario 2 an exception was thrown when validating the working directory for DIXF (button in form DIXF Parameters) > Microsoft.Dynamics.AX.Framework.Tools.DMF.

ServiceProxy.DmfEntityProxy.DoWork[T](Func`1 work).

This worked fine in scenario 1.

Issue 1 - Local Security Group not created by the installer when installing the DIXF AOS Component:

When installing the DIXF Framework Service, a new Local Security Group called Microsoft Dynamics AX Data Import Export Framework Service Users was created and populated with the AD Account hosting the DIXF Framework Service. This Local Security Group was not created by the installer when installing the DIXF AOS Component on the AOS servers. After manually creating this Security Group on the AOS servers, populating it according to the instructions given in Install the Data import/export framework (AX 2012 R3) [AX 2012] and restarting the AOS servers (caching), the error still was thrown.

Issue 2 - Required membership in new Local Security Group:

The instructions on TechNet states that the DIXF Framework Service Account must be added to the Security Group on the server hosting the Framework Service, and similarly the AOS Service Account to be added to the Security Group on the servers hosting the AOS Service and the DIXF AOS Component. In scenario 2 I had to add both Service Accounts to the Security Group on all three servers followed by a restart of the AOS servers, to get the validation working without throwing the exception (issue solved)

It could of course be something related to the In-Place Upgrade procedure, but I doubt this since you have to uninstall all pre AX 2012 R3 Components before any AX 2012 R3 Components can be installed. My best practise includes a restart of each server after performing the uninstall and other clean up tasks, and I think the difference between the scenarios (single versus multi-server setup), points in the direction of the Security Groups not being created automatically by the installer and that all Accounts used by DIXF, must be added to each group.

AX 2012 R3 AOS Service Principal Name (SPN)

March 26 2015 - please also read my Update on this subject.

After working my way through both a new installation and upgrades from AX 2012 R2 (CU7), I find at least one glitch that 1) logs an Error in Event Log - Application when stopping and starting the AOS Service  and 2) that possibly increases the time to start the AOS Service.

I notice the following events beeing logged:

• Start of AOS Service > Object Server 01:  RPC error: Failed to register service principal name (SPN): '<GUID>/<server>.<FQDN>:<AOS RPC Port#>' 
• Stop of AOS Service > Object Server 01:  RPC error: Failed to unregister service principal name (SPN): <GUID>/<server>.<FQDN>:<AOS RPC Port#>' 

This seems to be related to utilizing Managed Service Accounts (an option when specifying the AOS Service Account) or not, where it seems like the AOS Service operates like it's always hosted by a Managed Service Account. In all installations I have used "Use the following account" and not the "Use a managed service account", but despite this deliberate choice, the messages are beeing logged.

Maybe it's difficult or even impossible to separate the different types of Service Accounts, but it should be possible to have this as a property for the Service (registry). The time it takes to start the AOS Service seems to increase in each release of AX 2012 due to the increased number of processes the AOS Service must handle and possible time consuming tasks like this, should indeed be avoided.

Bottom line - I'm supprised to see SPN errors beeing logged like this even when not utilizing a Managed Service Account (which requires a SPN and new requirements regarding integration to objecst in Active Directory).

AX 2012, AifMessageInspector::AfterReceiveRequest - Failed to Logon to Microsoft Dynamics AX

From what I can see, many people have stumbled upon (aka experienced) this beeing logged in the Event Log (Application) on servers hosting AOS instances. I also see this beeing logged especially on AOS servers hosting AOS instances handling Enterprise Portal and Enterprise Search load, but also on other AOS servers (more sporadic).

Some relevant examples:

• http://community.dynamics.com/ax/f/33/t/120096.aspx
• http://community.dynamics.com/ax/f/33/t/109144.aspx
• http://dynamicsuser.net/forums/p/72354/385352.aspx

  
  

In the last link above, Martin Dráb suggests to look at the Service Operations granted and this is the best suggestion I have seen so far. I have not verified this yet, but I would recommend to have a look at the Security Roles and especially the Service Operations granted to a specific Security Role. A good starting point is the articles Services, service operations, and service groups [AX 2012] and About role-based security in services and AIF [AX 2012] on TechNet.

My hypotese is that AX simply reports back Failed to Logon instead of a more informative description pointing to missing Privilegies or Duties.

All comments are welcomed to share some experience and reach a final solution for this rather hard to diagnose issue (no information about which Service except it's an AIF Document Service and what user is Failing to Logon to AX).

WCF Tracing could perhaps reveal the AIF Document Service and User (I have not looked at this yet), but Exception Logging will not catch this (the issue happens before the actual Message reaches the Queue in AX).

The only solution I would not recommend of the ones I have seen suggested, is to add the Business Connector Proxy User (System Account) as a User in AX. Why? This is a privileged user account in AX and the fact that it's covered by a specific form in AX, should tell you that this account should not be treated as a normal user in AX...

AX 2012 R2 CU6, Feature to reduce the effect of Parameter Sniffing

Working with many AX solutions over the years, I have seen Parameter Sniffing in many solutions and the effect of "bad Execution Plans" in SQL Server, can be rather "not so nice" to all end users.

Most of the solutions I see have many companies or Legal Enteties (LE) as they are refered to in AX 2012, but the distribution of the data in shared tables, are very commonly skewed where one company or LE typically allocates 80 to 9x percent of the rows while 8 - 10 companies/LE contribute to the rest of the rows in individual shared tables. Microsoft introduced PARTITION as a new way of isolating companies or LE in AX 2012 R2 and this led to another column in most tables contributing to the parameter sniffing issue. AX 2012 R2 is designed to have several partitions and many LE each representing a balanced data distribution (each partition and Company/LE having roughly the same number of rows), but in real life, this is very seldom what customers end up with.

I would say that Parameter Sniffing is not a phenomen (it's in fact a SQL Server by design behaviour), but it tend to hit AX solutions very hard when the design criteria is not met. The x-factor lies in the fact that Microsoft utilizes Parameterized Queries heavily to make sure every Query is executed "the same way" (also the main reason for recommending MAXDOP = 1) each time and to reduce the number of SQL Compiles and Recompiles (has a small CPU cost). But the downside is that the actual value for the parameter when the statement reaches SQL Server for the first time (or the SQL Server is experiencing mamory pressure with flushing of execution plans), must represent "an average" partions and/or company/LE. The x-factor can simply be caused by the the first user logging on to AX the morning (early bird) after the AOS instances and/or SQL Server has been restarted and representing a small company with regards to the number of rows in shared tables, can "trick" SQL Server in generating a execution plan where SQL Server find it most efficent (based on the cost) to perform a table or index scan. When an user representing the largest company (number of rows in the underlying tables) implicitly reuses the same execution plan, all sorts of bad performance reports starts popping up. Looking at the SQL Server you will probably notice constantly high CPU utilization and digging a little further, You will probably identify one or several statements consuming much CPU time (some of then probably also beeing utilized by API Cursors).

Microsoft introduced a new feature in AX 2012 R2 CU6 to overcome this vulnerability and it's a kernel feature that's turned off by default. The feature is well described in the article Overcoming parameter sniffing issue in Microsoft Dynamics AX 2012-R2 – CU6 written by the AX Performance Team.

I have been aware of this rather important change (results in values for the columns PARTITION and DATAAREAID beeing always sent to SQL Server as literals or actual values instead of parameterized values), but I tend to await activation until I actually see (have proofe of) that a solution is impacted by Parameter Sniffing. Last week I saw proof of this happining in a busy solution focused around Sales Order processing and after activating the changes in the table SYSGLOBALCONFIGURATION followed by a stop - start sequence of the AOS instances, we imediately saw SQL Server generating execution plans using literals for the columns described which again "calmed" the solution. At the same time the number of SQL Compiles and Recompiles increased somewhat, but this was expected as the number of Execution Plans for each Parameterized Statement will increase by the product of the number of Partitions and the number of individual companies/LE utilizing a spesific module in AX.

This is perhaps one of the most important changes Microsoft has introduced and I have not seen any other solution improving (trace flags, force literals keyword etc.) or near eliminating this issue until this feature was introduced. Everyone involved in implementing AX 2012 solutions should be aware of this and be able to utilize this feature when needed. I say AX 2012 since MS stated that this change was expected to be back ported to AX 2012 (I don't know the status), but it's confirmed in AX 2012 R2 CU6 and later binary builds. Customers still utilizing AX 4.0 or AX 2009, will probably never benefit from this feature.

Many factors play a part in this story such as SQL Server Statistics, Max Degree of Parallism and API Cursors, but this is the single most efficent method of helping SQL Server generating the most representative (balanced) Execution Plans for AX based on my experience.