New Blog - DynamicsAX Technical Journal

Please visit my new Blog for Dynamics AX (#MSDynAX) for my thoughts and experience with the New Dynamics AX

Life Cycle Services, AX Update Installer (axupdateinstaller.exe) Episode x

When applying CU10 to a Existing AX 2012 R3 solution, I once again stumbled upon the mysterious Date tied to Option "Most recent" as shown below.

I first applied CU10 and then, "Most recent" during week 2 2016 to catch up with hotfixes released after CU 10 (released 24 Nov 2015).

When I today once again checked if Microsoft had increased the "Until Date", it still was showing December 9 2015...

Dear Microsoft - why is this date still December 9 2015? And what's the reason for not updating this value centrally in LCS?

Dynamics AX

Just as a "ping" to my Readers - I'm still around. Busy times as usual and especially one AX 2012 R3 Project has kept me busy for some time. And when we just got the grips on AX 2012 R3, we have to prepare for a new version... This post gave me an opportunity to share some general information related to my work and some initial thoughts about Dynamics AX...


Besides working in the Project mentioned above, I have also worked with a customer running AX 2012 R2 CU7 on replacing older physical servers running Windows Server 2008 R2, with New Virtual Servers running under vSphere 5.x and Windows Server 2012 R2. The most interesting part of this delivery, is the fact that they are running on the Nutanix platform. Compared to a traditional vSphere implementation with classic servers (hosts) and a SAN, this is my first experience with Converged Infrastructure. Not at the Hypervisor or Hardware level, but with Virtual Machines specified by us (me) and provisioned by the Customer regarding to the specification. Without going into all details, Nutanix is a player within the Converged Infrastructure area where one of the main differences from the classic approach, is that the Storage Layer is hosted within the same Physical Unit (appliance) as the other key System Resources (CPU, Memory, networking etc.) and still presentad as a SAN when deploying multiple Nutanix Appliances. The goal is basically to reduce the distance between the Storage consumer and the Storage Devices compared to the classic approach where the data has to travel through a Host Bus Adapter, a Fiber Channel which again goes through a/several Fiber Switch(es) before arriving at the SAN Controller and finally the Physical Storage Device(s). Over simplified, but you probably see the difference. It will be interesting to see how AX performs on this platform.


In between this, I was also lucky enough to attend the AX 7 Technical Preview in Redmond. The content was valuable for most of the sessions I attended with BRK 101 Better together: How Microsoft Dynamics AX 7 leverages SQL Azure and SQL Server 2016 as the single most interesting one for me. A lot of stuff going on in this area after Microsoft decided to move the Dynamics Team into the Cloud and Enterprise (C+E) team. I recommend attending the Technical Conference in 2016 based on what I saw in the Preview Conference and the fact that Dynamics AX is a technical release.


I have looked a little bit at AX 7 from CTP4 to CTP7, but I will not conclude on anything before working more with what is now officially named Dynamics AX to be released in February 2016. The main reason is of course the fact that Dynamics AX represents an architectural change to leverage Azure Public Cloud as the only platform in the initial release (let's call it Dynamics AX R1) and the number of changes this implies especially for the Technical Roles. Seen from Our part of the world (North Europe) with the current regulations, the lack of an Azure Data Center in the Nordics, might imply a challenge to get customers to run AX in the Cloud. And I know that we are operating in a small market compared to the US, but the lack of a Nordic Azure Data Center, will probably limit the adoption of AX R1 in the Nordics. But Test, Demo and Development deployments in Azure is still the only way to run AX R1 until the OnPremise (R2) version is released. In the meantime Microsoft has to release Windows Server 2016 with the Azure Pack and SQL Server 2016 since these are three Key Enablers for AX OnPremise deployments. And finally customers have the possibility to migrate from Azure Public Cloud to an Azure Private Cloud for their Production Deployments... At least this is how I read the current Road Map.


The only thing I'm absolutely sure about, is that the future of AX in many ways will be different with a set of new challenges for everyone involved. I have seen the product evolve from the early days (Damgard Axapta 1.5) through the transition since Microsoft bought Navision, and if I used the term Microsoft product when describing AX 2012 (back then the Microsoft Stack was Central), Dynamics AX is indeed a Microsoft Product underpinned by Microsoft Strategic positioning as a Public Cloud provider. It's worth reflecting more around  the fact that Microsoft is getting closer to the Customers and maybe I'll write a post on this later.

AX 2012 R3 - Data Import Export Framework - Key not valid for use in specified state

I came accross this issue after replacing the Business Database in a Test Environment. This should not be a very common issue, but I'm sharing it since the possible solution is rather simple.

Scenario

When validating a Data Source (DB) from the form Data Import Export Framework > Setup > Source Data Formats, the following exception was thrown:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Security.Cryptography.CryptographicException: Key not valid for use in specified state.

After examining the underlying code, it pointed to the a specific table.

Solution/workaround

NB! I have deliberately choosen not to reveal the real table or column name (pretty obvious)

1. Run a simple SELECT from the table.
2. Check if a value is stored in the "Secret" Column for the Data Source Type (typically DB:Database).
3. If you find a value in this "Secret" Column, try to run a simple UPDATE statement to clear the value.
4. Do a Log Off - Log On Sequence in the AX Windows Client and try to validate the Data Source again and you will probably have solved your issue.

Hypothesis

When replacing the Business Database, it's always a Best Practice to create a new GLOBALGUID. It could be that the GLOBALGUID is used as the or part of the Encryption Key. After changing the GLOBALGUID, it's no longer possible to Decrypt the information stored in the table (and that's the way it should be).

Update on AX 2012 R3 AOS Service Principal Name (SPN)

I wrote about this back in June 2014 and based on the experience gained since this and the fact that Microsoft just published some relevant information, it's necessary with an update to this subject.

First of all, this is all about securing RPC Connections requiring Kerberos Authentication. RPC Connections are connections through the AX Windows Client or the .NET Business Connector. Not all AX Environments will require Kerberos Authentication, but the AOS Binary will by default try to  register (and de-register) a Service Principal Name (SPN) at Service Start up (and Shutdown) to support "a Kerberos and NTLM mix" (see URL for corresponding Microsoft Blog post below).

Secondly, this has nothing to do with the installer confusing a Standard Service Account with a Managed Service Account in AD. The number part of the SPN is in fact the reference to the Service Class for the AOS instance.

And finally, Microsoft recently (March 20 2015) published information on their Microsoft Dynamics AX Support Blog titled Enhanced security with Kerberos only authentication in Microsoft Dynamics AX. This is the only information I have found covering this subject and it contains very valuable information to understand the new behavior. And most importantly it reveals that this is now the default behavior for all Binaries released after AX 2012 R2 CU7 covering AX 2009, AX 2012 R2 and AX 2012 R3.

If not already described as a requirement by the customer, you should ask your customer if they require Kerberos or not. If they require Kerberos, you need to approach this as a separate configuration task since it requires more that the AOS successfully registering the SPN in AD. If the customer does not require Kerberos (can live with NTLM) Authentication and you want to get rid of the Error logged at AOS Instance Start and Stop, all you have to do is

1. Regedit - Add a new String Value to the Current AOS Configuration with the name authn_regspn
2. Enter 0 (zero) as the value of authn_regspn

This will disable automatic registration and de-registration of the SPN, and avoid the message from being logged to the Event Log every time the AOS Instance starts and stops. And it will possibly also shave off some seconds from the start and stop sequence.

Additional information can be found in the Dynamics AX Community Forum.

I will try to write about a possible approach to configure AX for Kerberos Authentication in a separate post.

AX 2012 R3 - Data Import Export Framework - Could not load file or assembly Microsoft.SQLServer.ManagedDTS, Version=11.0.0.0

Scenario

• A single server install of AX 2012 R3 used for Data Migration purposes
• SQL Server 2012 was originially installed on the server together with all DIXF Components
• SQL Server 2014 implemented (uninstall SQL Server 2012 Services and Shared Components and install SQL Server 2014 Services with Shared Components) - DIXF Components kept

After restoring databases from another AX solution and performing the needed operations on the databases, Preview in DIXF raised a Stop error in AX with the following message "Could not load file or assembly Microsoft.SQLServer.ManagedDTS, Version=11.0.0.0".

Solution

1. Stop the AOS instance
2. Reinstall (uninstall - install) DIXF Components
3. Start the AOS instance
4. Verify DIXF Preview functionality

Hypothesis

DIXF registers the reference (in AOT) to this SSIS assembly at DIXF install time.

Microsoft Dynamics Technical Conference 2015

First of all - A Happy New Year!

I attended The AX Technical Conference last year and after following the progress of the upcoming Conference (Session Catalog), I have concluded that I can't justify the cost (attendance, travel, living and lost Income for my employeer) of attending this year. The main reasons are the lack of deep technical sessions (level 400) and the fact that the Conference now also covers CRM which is not on my agenda.

I found last years Conference to be "light" with regards to technical depth and coverage, but my expectations where based on what was called the Technical Briefing back in the days when Damgaard owned Axapta (some years ago - I know) and not the fact that AX 2012 R3 was a functional release. My latest experience before AXTech 2014, was Convergence in 2008 and even this conference (more customer oriented) was more technical compared to AXTech 2014.

I see the dilemma for Microsoft with the current product focus around added functionality (horizontally and vertically). The first event or conference covering AX 7 (AX 2016?) will be more interesting to me, since this release means a real architectural change (product revolution and a tehnological shift). At the time of writing this post (the first one in 2015), this is still unknown, and I don't expect AX 7 to be covered until the next Technical Conference in 2016. I wonder what Microsoft will name this conference when it again quailify for AX Technical Conference...

I have had my first looks at AX 7 CTP4 in Azure and I will probably post some information when the time allows it.

Update on Life Cycle Services, AX Update Installer

This is a short follow up on my last post.

LCS has been down for partners in Norway for a while, but it is available now (maybe a candidate for a later post titled “The downside of being dependent upon services in the Cloud”).

A short check today (October 31 2014) revealed a change in the mysterious “Updates until a given date” limitation and right now, this includes Updates until October 19 2014 as shown below:

Like people blogging about the different updates (builds) for AX, SQL Server and SharePoint (very useful information indeed and a life saver), this could also be a hot candidate for such information.

But the gap is not as big as it was last time and it seems like Microsoft has included most of the available updates for AX 2012 R3 in the current range.

Don't get me wrong - I think Life Cycle Services will be an important part of the future services around AX, and clearly something Customers (projects are initiated by Customers) and Partners (invited to participate) will benefit from in a long term relationship. Many thoughts and reflections could be tied to this, but I'll leave it for now...

Life Cycle Services, AX Update Installer

Microsoft is now investing heavily in Life Cycle Services (LCS) with monthly releases.

One of the most interesting features for Technical resources is AX Update Installer found under the Updates tile for AX 2012 R3 projects. This is the content after opening the tile:

After downloading the package, you have to extract the special build of AXUpdate.exe and then execute AXUpdate.exe to connect to the project in Life Cycle Services and get the magic started.

I will not go into details around this, since it is a quite straightforward and self-explaining process. One thing to mention is the requirement to have the System Diagnostics configured and at least have uploaded the Discovered Environment once. This uploads (push) Diagnostics information from your On Premise solution to Life Cycle Services including Metadata, which is important for several of the technical features in LCS like Cloud Power Support and Updates.

What I found compelling about the new AXUpdate approach was that it seemed to address the pain trying to keep up with the number of hotfixes released both for the application and for binaries. Therefore, I went through a test earlier in September to see how this really fitted with the description and my expectations. The expected benefit was to be able to produce solution specific update packages without the need to identify and download many individual fixes between Cumulative Updates (a collection of hotfixes and new features). This would be very efficient in a project where we are approaching GOLIVE on R3 (CU8 will be the first CU for R3, but no formal information about expected release date yet).

Everything played surprisingly well and I managed to update a default AX 2012 R3 solution (Model Store) without any modifications with approximately 500 application fixes and binaries for all server roles. The possibility to create and store an Update Package is another nice feature.

After finishing the Update process including the The Model Store has been modified Wizard, it was time for QA and I therefore opened Help > About from the AX Windows Client. As expected I now had many new Models in the SYP-layer (you get one SYP-model for each KB number), but when I checked the current Kernel Build number against the latest/current build number, I discovered that the updated binaries had a lower Build number than the most recent released Binary Build. This was indeed not the expected outcome…

After a couple of days I ran AXUpdate.exe again to check if any new hotfixes where available and the answer was zero (0) applicable updates. I then reviewed one of the first steps in the process after logging in to LCS called Select updates and this revealed the reason as shown below (performed October 06):

In the Combobox to Select a group of updates to evaluate, Most recent was explained as “Contains all updates until August 04, 2014”.

I have not checked after this, but Microsoft has for whatever reason defined a limitation on the set of available updates to include updates until a given date. If this was (is) intentionally it is hard to understand why since the power of AXUpdate.exe connected to LCS, is to address the pain of discovering available updates and downloading them periodically (or when needed) as a package.

I will check this again as soon as LCS is available (still inaccessible for me) to see what the date limit is right now – hopefully this was a onetime glitch and not an intended limitation.

Bottom line – do not be surprised if you do not get all available updates when using AXUpdateInstaller connected to LCS. I would like to add that this was a possible onetime glitch, but it could also explain why you do not end up with the updates you expect.

AX 2012 R3, Error running Cmdlet Test-AXReportServerConfiguration -The Network Path not Found

After successfully installing Reporting Extensions and modifying the Report Configuration, I wanted to test the configuration from the RS Server. The cmdlet test-axtestreportserverconfiguration is obviously the best candidate to do the job, but it threw an error stating "The Network Path was not found" and "Could not open the registry to read the EnableLUA key" like shown below (ran the AX Management Shell as Administrator):

I have experienced issues due to UAC (User Access Control), but after first trying to disable UAC completely (registry, EnableLUA key) on the RS Server, the error still was thrown.

I then noticed that the error actually was thrown when the cmdlet tried to read the EnableLUA key on the AOS Server (it's a two-phased test) and after checking for some references, I found an article on TechNet titled "Configure Remote Management in Server Manager" (Windows Server 2012 & R2) and the section "To configure MMC or other tool remote management over DCOM", seemed to point in the right direction.

After enabling the following rules (and Enabling EnableLUA again), the cmdlet worked as expected.

• COM+ Network Access (DCOM-In)*
• Remote Event Log Management (NP-In)
• Remote Event Log Management (RPC)
• Remote Event Log Management (RPC-EPMAP)

*It's probably the DCOM-In rule that's most important (I did not check combinations of the rules since I was working on building a Production solution for a customer).

Verification

Windows Firewall will have different rules enabled according to the customers policy and you should pay attention to the Firewall rules when experiencing issues during installation of AX Components or when a cmdlet throws errors. I have described several issues due to Windows Firewall in other posts related to installation issues and this was the first one related to PowerShell cmdlets.

AX 2012 R3 Data Import Export Framework (DIXF), Exception at Microsoft.Dynamics.AX.Framework.Tools.DMF.ServiceProxy.DmfEntityProxy.DoWork[T](Func`1 work)

A preliminary warning about a potential issue for Data Import Export Framework released for AX 2012 R3 based on experience from 

• Scenario 1: Single-Server install (AOS and SSDS + SSIS with all DIXF Components) 
• Scenario 2: Multi-Server install (2x AOS with DIXF AOS and Client Component, SSDS + SSIS with DIXF Framework Service Component)

In scenario 1, the Service Account for the AOS Service was utilized for the DIXF Framework Service while in scenario 2 a separate Integration User Account was utilized for the DIXF Framework Service.

Symptom:

In scenario 2 an exception was thrown when validating the working directory for DIXF (button in form DIXF Parameters) > Microsoft.Dynamics.AX.Framework.Tools.DMF.

ServiceProxy.DmfEntityProxy.DoWork[T](Func`1 work).

This worked fine in scenario 1.

Issue 1 - Local Security Group not created by the installer when installing the DIXF AOS Component:

When installing the DIXF Framework Service, a new Local Security Group called Microsoft Dynamics AX Data Import Export Framework Service Users was created and populated with the AD Account hosting the DIXF Framework Service. This Local Security Group was not created by the installer when installing the DIXF AOS Component on the AOS servers. After manually creating this Security Group on the AOS servers, populating it according to the instructions given in Install the Data import/export framework (AX 2012 R3) [AX 2012] and restarting the AOS servers (caching), the error still was thrown.

Issue 2 - Required membership in new Local Security Group:

The instructions on TechNet states that the DIXF Framework Service Account must be added to the Security Group on the server hosting the Framework Service, and similarly the AOS Service Account to be added to the Security Group on the servers hosting the AOS Service and the DIXF AOS Component. In scenario 2 I had to add both Service Accounts to the Security Group on all three servers followed by a restart of the AOS servers, to get the validation working without throwing the exception (issue solved)

It could of course be something related to the In-Place Upgrade procedure, but I doubt this since you have to uninstall all pre AX 2012 R3 Components before any AX 2012 R3 Components can be installed. My best practise includes a restart of each server after performing the uninstall and other clean up tasks, and I think the difference between the scenarios (single versus multi-server setup), points in the direction of the Security Groups not being created automatically by the installer and that all Accounts used by DIXF, must be added to each group.

AX 2012 R3 AOS Service Principal Name (SPN)

March 26 2015 - please also read my Update on this subject.

After working my way through both a new installation and upgrades from AX 2012 R2 (CU7), I find at least one glitch that 1) logs an Error in Event Log - Application when stopping and starting the AOS Service  and 2) that possibly increases the time to start the AOS Service.

I notice the following events beeing logged:

• Start of AOS Service > Object Server 01:  RPC error: Failed to register service principal name (SPN): '<GUID>/<server>.<FQDN>:<AOS RPC Port#>' 
• Stop of AOS Service > Object Server 01:  RPC error: Failed to unregister service principal name (SPN): <GUID>/<server>.<FQDN>:<AOS RPC Port#>' 

This seems to be related to utilizing Managed Service Accounts (an option when specifying the AOS Service Account) or not, where it seems like the AOS Service operates like it's always hosted by a Managed Service Account. In all installations I have used "Use the following account" and not the "Use a managed service account", but despite this deliberate choice, the messages are beeing logged.

Maybe it's difficult or even impossible to separate the different types of Service Accounts, but it should be possible to have this as a property for the Service (registry). The time it takes to start the AOS Service seems to increase in each release of AX 2012 due to the increased number of processes the AOS Service must handle and possible time consuming tasks like this, should indeed be avoided.

Bottom line - I'm supprised to see SPN errors beeing logged like this even when not utilizing a Managed Service Account (which requires a SPN and new requirements regarding integration to objecst in Active Directory).

AX 2012, AifMessageInspector::AfterReceiveRequest - Failed to Logon to Microsoft Dynamics AX

From what I can see, many people have stumbled upon (aka experienced) this beeing logged in the Event Log (Application) on servers hosting AOS instances. I also see this beeing logged especially on AOS servers hosting AOS instances handling Enterprise Portal and Enterprise Search load, but also on other AOS servers (more sporadic).

Some relevant examples:

• http://community.dynamics.com/ax/f/33/t/120096.aspx
• http://community.dynamics.com/ax/f/33/t/109144.aspx
• http://dynamicsuser.net/forums/p/72354/385352.aspx

  
  

In the last link above, Martin Dráb suggests to look at the Service Operations granted and this is the best suggestion I have seen so far. I have not verified this yet, but I would recommend to have a look at the Security Roles and especially the Service Operations granted to a specific Security Role. A good starting point is the articles Services, service operations, and service groups [AX 2012] and About role-based security in services and AIF [AX 2012] on TechNet.

My hypotese is that AX simply reports back Failed to Logon instead of a more informative description pointing to missing Privilegies or Duties.

All comments are welcomed to share some experience and reach a final solution for this rather hard to diagnose issue (no information about which Service except it's an AIF Document Service and what user is Failing to Logon to AX).

WCF Tracing could perhaps reveal the AIF Document Service and User (I have not looked at this yet), but Exception Logging will not catch this (the issue happens before the actual Message reaches the Queue in AX).

The only solution I would not recommend of the ones I have seen suggested, is to add the Business Connector Proxy User (System Account) as a User in AX. Why? This is a privileged user account in AX and the fact that it's covered by a specific form in AX, should tell you that this account should not be treated as a normal user in AX...

AX 2012 R2 CU6, Feature to reduce the effect of Parameter Sniffing

Working with many AX solutions over the years, I have seen Parameter Sniffing in many solutions and the effect of "bad Execution Plans" in SQL Server, can be rather "not so nice" to all end users.

Most of the solutions I see have many companies or Legal Enteties (LE) as they are refered to in AX 2012, but the distribution of the data in shared tables, are very commonly skewed where one company or LE typically allocates 80 to 9x percent of the rows while 8 - 10 companies/LE contribute to the rest of the rows in individual shared tables. Microsoft introduced PARTITION as a new way of isolating companies or LE in AX 2012 R2 and this led to another column in most tables contributing to the parameter sniffing issue. AX 2012 R2 is designed to have several partitions and many LE each representing a balanced data distribution (each partition and Company/LE having roughly the same number of rows), but in real life, this is very seldom what customers end up with.

I would say that Parameter Sniffing is not a phenomen (it's in fact a SQL Server by design behaviour), but it tend to hit AX solutions very hard when the design criteria is not met. The x-factor lies in the fact that Microsoft utilizes Parameterized Queries heavily to make sure every Query is executed "the same way" (also the main reason for recommending MAXDOP = 1) each time and to reduce the number of SQL Compiles and Recompiles (has a small CPU cost). But the downside is that the actual value for the parameter when the statement reaches SQL Server for the first time (or the SQL Server is experiencing mamory pressure with flushing of execution plans), must represent "an average" partions and/or company/LE. The x-factor can simply be caused by the the first user logging on to AX the morning (early bird) after the AOS instances and/or SQL Server has been restarted and representing a small company with regards to the number of rows in shared tables, can "trick" SQL Server in generating a execution plan where SQL Server find it most efficent (based on the cost) to perform a table or index scan. When an user representing the largest company (number of rows in the underlying tables) implicitly reuses the same execution plan, all sorts of bad performance reports starts popping up. Looking at the SQL Server you will probably notice constantly high CPU utilization and digging a little further, You will probably identify one or several statements consuming much CPU time (some of then probably also beeing utilized by API Cursors).

Microsoft introduced a new feature in AX 2012 R2 CU6 to overcome this vulnerability and it's a kernel feature that's turned off by default. The feature is well described in the article Overcoming parameter sniffing issue in Microsoft Dynamics AX 2012-R2 – CU6 written by the AX Performance Team.

I have been aware of this rather important change (results in values for the columns PARTITION and DATAAREAID beeing always sent to SQL Server as literals or actual values instead of parameterized values), but I tend to await activation until I actually see (have proofe of) that a solution is impacted by Parameter Sniffing. Last week I saw proof of this happining in a busy solution focused around Sales Order processing and after activating the changes in the table SYSGLOBALCONFIGURATION followed by a stop - start sequence of the AOS instances, we imediately saw SQL Server generating execution plans using literals for the columns described which again "calmed" the solution. At the same time the number of SQL Compiles and Recompiles increased somewhat, but this was expected as the number of Execution Plans for each Parameterized Statement will increase by the product of the number of Partitions and the number of individual companies/LE utilizing a spesific module in AX.

This is perhaps one of the most important changes Microsoft has introduced and I have not seen any other solution improving (trace flags, force literals keyword etc.) or near eliminating this issue until this feature was introduced. Everyone involved in implementing AX 2012 solutions should be aware of this and be able to utilize this feature when needed. I say AX 2012 since MS stated that this change was expected to be back ported to AX 2012 (I don't know the status), but it's confirmed in AX 2012 R2 CU6 and later binary builds. Customers still utilizing AX 4.0 or AX 2009, will probably never benefit from this feature.

Many factors play a part in this story such as SQL Server Statistics, Max Degree of Parallism and API Cursors, but this is the single most efficent method of helping SQL Server generating the most representative (balanced) Execution Plans for AX based on my experience.

Hyper-V versus VMWare vSphere

This could be a long post with a lot of arguments, but I'll keep it short and just summarize based on some experience from the field.

I guess I have seen and worked with something between 15 to 30 different AX installations utilizing VMWare vSphere 5x ranging from Private Clouds to Public Clouds (typically a shared service provider). The common experience is that virtual machines (vms) with more than one (1) vCPU, seems to have performance issues. The normal approach will be to take the number of vCPUs down to one (1) and monitor the effect which in 9 out of 10 situations are improved performance. But it often ends up in a discussion with the admins arguing that there are no signs of wait time (like CO-STOP and not at least READY-time) and nothing is done (the worst case I have seen was a SQL Server 2008 R2 vm utilizing 8 vCPUs without any good explanation of why it needed 8 vCPUs - the customer still runs with 8 vCPUs...). These situations are hard and they require a lot of stamnia to succeed.

I have also been lucky enough to work with Hyper-V running on Windows Server 2012 R2 (5 solutions on 2 different platforms) and the experience is that Hyper-V guests with multiple vCPUs (sockets and/or cores) outperforms vSphere. The latest experience is with a customer outsourcing everything to a third party hosting partner normally utilizing VMWare vSphere. The first experience was that we got a very good effect by configuring all vms with 1 vCPU (all vms had 4 vCPUs when we started), but still not good enough. In this situation we also had the exact same solution installed on Hyper-V (very similar hardware and SAN) and even with less vCPU allocated in total, the solution utilizing Hyper-V was used as the benchmark by the customer. The VMWare plattform was a test plattform and it was most likely overcommitted on v/pCPU (not ununsual on test plattforms). Even moving the storage for most vms to SSD in a High End SAN, did'nt solve the performance issue. The customer then entered an agreement with the hosting partner to build a new Hyper-V platform on Windows Server 2012 R2. The immediate impression when working with the new platform, is that it performs very well and even better than the other (comparable) Hyper-V platform most likely due to better pCPU specification (clock frequency and CPU model is important). The AX solution is beeing built and the customer has not tested it, but the servers are snappy when working interactively which is always a good indicator (logon takes just a few secounds compared to the same solution running on VMWare). And we utilize vmxnet3 with RSS enabled on VMWare vms...

It seems to be a major architectural difference between VMWare vSphere 5.x and Hyper-V on Windows Server 2012 R2 where I think vSphere uses the UNIX approach when scheduling vCPU to pCPU (all vCPUs has to move in at the same speed - if one vCPU lacks behind, the others are sloved down) while Hyper-V seems to use a more individual CPU scheduling (each vCPU is scheduled on it's own - they move as individuals instead of as a group).

A lot of factors is in play, but I expect to see more and more platforms utilizing Hyper-V on Windows Server 2012 R2 (core) in the future. My advise is to compare the hypervisors on identical hardware and storage before making the final decision. As a not confirmed side note, I have heard that Microsoft is planning a major upgrade of their Azure Data Centers where the keyword is Windows Server 2012 R2 (still utilizing Hyper-V on Windows Server 2012). This is not verified information and highly unofficial, but if it's true, we can expect a major increase in performance for servers running in Azure.

AX 2012 R2 CU6 - Setup Failed ((Exception from HRESULT: 0x80070422)

A new issue encountered yesterday when installing AOS, Client, Debugger and Management Utilities on a brand new (virtual) Windows Server 2012 Standard guest (Hyper-V):

Symptom:

The Installer kicks off as usual, but reports failure for Client, Debugger and Management Utilities (very unusual) and success for AOS (nothing was actually installed).

The Setup log contains the following relevant information:

2014-02-11 18:08:41Z Component installation task stopped due to an error.

2014-02-11 18:08:41Z *********************************************************************************
2014-02-11 18:08:41Z *********************************************************************************

2014-02-11 18:08:41Z The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)

2014-02-11 18:08:41Z *********************************************************************************
2014-02-11 18:08:41Z *********************************************************************************

2014-02-11 18:08:41Z System.Runtime.InteropServices.COMException (0x80070422): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)
2014-02-11 18:08:41Z   at WUApiLib.UpdateServiceManagerClass.QueryServiceRegistration(String ServiceID)
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.MuApiWrapper.IsMachineAlredyOptedIntoMU()
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.Support.SetupQueue.MsiInstallationClass.SearchDownloadInstallMicrosoftUpdates(String executionName)
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.Support.SetupQueue.AOSInstaller.PerformWork()
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.Support.SetupQueue.InstallationClass.DoWork(Object sender, DoWorkEventArgs e)
2014-02-11 18:08:41Z   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

2014-02-11 18:08:41Z *********************************************************************************
2014-02-11 18:08:42Z S260FinishedInfo


2014-02-11 18:08:59Z === Setup was completed successfully.

2014-02-11 18:08:59Z === Setup logging ended: 2/11/2014 6:08:59 PM ErrorLevel/ExitCode: '0' === 

Root cause:

The Windows Update Service (Service Control Manager) was Disabled

Solution:

Alter the Windows Update Service from Disabled to Manual (triggered) or Automatic (I used the first alternative)

Additional information:

I did'nt select Use Windows Updates, but it seems like the installer ignores this and tries to start the Windows Update Service. If the Windows Update Service is Disabled, setup fails.

AX Technical Conference 2014, brief summary

My ambition was to write a blog post for each day, but I decided to write a brief summary to highlight the key takeaways from the conference. I also think it’s important to respect the disclaimer given at the start of each session (not go into the details revealed).

AX 2012 R3 is mainly a functional release adding new functionality (Warehouse Management and Transportation) and improvements/extensions within existing industries like Retail. In addition, Life Cycle Services (LCS) was emphasized as the central HUB throughout the lifecycle of an AX solution bundling many of the individual services and tool sets (add-ons) in an Azure hosted environment. At least this is what Microsoft want customers and partners to adopt and utilize to manage a solution as a project including Project Management, Diagnostics and Configuration in addition to Issue Management and Servicing (when submitting an Issue, a repro solution will automatically be deployed in Azure reflecting the parameter settings from the running solution and Contoso data). The Deployment Tool for Azure was also presented making it possible to deploy an AX solution to Azure IaaS (Demo, Development, Test and Production) utilizing templates. The installer for AX 2012 R2 CU7 already revealed an integration to LCS when selecting the Advanced Update option utilizing the metadata to filter hotfixes related to a specific process. Mobile applications was also given much attention.

The key message was that R3 will be the foundation for the future and an unofficial indicator of the increase in functionality, was +20% compared to R2 measured in code size (Model Store Database). Another measure was that R3 included more new functionality than the complete AX 2009 application.
All in all a load of information to digest and try to adopt in customer deliveries. Microsoft talked a lot of Azure and the reason for describing R3 as the foundation for the future, was supported by some pieces of information around the next major AX release (AX7) code name “Rainier” (a monumental mountain south-east of Seattle WA). From what I understand, this will be a major architectural release cloud enabling AX (no surprise given the focus on Azure). Regarding Azure, I think Microsoft is paying much attention to the US market already covered by four Azure Data Centers (what about the rest of the world like Europe?) and not at least little information regarding the cost of utilizing Azure (some kind of subscription required).

All in all good value, but I personally think the name of the conference is misleading (AX Technical Conference) since it’s not very technical (one level 400 session, rest level 200 and 300) compared to the old Technical Briefings (Damgaard/Navision). A lot of sessions was designed for the Business roles (US Payroll, eProcurement, Demand Forecasting etc.) which I find hard to relate to the technical roles (Developers and Technical resources). The AX Conference would perhaps be a better name given the broad content presented. Maybe I’m the only one, but my expectation was deeper technical coverage. Some logistic challenges with full sessions was also a reality and I would recommend Microsoft either to decrease the number of attendants or book a venue with overcapacity. It’s a little disappointing to miss a session (I actually missed two sessions due to full rooms) when travelling from Norway, but I guess it’s difficult to predict the interest for each session when designing the room plan.

But I'm rest asured that the future will bring AX closer to the vision of Empowering People and that the community must work hard to adopt the constant changes (Dynamics).

My verdict is 5 out of 10 stars with the Q & A (Ask The Experts) as the most valuable sessions and a very good keynote on day two.

If the time allows, I will go a little deeper around my personal predictions around AX and Azure, and maybe also some follow up on specific sessions from #AXTech2014.

Update

Since R3 is a functional release, it probably makes sense that AXTech2014 focused most on the functional aspects of R3. But even this makes it hard to call it a Technical Conference in my oppinion - maybe next year will fullfill my expectations (AX7)...

AX Technical Conference (#AXTech2014)

The Technical Conference is a major event in the AX Community and I'm scheduled to attend (thank you). It's been a while since I last attended a similar conference (I think it was Convergence in Copenhagen back in 2008) and after looking at the agenda and session catalogue, it seems to cover a lot of ground both from a functional and technical point of view.

The conference is called the AX Technical Conference, but it seems like Microsoft will cover the functional area. The focus is of cource AX 2012 R3 scheduled for RTM in Q2 2014 (CTP 4 right now). AX 2012 R3 seems to be a functional release incorporating the Warehouse and Transportation functionality bought from Blue Horseshoe (Industry focus) and improvements within the Retail area. In addition some technical news like the Deployment Tool for Azure. The latter is no supprise given the major focus on Azure which we also see in the Norwegian market (more in a later post togheter with some thoughts around the future).

So what do I expect? Well most of all this is a good opportunity to sit down and pay attention to the people closest to the product and an opportunity to interact with key resources. In addition I expect to go back to Norway next week with a better understanding of R3 and increased understanding of what we have ahead of us the next year. With the next major release of AX ("Rainier") beeing scheduled for release as soon as Q4 2014, I also expect some information related to this which probably will introduce some ground braking changes for the presentation layer in AX. All in all a good opportunity to slow down from the rather hectic daily schedule and fill up with valuable targeted and product specific information.

I'm not very into NFL, but I noticed that the Seattle Seahawks has reached the Super Bowl final and we will probably notice a little bit of interest around this event also...

I'm off to Seattle tomorrow (11 hour flight) and I will try to report the headlines from each day next week and give a final assesment based on my expectations when I return to Norway.

AX 2012 - Unable to write the generated WCF configuration to local storage

I noticed the following entry logged in Event Log, Application on 3 EP/ES servers:

Unable to write the generated WCF configuration to local storage. The generated WCF configuration will be used from memory. The contents of the new configuration are written to the following temp file: C:\Users\i<BC Account> \AppData\Local\Temp\Microsoft.Dynamics.AX.Framework.Services.Client.Configuration.ClientConfigurationInternal.log.

Exception details:
System.ComponentModel.Win32Exception (0x80004005): Access is denied
   at Microsoft.Dynamics.AX.Framework.Client.ConfigurationModel.NativeRegistry.OpenNativeRegistryKey(String keyPath, Boolean writable)
   at Microsoft.Dynamics.AX.Framework.Client.ConfigurationModel.MachineWideRegistryConfigurationModel.GetConfigurationKey(Boolean writable)
   at Microsoft.Dynamics.AX.Framework.Client.ConfigurationModel.RegistryConfigurationModel.WriteWcfConfiguration(String configuration, Guid versionId)
   at Microsoft.Dynamics.AX.Framework.Services.Client.Configuration.ClientConfigurationInternal.GetClientConfiguration()

All servers configured to use local configurations stored in the registry.

Possible solution:

1. Grant the AD Account used as the Business Connector Proxy Account, Full Control to the current configuration in the registry (remember HKEY_LOCAL_MACHINE - not HKEY_CURRENT_USER)
2. Use the approach described in Install multiple Enterprise Portals on the same server [AX 2012] (TechNet) to use persistent (AXC-files) configuration files instead of local configurations stored in the registry

As a side note, this is another example of undocumented requirements, and in my mind something the installer should configure during installation or at least describe as a manual configuration item.

Apparently the Business Connector binary (assembly) has built in functionality to automatically refresh the WCF configuration part of the AX configuration requiring additional configuration steps to be performed.

Update:

This could also be caused by Group Policy (domain) settings applied after the AX Components were installed.

AX 2012 R2 - Minor issue (Startup Procedures SQL Server)

Most technical resources working with AX 2012 R2 are probably aware of the fact that AX deploys a Stored Procedure to the Master database to grant permissions to TempDb to the Service Account hosting tht AOS Service (instance). This is necessary since TempDb is created at startup, but be aware that the setup routine (AX AOS) adds one procedure for each AOS instance installed. Since these Stored Procedures are launched each time the SQL instance is started, it makes no sense to do this for each AOS instance. Let's say you have 10 AOS instances in Your environment utilizing the same Service Account. SQL Server will then execute the same statement 10 times during the startup sequence for the SQL Server instance. I discovered this by enabling Trace Flag 1204 and 1222 (records Dead Lock situations in the SQL Log).

Example (4 AOS instances, 1 on SQL Server used for full X++ compilation):

Stored Procedures created in the Master db:

 



Note that these Stored Procedures are identical if you use one Service Account for all AOS instances (would make sense to have one for each Service Account used).

Inside the Stored Procedure we find the following check:

if (@dbaccesscount <> 0) 

The variable @dbaccesscount holds the number of logins matching the Service Account found in TempDb. And it then grants the required roles for the Service Account if @dbaccesscount is not equal to 0... In my mind the criteria should simply be if (@dbaccesscount = 0).

You have the possibility to either change the logic (breaks the support rule?) or you could do like me and disable all except one of them. By disable I mean ALTER the procedure from launching as a Startup Procedure by running sp_procoption.

Again remember that You should have one Startup Procedure for EACH Service Account used, but it's not required to execute the same grant statement more than ONCE for each Service Account used.

Finally - Why bother? I like to have the systems clean and as efficent as possible, and it's not worth spending extra CPU cycles especially in virtual environments.

So long