Update on Life Cycle Services, AX Update Installer

This is a short follow up on my last post.

LCS has been down for partners in Norway for a while, but it is available now (maybe a candidate for a later post titled “The downside of being dependent upon services in the Cloud”).

A short check today (October 31 2014) revealed a change in the mysterious “Updates until a given date” limitation and right now, this includes Updates until October 19 2014 as shown below:

Like people blogging about the different updates (builds) for AX, SQL Server and SharePoint (very useful information indeed and a life saver), this could also be a hot candidate for such information.

But the gap is not as big as it was last time and it seems like Microsoft has included most of the available updates for AX 2012 R3 in the current range.

Don't get me wrong - I think Life Cycle Services will be an important part of the future services around AX, and clearly something Customers (projects are initiated by Customers) and Partners (invited to participate) will benefit from in a long term relationship. Many thoughts and reflections could be tied to this, but I'll leave it for now...

Life Cycle Services, AX Update Installer

Microsoft is now investing heavily in Life Cycle Services (LCS) with monthly releases.

One of the most interesting features for Technical resources is AX Update Installer found under the Updates tile for AX 2012 R3 projects. This is the content after opening the tile:

After downloading the package, you have to extract the special build of AXUpdate.exe and then execute AXUpdate.exe to connect to the project in Life Cycle Services and get the magic started.

I will not go into details around this, since it is a quite straightforward and self-explaining process. One thing to mention is the requirement to have the System Diagnostics configured and at least have uploaded the Discovered Environment once. This uploads (push) Diagnostics information from your On Premise solution to Life Cycle Services including Metadata, which is important for several of the technical features in LCS like Cloud Power Support and Updates.

What I found compelling about the new AXUpdate approach was that it seemed to address the pain trying to keep up with the number of hotfixes released both for the application and for binaries. Therefore, I went through a test earlier in September to see how this really fitted with the description and my expectations. The expected benefit was to be able to produce solution specific update packages without the need to identify and download many individual fixes between Cumulative Updates (a collection of hotfixes and new features). This would be very efficient in a project where we are approaching GOLIVE on R3 (CU8 will be the first CU for R3, but no formal information about expected release date yet).

Everything played surprisingly well and I managed to update a default AX 2012 R3 solution (Model Store) without any modifications with approximately 500 application fixes and binaries for all server roles. The possibility to create and store an Update Package is another nice feature.

After finishing the Update process including the The Model Store has been modified Wizard, it was time for QA and I therefore opened Help > About from the AX Windows Client. As expected I now had many new Models in the SYP-layer (you get one SYP-model for each KB number), but when I checked the current Kernel Build number against the latest/current build number, I discovered that the updated binaries had a lower Build number than the most recent released Binary Build. This was indeed not the expected outcome…

After a couple of days I ran AXUpdate.exe again to check if any new hotfixes where available and the answer was zero (0) applicable updates. I then reviewed one of the first steps in the process after logging in to LCS called Select updates and this revealed the reason as shown below (performed October 06):

In the Combobox to Select a group of updates to evaluate, Most recent was explained as “Contains all updates until August 04, 2014”.

I have not checked after this, but Microsoft has for whatever reason defined a limitation on the set of available updates to include updates until a given date. If this was (is) intentionally it is hard to understand why since the power of AXUpdate.exe connected to LCS, is to address the pain of discovering available updates and downloading them periodically (or when needed) as a package.

I will check this again as soon as LCS is available (still inaccessible for me) to see what the date limit is right now – hopefully this was a onetime glitch and not an intended limitation.

Bottom line – do not be surprised if you do not get all available updates when using AXUpdateInstaller connected to LCS. I would like to add that this was a possible onetime glitch, but it could also explain why you do not end up with the updates you expect.

AX 2012 R3, Error running Cmdlet Test-AXReportServerConfiguration -The Network Path not Found

After successfully installing Reporting Extensions and modifying the Report Configuration, I wanted to test the configuration from the RS Server. The cmdlet test-axtestreportserverconfiguration is obviously the best candidate to do the job, but it threw an error stating "The Network Path was not found" and "Could not open the registry to read the EnableLUA key" like shown below (ran the AX Management Shell as Administrator):

I have experienced issues due to UAC (User Access Control), but after first trying to disable UAC completely (registry, EnableLUA key) on the RS Server, the error still was thrown.

I then noticed that the error actually was thrown when the cmdlet tried to read the EnableLUA key on the AOS Server (it's a two-phased test) and after checking for some references, I found an article on TechNet titled "Configure Remote Management in Server Manager" (Windows Server 2012 & R2) and the section "To configure MMC or other tool remote management over DCOM", seemed to point in the right direction.

After enabling the following rules (and Enabling EnableLUA again), the cmdlet worked as expected.

• COM+ Network Access (DCOM-In)*
• Remote Event Log Management (NP-In)
• Remote Event Log Management (RPC)
• Remote Event Log Management (RPC-EPMAP)

*It's probably the DCOM-In rule that's most important (I did not check combinations of the rules since I was working on building a Production solution for a customer).

Verification

Windows Firewall will have different rules enabled according to the customers policy and you should pay attention to the Firewall rules when experiencing issues during installation of AX Components or when a cmdlet throws errors. I have described several issues due to Windows Firewall in other posts related to installation issues and this was the first one related to PowerShell cmdlets.

AX 2012 R3 Data Import Export Framework (DIXF), Exception at Microsoft.Dynamics.AX.Framework.Tools.DMF.ServiceProxy.DmfEntityProxy.DoWork[T](Func`1 work)

A preliminary warning about a potential issue for Data Import Export Framework released for AX 2012 R3 based on experience from 

• Scenario 1: Single-Server install (AOS and SSDS + SSIS with all DIXF Components) 
• Scenario 2: Multi-Server install (2x AOS with DIXF AOS and Client Component, SSDS + SSIS with DIXF Framework Service Component)

In scenario 1, the Service Account for the AOS Service was utilized for the DIXF Framework Service while in scenario 2 a separate Integration User Account was utilized for the DIXF Framework Service.

Symptom:

In scenario 2 an exception was thrown when validating the working directory for DIXF (button in form DIXF Parameters) > Microsoft.Dynamics.AX.Framework.Tools.DMF.

ServiceProxy.DmfEntityProxy.DoWork[T](Func`1 work).

This worked fine in scenario 1.

Issue 1 - Local Security Group not created by the installer when installing the DIXF AOS Component:

When installing the DIXF Framework Service, a new Local Security Group called Microsoft Dynamics AX Data Import Export Framework Service Users was created and populated with the AD Account hosting the DIXF Framework Service. This Local Security Group was not created by the installer when installing the DIXF AOS Component on the AOS servers. After manually creating this Security Group on the AOS servers, populating it according to the instructions given in Install the Data import/export framework (AX 2012 R3) [AX 2012] and restarting the AOS servers (caching), the error still was thrown.

Issue 2 - Required membership in new Local Security Group:

The instructions on TechNet states that the DIXF Framework Service Account must be added to the Security Group on the server hosting the Framework Service, and similarly the AOS Service Account to be added to the Security Group on the servers hosting the AOS Service and the DIXF AOS Component. In scenario 2 I had to add both Service Accounts to the Security Group on all three servers followed by a restart of the AOS servers, to get the validation working without throwing the exception (issue solved)

It could of course be something related to the In-Place Upgrade procedure, but I doubt this since you have to uninstall all pre AX 2012 R3 Components before any AX 2012 R3 Components can be installed. My best practise includes a restart of each server after performing the uninstall and other clean up tasks, and I think the difference between the scenarios (single versus multi-server setup), points in the direction of the Security Groups not being created automatically by the installer and that all Accounts used by DIXF, must be added to each group.

AX 2012 R3 AOS Service Principal Name (SPN)

March 26 2015 - please also read my Update on this subject.

After working my way through both a new installation and upgrades from AX 2012 R2 (CU7), I find at least one glitch that 1) logs an Error in Event Log - Application when stopping and starting the AOS Service  and 2) that possibly increases the time to start the AOS Service.

I notice the following events beeing logged:

• Start of AOS Service > Object Server 01:  RPC error: Failed to register service principal name (SPN): '<GUID>/<server>.<FQDN>:<AOS RPC Port#>' 
• Stop of AOS Service > Object Server 01:  RPC error: Failed to unregister service principal name (SPN): <GUID>/<server>.<FQDN>:<AOS RPC Port#>' 

This seems to be related to utilizing Managed Service Accounts (an option when specifying the AOS Service Account) or not, where it seems like the AOS Service operates like it's always hosted by a Managed Service Account. In all installations I have used "Use the following account" and not the "Use a managed service account", but despite this deliberate choice, the messages are beeing logged.

Maybe it's difficult or even impossible to separate the different types of Service Accounts, but it should be possible to have this as a property for the Service (registry). The time it takes to start the AOS Service seems to increase in each release of AX 2012 due to the increased number of processes the AOS Service must handle and possible time consuming tasks like this, should indeed be avoided.

Bottom line - I'm supprised to see SPN errors beeing logged like this even when not utilizing a Managed Service Account (which requires a SPN and new requirements regarding integration to objecst in Active Directory).

AX 2012, AifMessageInspector::AfterReceiveRequest - Failed to Logon to Microsoft Dynamics AX

From what I can see, many people have stumbled upon (aka experienced) this beeing logged in the Event Log (Application) on servers hosting AOS instances. I also see this beeing logged especially on AOS servers hosting AOS instances handling Enterprise Portal and Enterprise Search load, but also on other AOS servers (more sporadic).

Some relevant examples:

• http://community.dynamics.com/ax/f/33/t/120096.aspx
• http://community.dynamics.com/ax/f/33/t/109144.aspx
• http://dynamicsuser.net/forums/p/72354/385352.aspx

  
  

In the last link above, Martin Dráb suggests to look at the Service Operations granted and this is the best suggestion I have seen so far. I have not verified this yet, but I would recommend to have a look at the Security Roles and especially the Service Operations granted to a specific Security Role. A good starting point is the articles Services, service operations, and service groups [AX 2012] and About role-based security in services and AIF [AX 2012] on TechNet.

My hypotese is that AX simply reports back Failed to Logon instead of a more informative description pointing to missing Privilegies or Duties.

All comments are welcomed to share some experience and reach a final solution for this rather hard to diagnose issue (no information about which Service except it's an AIF Document Service and what user is Failing to Logon to AX).

WCF Tracing could perhaps reveal the AIF Document Service and User (I have not looked at this yet), but Exception Logging will not catch this (the issue happens before the actual Message reaches the Queue in AX).

The only solution I would not recommend of the ones I have seen suggested, is to add the Business Connector Proxy User (System Account) as a User in AX. Why? This is a privileged user account in AX and the fact that it's covered by a specific form in AX, should tell you that this account should not be treated as a normal user in AX...

AX 2012 R2 CU6, Feature to reduce the effect of Parameter Sniffing

Working with many AX solutions over the years, I have seen Parameter Sniffing in many solutions and the effect of "bad Execution Plans" in SQL Server, can be rather "not so nice" to all end users.

Most of the solutions I see have many companies or Legal Enteties (LE) as they are refered to in AX 2012, but the distribution of the data in shared tables, are very commonly skewed where one company or LE typically allocates 80 to 9x percent of the rows while 8 - 10 companies/LE contribute to the rest of the rows in individual shared tables. Microsoft introduced PARTITION as a new way of isolating companies or LE in AX 2012 R2 and this led to another column in most tables contributing to the parameter sniffing issue. AX 2012 R2 is designed to have several partitions and many LE each representing a balanced data distribution (each partition and Company/LE having roughly the same number of rows), but in real life, this is very seldom what customers end up with.

I would say that Parameter Sniffing is not a phenomen (it's in fact a SQL Server by design behaviour), but it tend to hit AX solutions very hard when the design criteria is not met. The x-factor lies in the fact that Microsoft utilizes Parameterized Queries heavily to make sure every Query is executed "the same way" (also the main reason for recommending MAXDOP = 1) each time and to reduce the number of SQL Compiles and Recompiles (has a small CPU cost). But the downside is that the actual value for the parameter when the statement reaches SQL Server for the first time (or the SQL Server is experiencing mamory pressure with flushing of execution plans), must represent "an average" partions and/or company/LE. The x-factor can simply be caused by the the first user logging on to AX the morning (early bird) after the AOS instances and/or SQL Server has been restarted and representing a small company with regards to the number of rows in shared tables, can "trick" SQL Server in generating a execution plan where SQL Server find it most efficent (based on the cost) to perform a table or index scan. When an user representing the largest company (number of rows in the underlying tables) implicitly reuses the same execution plan, all sorts of bad performance reports starts popping up. Looking at the SQL Server you will probably notice constantly high CPU utilization and digging a little further, You will probably identify one or several statements consuming much CPU time (some of then probably also beeing utilized by API Cursors).

Microsoft introduced a new feature in AX 2012 R2 CU6 to overcome this vulnerability and it's a kernel feature that's turned off by default. The feature is well described in the article Overcoming parameter sniffing issue in Microsoft Dynamics AX 2012-R2 – CU6 written by the AX Performance Team.

I have been aware of this rather important change (results in values for the columns PARTITION and DATAAREAID beeing always sent to SQL Server as literals or actual values instead of parameterized values), but I tend to await activation until I actually see (have proofe of) that a solution is impacted by Parameter Sniffing. Last week I saw proof of this happining in a busy solution focused around Sales Order processing and after activating the changes in the table SYSGLOBALCONFIGURATION followed by a stop - start sequence of the AOS instances, we imediately saw SQL Server generating execution plans using literals for the columns described which again "calmed" the solution. At the same time the number of SQL Compiles and Recompiles increased somewhat, but this was expected as the number of Execution Plans for each Parameterized Statement will increase by the product of the number of Partitions and the number of individual companies/LE utilizing a spesific module in AX.

This is perhaps one of the most important changes Microsoft has introduced and I have not seen any other solution improving (trace flags, force literals keyword etc.) or near eliminating this issue until this feature was introduced. Everyone involved in implementing AX 2012 solutions should be aware of this and be able to utilize this feature when needed. I say AX 2012 since MS stated that this change was expected to be back ported to AX 2012 (I don't know the status), but it's confirmed in AX 2012 R2 CU6 and later binary builds. Customers still utilizing AX 4.0 or AX 2009, will probably never benefit from this feature.

Many factors play a part in this story such as SQL Server Statistics, Max Degree of Parallism and API Cursors, but this is the single most efficent method of helping SQL Server generating the most representative (balanced) Execution Plans for AX based on my experience.

Hyper-V versus VMWare vSphere

This could be a long post with a lot of arguments, but I'll keep it short and just summarize based on some experience from the field.

I guess I have seen and worked with something between 15 to 30 different AX installations utilizing VMWare vSphere 5x ranging from Private Clouds to Public Clouds (typically a shared service provider). The common experience is that virtual machines (vms) with more than one (1) vCPU, seems to have performance issues. The normal approach will be to take the number of vCPUs down to one (1) and monitor the effect which in 9 out of 10 situations are improved performance. But it often ends up in a discussion with the admins arguing that there are no signs of wait time (like CO-STOP and not at least READY-time) and nothing is done (the worst case I have seen was a SQL Server 2008 R2 vm utilizing 8 vCPUs without any good explanation of why it needed 8 vCPUs - the customer still runs with 8 vCPUs...). These situations are hard and they require a lot of stamnia to succeed.

I have also been lucky enough to work with Hyper-V running on Windows Server 2012 R2 (5 solutions on 2 different platforms) and the experience is that Hyper-V guests with multiple vCPUs (sockets and/or cores) outperforms vSphere. The latest experience is with a customer outsourcing everything to a third party hosting partner normally utilizing VMWare vSphere. The first experience was that we got a very good effect by configuring all vms with 1 vCPU (all vms had 4 vCPUs when we started), but still not good enough. In this situation we also had the exact same solution installed on Hyper-V (very similar hardware and SAN) and even with less vCPU allocated in total, the solution utilizing Hyper-V was used as the benchmark by the customer. The VMWare plattform was a test plattform and it was most likely overcommitted on v/pCPU (not ununsual on test plattforms). Even moving the storage for most vms to SSD in a High End SAN, did'nt solve the performance issue. The customer then entered an agreement with the hosting partner to build a new Hyper-V platform on Windows Server 2012 R2. The immediate impression when working with the new platform, is that it performs very well and even better than the other (comparable) Hyper-V platform most likely due to better pCPU specification (clock frequency and CPU model is important). The AX solution is beeing built and the customer has not tested it, but the servers are snappy when working interactively which is always a good indicator (logon takes just a few secounds compared to the same solution running on VMWare). And we utilize vmxnet3 with RSS enabled on VMWare vms...

It seems to be a major architectural difference between VMWare vSphere 5.x and Hyper-V on Windows Server 2012 R2 where I think vSphere uses the UNIX approach when scheduling vCPU to pCPU (all vCPUs has to move in at the same speed - if one vCPU lacks behind, the others are sloved down) while Hyper-V seems to use a more individual CPU scheduling (each vCPU is scheduled on it's own - they move as individuals instead of as a group).

A lot of factors is in play, but I expect to see more and more platforms utilizing Hyper-V on Windows Server 2012 R2 (core) in the future. My advise is to compare the hypervisors on identical hardware and storage before making the final decision. As a not confirmed side note, I have heard that Microsoft is planning a major upgrade of their Azure Data Centers where the keyword is Windows Server 2012 R2 (still utilizing Hyper-V on Windows Server 2012). This is not verified information and highly unofficial, but if it's true, we can expect a major increase in performance for servers running in Azure.

AX 2012 R2 CU6 - Setup Failed ((Exception from HRESULT: 0x80070422)

A new issue encountered yesterday when installing AOS, Client, Debugger and Management Utilities on a brand new (virtual) Windows Server 2012 Standard guest (Hyper-V):

Symptom:

The Installer kicks off as usual, but reports failure for Client, Debugger and Management Utilities (very unusual) and success for AOS (nothing was actually installed).

The Setup log contains the following relevant information:

2014-02-11 18:08:41Z Component installation task stopped due to an error.

2014-02-11 18:08:41Z *********************************************************************************
2014-02-11 18:08:41Z *********************************************************************************

2014-02-11 18:08:41Z The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)

2014-02-11 18:08:41Z *********************************************************************************
2014-02-11 18:08:41Z *********************************************************************************

2014-02-11 18:08:41Z System.Runtime.InteropServices.COMException (0x80070422): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)
2014-02-11 18:08:41Z   at WUApiLib.UpdateServiceManagerClass.QueryServiceRegistration(String ServiceID)
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.MuApiWrapper.IsMachineAlredyOptedIntoMU()
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.Support.SetupQueue.MsiInstallationClass.SearchDownloadInstallMicrosoftUpdates(String executionName)
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.Support.SetupQueue.AOSInstaller.PerformWork()
2014-02-11 18:08:41Z   at Microsoft.Dynamics.Setup.Support.SetupQueue.InstallationClass.DoWork(Object sender, DoWorkEventArgs e)
2014-02-11 18:08:41Z   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

2014-02-11 18:08:41Z *********************************************************************************
2014-02-11 18:08:42Z S260FinishedInfo


2014-02-11 18:08:59Z === Setup was completed successfully.

2014-02-11 18:08:59Z === Setup logging ended: 2/11/2014 6:08:59 PM ErrorLevel/ExitCode: '0' === 

Root cause:

The Windows Update Service (Service Control Manager) was Disabled

Solution:

Alter the Windows Update Service from Disabled to Manual (triggered) or Automatic (I used the first alternative)

Additional information:

I did'nt select Use Windows Updates, but it seems like the installer ignores this and tries to start the Windows Update Service. If the Windows Update Service is Disabled, setup fails.

AX Technical Conference 2014, brief summary

My ambition was to write a blog post for each day, but I decided to write a brief summary to highlight the key takeaways from the conference. I also think it’s important to respect the disclaimer given at the start of each session (not go into the details revealed).

AX 2012 R3 is mainly a functional release adding new functionality (Warehouse Management and Transportation) and improvements/extensions within existing industries like Retail. In addition, Life Cycle Services (LCS) was emphasized as the central HUB throughout the lifecycle of an AX solution bundling many of the individual services and tool sets (add-ons) in an Azure hosted environment. At least this is what Microsoft want customers and partners to adopt and utilize to manage a solution as a project including Project Management, Diagnostics and Configuration in addition to Issue Management and Servicing (when submitting an Issue, a repro solution will automatically be deployed in Azure reflecting the parameter settings from the running solution and Contoso data). The Deployment Tool for Azure was also presented making it possible to deploy an AX solution to Azure IaaS (Demo, Development, Test and Production) utilizing templates. The installer for AX 2012 R2 CU7 already revealed an integration to LCS when selecting the Advanced Update option utilizing the metadata to filter hotfixes related to a specific process. Mobile applications was also given much attention.

The key message was that R3 will be the foundation for the future and an unofficial indicator of the increase in functionality, was +20% compared to R2 measured in code size (Model Store Database). Another measure was that R3 included more new functionality than the complete AX 2009 application.
All in all a load of information to digest and try to adopt in customer deliveries. Microsoft talked a lot of Azure and the reason for describing R3 as the foundation for the future, was supported by some pieces of information around the next major AX release (AX7) code name “Rainier” (a monumental mountain south-east of Seattle WA). From what I understand, this will be a major architectural release cloud enabling AX (no surprise given the focus on Azure). Regarding Azure, I think Microsoft is paying much attention to the US market already covered by four Azure Data Centers (what about the rest of the world like Europe?) and not at least little information regarding the cost of utilizing Azure (some kind of subscription required).

All in all good value, but I personally think the name of the conference is misleading (AX Technical Conference) since it’s not very technical (one level 400 session, rest level 200 and 300) compared to the old Technical Briefings (Damgaard/Navision). A lot of sessions was designed for the Business roles (US Payroll, eProcurement, Demand Forecasting etc.) which I find hard to relate to the technical roles (Developers and Technical resources). The AX Conference would perhaps be a better name given the broad content presented. Maybe I’m the only one, but my expectation was deeper technical coverage. Some logistic challenges with full sessions was also a reality and I would recommend Microsoft either to decrease the number of attendants or book a venue with overcapacity. It’s a little disappointing to miss a session (I actually missed two sessions due to full rooms) when travelling from Norway, but I guess it’s difficult to predict the interest for each session when designing the room plan.

But I'm rest asured that the future will bring AX closer to the vision of Empowering People and that the community must work hard to adopt the constant changes (Dynamics).

My verdict is 5 out of 10 stars with the Q & A (Ask The Experts) as the most valuable sessions and a very good keynote on day two.

If the time allows, I will go a little deeper around my personal predictions around AX and Azure, and maybe also some follow up on specific sessions from #AXTech2014.

Update

Since R3 is a functional release, it probably makes sense that AXTech2014 focused most on the functional aspects of R3. But even this makes it hard to call it a Technical Conference in my oppinion - maybe next year will fullfill my expectations (AX7)...

AX Technical Conference (#AXTech2014)

The Technical Conference is a major event in the AX Community and I'm scheduled to attend (thank you). It's been a while since I last attended a similar conference (I think it was Convergence in Copenhagen back in 2008) and after looking at the agenda and session catalogue, it seems to cover a lot of ground both from a functional and technical point of view.

The conference is called the AX Technical Conference, but it seems like Microsoft will cover the functional area. The focus is of cource AX 2012 R3 scheduled for RTM in Q2 2014 (CTP 4 right now). AX 2012 R3 seems to be a functional release incorporating the Warehouse and Transportation functionality bought from Blue Horseshoe (Industry focus) and improvements within the Retail area. In addition some technical news like the Deployment Tool for Azure. The latter is no supprise given the major focus on Azure which we also see in the Norwegian market (more in a later post togheter with some thoughts around the future).

So what do I expect? Well most of all this is a good opportunity to sit down and pay attention to the people closest to the product and an opportunity to interact with key resources. In addition I expect to go back to Norway next week with a better understanding of R3 and increased understanding of what we have ahead of us the next year. With the next major release of AX ("Rainier") beeing scheduled for release as soon as Q4 2014, I also expect some information related to this which probably will introduce some ground braking changes for the presentation layer in AX. All in all a good opportunity to slow down from the rather hectic daily schedule and fill up with valuable targeted and product specific information.

I'm not very into NFL, but I noticed that the Seattle Seahawks has reached the Super Bowl final and we will probably notice a little bit of interest around this event also...

I'm off to Seattle tomorrow (11 hour flight) and I will try to report the headlines from each day next week and give a final assesment based on my expectations when I return to Norway.

AX 2012 - Unable to write the generated WCF configuration to local storage

I noticed the following entry logged in Event Log, Application on 3 EP/ES servers:

Unable to write the generated WCF configuration to local storage. The generated WCF configuration will be used from memory. The contents of the new configuration are written to the following temp file: C:\Users\i<BC Account> \AppData\Local\Temp\Microsoft.Dynamics.AX.Framework.Services.Client.Configuration.ClientConfigurationInternal.log.

Exception details:
System.ComponentModel.Win32Exception (0x80004005): Access is denied
   at Microsoft.Dynamics.AX.Framework.Client.ConfigurationModel.NativeRegistry.OpenNativeRegistryKey(String keyPath, Boolean writable)
   at Microsoft.Dynamics.AX.Framework.Client.ConfigurationModel.MachineWideRegistryConfigurationModel.GetConfigurationKey(Boolean writable)
   at Microsoft.Dynamics.AX.Framework.Client.ConfigurationModel.RegistryConfigurationModel.WriteWcfConfiguration(String configuration, Guid versionId)
   at Microsoft.Dynamics.AX.Framework.Services.Client.Configuration.ClientConfigurationInternal.GetClientConfiguration()

All servers configured to use local configurations stored in the registry.

Possible solution:

1. Grant the AD Account used as the Business Connector Proxy Account, Full Control to the current configuration in the registry (remember HKEY_LOCAL_MACHINE - not HKEY_CURRENT_USER)
2. Use the approach described in Install multiple Enterprise Portals on the same server [AX 2012] (TechNet) to use persistent (AXC-files) configuration files instead of local configurations stored in the registry

As a side note, this is another example of undocumented requirements, and in my mind something the installer should configure during installation or at least describe as a manual configuration item.

Apparently the Business Connector binary (assembly) has built in functionality to automatically refresh the WCF configuration part of the AX configuration requiring additional configuration steps to be performed.

Update:

This could also be caused by Group Policy (domain) settings applied after the AX Components were installed.