Integrating with Dynamics CRM using Advantco’s DYNCRM adapter

Introduction

We recently had a requirement for integration with Microsoft Dynamics CRM Online. After the initial assessment, it became clear that the integration would be challenging as it requires certain functionality that is not available in standard SAP’s adapters. Software vendors typically don’t play nice with each other, so this shouldn’t be much of a surprise – even though all of them claim to adhere to open standards!

Dynamics CRM Online provides the following web services for integration:-

1. Organisation Service

This is the older SOAP-based web service.

2. Web API

This is the newer OData-based web service that is “recommended” for new integrations, but is not yet on complete parity with the SOAP-based web service, and has limitations. (Don’t all the software vendors want you to move to “the latest and the greatest” even though it is actually not the greatest!)

Even though there were two options available, neither one was straightforward to achieve. For the SOAP-based web service, on the surface, it would have sounded easy, but the authentication was not. It required Active Directory authentication against an on-premise ADFS (Active Directory Federation Services) server, with the security token being passed in the header section of the SOAP envelope – do you think the standard SOAP adapter (or even Axis) can handle this…. NO!

As for the OData-based Web API, this presented different challenges. From an authentication point of view, it requires OAuth 2.0 (in particular the Resource Owner Credentials grant type). From a content point of view, it uses the OData v4 format and only in JSON. Again none of this is possible from the standard SAP adapters (whether REST or OData).

So the alternatives are to develop a customized solution or explore third-party adapter availability with any of the SAP Partners.

My good friend Vadim Klimov had worked on a similar requirement previously and went with the custom approach, and this is documented in his blog, Enablement of OAuth 2.0 Authorization in Receiver Communication Channels Using Custom Adapter Module in Advanced Adapter Engine of PI/PO.

However, we decided to go with the third party adapter approach, and went with Advantco’s Dynamic CRM adapter. The following blog by Peter Ha provides some of the details of Advantco’s offering – Microsoft Dynamics CRM integration with SAP PI/PO.

In the following sections, I will share some of my experience working with Advantco’s Dynamic CRM adapter.

Likes

1. Dynamics CRM Workbench

Similar to Advantco’s REST adapter, the Dynamics CRM adapter also comes with its own Workbench. This provides useful developer-friendly features such as connection testing, schema generation (more on this in the next point) and a query function.

The connection testing feature is always useful as it allows us to troubleshoot connection issues even before development begins.

Likewise, the query function is also useful when troubleshooting issues as it allows the developer to execute Dynamics FetchXML queries – this can come in handy when the developer do not have access to the Dynamics CRM system.

2. Schema generation

Although the Dynamics CRM web services are based on SOAP or OData, internally the Advantco adapter do not work with the SOAP or OData schema, but instead works based on an intermediate Advantco schema. The workbench is used to generate this intermediate schema based on the Dynamics CRM entities (and potentially relationships) that will be used for a particular integration. This intermediate format is imported into ESR and used for mapping purposes. Depending on the type of protocol used in the communication channel (SOAP or REST), during runtime the adapter would automatically convert the payload to the corresponding SOAP-XML or OData-JSON format.

This adds a layer of abstraction for the PI developer who is then able to work independent of the actual type of web service used. The intention is also to allow an existing integration using SOAP web service to be switched to the OData service just by changing the channel protocol, without any design changes on the ESR and mapping. This is indeed a very handy feature especially when the OData-based Web API service matures and the SOAP-based Organization Service gets deprecated in the future.

3. Handling multiple calls using query lookup

A common requirement in Dynamics CRM integration is handling relationships between entities. This requires data to be queried from the Dynamics CRM system before constructing the request message. In a traditional PI based integration, this would have required a lookup from mapping or a BPM to perform multiple calls to the target system.

However, the adapter introduces a query feature that is available during schema generation. This adds a Query section in the generated schema as shown below. This allows the query (based on FetchXML) to be constructed during design time, and executed by the adapter automatically during runtime. From PI monitoring point of view, there is only a single PI message, but internally the adapter calls the Dynamics CRM system multiple times – initially to query all the required data, and finally constructing the request message for a final call.

4. Support

What can I say – if you are used to the ping-pong matches whenever you raise a ticket with SAP’s Support team, you will definitely love Advantco’s support!! They are knowledgeable, responsive and helpful 🙂

Possible Areas For Improvements

Well, in this world, nothing is perfect, so these are some of the areas where I feel could be improved.

1. Abstraction due to intermediate schema and logging of multiple calls

Although the usage of the intermediate schema is IMHO an innovative approach for the adapter, it can potentially be a double-edge sword. The layer of abstractions can potentially remove the developer from being able to understand the actual working of the SOAP or OData based services. Personally, I prefer to know what is actually being sent during the call to the Dynamics CRM system, especially when I need to troubleshoot some issue. In our experience, we encountered some issues and the Dynamics CRM consultant asked us to provide the SOAP payload that was used.

Additionally, as mentioned in the previous section, the adapter can potentially perform multiple calls and these are not immediately obvious in the main monitors.

For both the above cases, these require setting up debug levels for logging and tracing in NWA, and is only available in NWA’s Developer Traces. This can be a bit cumbersome, and I would have preferred if the actual content or multiple calls be available in one of the following places instead:-

• Message Log in Message Monitor (similar to their REST adapter)
• As an additional log version in the pipeline viewable from Message Monitor (possibly configurable via a parameter in the channel)
• Using the normal NWA Log Viewer (similar to TraceHTTP for the SOAP adapter)

2. Default keystore for SSL certificates

By default, the adapter uses the JVM’s keystore to check for trusted certificates. This is not consistent with other standard adapters which use the TrustedCAs view of NWA’s Key Storage. The JVM’s keystore requires OS level access and do not have visible indicators if a certificate is expiring or has expired. A developer who unknowingly uses the default JVM keystore might potentially hit an issue when the Root certificate expires at some point in the future.

3. Documentation on more complex features

It was a bit challenging when we tried using some of the more complex features of the adapters like referenced entity lookup or upsert. It is not so straightforward to understand how to design/configure accordingly based on the documentation. Anyway, documentation is always a challenge, not just with Advantco, but even Microsoft and SAP. Fortunately, Advantco’s support is just an email away and is able to guide us through some of these challenges.

Challenges

At the end of the day, we managed to achieve the integration with Dynamics CRM, but not without challenges along the way. The challenges are not so much related to the adapter but because of the nature of this “new type” of integration. Here are some of them for the benefit of others who might come across such requirement in the future.

On-premise ADFS-based authentication requires TLS 1.2 library with SNI support

Our authentication method is with an on-premise ADFS server. We encountered “Stream closed” error when trying to establish connectivity with Dynamics CRM based on this authentication method.

From the installation guide, this requires SNI support provided by the new TLS 1.2 library that is mentioned in SAP Note 2284059 (also mentioned in my previous blog Supported TLS versions for outbound connections on Java stack).

This required us to:-

i) Install the new library

ii) Add a new custom SSL config file to

iii) Add extension=server_name.noncritical in the custom SSL config file

Upsert operation only applicable for parent entity, and not related/referenced entities

Our requirement was to upsert records for a parent entity and a couple of its related entities. This posed challenges as we hit errors. In the end, through trial-and-error we found out that upsert only works for a parent entity, therefore we split our interface to perform upsert into multiple entities separately, i.e. instead of 1 upsert call to 1 parent entity and 2 related entities, perform 3 upsert calls.

Limitation of Microsoft API documentation

As I mentioned above, documentation was a challenge, and it was not easy/possible to understand the Dynamics CRM’s APIs as the documentation is written from a Microsoft .Net perspective.

Conclusion

Advantco has nailed it again with this solid and innovative offering that fills the gap for such type of integration requirements. The features of the adapter are very developer-friendly, which gives me the impression that they are a company “of developers, for developers” 🙂

Please reach out to our sales team at sales@advantco.com if you have any questions.