Solve hiccups with Sitecore Connect for Salesforce CRM

We managed to ship the whole deal of data flow noted so far in my blog posts related to this journey. When you deploy a new system, the journey does not end so soon. Even if you did your best while implementing, testing and deploying, their could still be unknowns which we will need to keep an eye for and solve to ensure the data continues to flow smoothly between the systems. We had couple of such issues that we needed to battle. Gist of problems and corresponding solutions below:

• Problem – ‘Contacts stopped syncing to Salesforce’: One not so fine day, the connector broke and contacts did not sync to Salesforce.
• Debugging & Solution –
  This needed a lot of debugging as logs were not helpful. Below is the message on pipeline batch log that is responsible to sync contacts from Sitecore to Salesforce. Lets look at the error

ManagedPoolThread #4 20:05:04 DEBUG [Data Exchange] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Sitecore.DataExchange.Providers.Salesforce.Queues.BaseSubmitQueuePipelineProcessor.SubmitBatch(PipelineStep pipelineStep, PipelineContext pipelineContext, OperationType operationType, List`1 inputList, ILogger logger)
ManagedPoolThread #4 20:05:04 ERROR [Data Exchange] InvalidBatch
ManagedPoolThread #4 20:05:04 DEBUG [Data Exchange] at Salesforce.Common.XmlHttpClient.<HttpGetAsync>d__4`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Salesforce.Common.XmlHttpClient.<HttpGetAsync>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Salesforce.Force.ForceClient.<GetBatchResultAsync>d__52.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Salesforce.Force.ForceClient.<GetBatchResultAsync>d__4f.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Salesforce.Force.ForceClient.<RunJobAndPollAsync>d__1e`1.MoveNext()

First look at error, Second look at error, I mean looked at it for like two hundred times, I have no clue why it would break. Log entries were not helpful at all in this case. Since all was well couple days ago, it gives a subtle hint that it could be data. I made reports from Experience Analytics and exported all the data in to excel from the day contact syncing had stopped and started looking for anything offbeat. I found the below two entries with few special characters, I had a hunch that that might be causing issues with pipeline batch. I removed the specific field that had special characters from value mappings and ran the pipeline again and boom, the contacts started syncing back again. Also, identified the special character that the process had problem with is ‘&’.

Now, we can not remove the mapping permanently, we need to find a solution to avoid this situation on any other field mappings and data collected on xConnect contact in general. To do this, we need a new reader and converter that do exactly what you are guessing – replace our bad character with space. We would also need a template to make this happen. Below are key components needed for this solution:

• Template that has place to enter the character and points to custom reader defined on code.
  

• Next is to actually define the Converter and Reader. I included the code for both below
  

using Sitecore.DataExchange.DataAccess;
using Sitecore.DependencyInjection;
using Sitecore.Marketing.Definitions;
using Sitecore.XConnect;
using System;
using System.Globalization;

namespace yournamespace
{
    public class ReplaceCharacterValueReader : IValueReader
    {

        public string SpecialCharacter { get; protected set; }

        public ReplaceCharacterValueReader(string character)
        {
            if (string.IsNullOrEmpty(character))
                throw new ArgumentOutOfRangeException(nameof(character), (object)character, "Empty Character");
            this.SpecialCharacter = character;
        }

        public virtual ReadResult Read(object source, DataAccessContext context)
        {
            if (!(source is string str))
                return ReadResult.NegativeResult(DateTime.Now);
            return string.IsNullOrWhiteSpace(str) || !str.Contains(SpecialCharacter) ? ReadResult.PositiveResult(source, DateTime.Now) : ReadResult.PositiveResult((object)str.Replace(SpecialCharacter,string.Empty), DateTime.Now);
        }
    }
}

using Sitecore.DataExchange;
using Sitecore.DataExchange.Attributes;
using Sitecore.DataExchange.Converters;
using Sitecore.DataExchange.DataAccess;
using Sitecore.DataExchange.Repositories;
using Sitecore.Services.Core.Model;

namespace yournamespace
{
    [SupportedIds(new string[] { "{7ABD90E0-5FC4-4547-8962-C0094F8A7CF7}" })]
    public class ReplaceCharacterValueReaderConverter : BaseItemModelConverter<IValueReader>
    {

        public const string FieldNameCharacter = "Character";

        public ReplaceCharacterValueReaderConverter(IItemModelRepository repository)
          : base(repository)
        {
        }

        protected override ConvertResult<IValueReader> ConvertSupportedItem(
          ItemModel source)
        {
            return this.PositiveResult((IValueReader)new ReplaceCharacterValueReader(this.GetStringValue(source, FieldNameCharacter)));
        }
    }
    }

• Last piece to the puzzle is to actually use a reader created based on template above as source value transformer on value mappings which could potentially have this special character or where applicable. In our case ‘Title’ field could definitely lead to this character. So, we ensured we plugged above in for this value mapping for sure.
  

That is it, we finally do not have to worry about this pesky little character stopping the data flow.

• Problem: I noticed an issue on Experience Profile where on contacts instead of seeing ‘Preferred’ on Contact Email address, it started showing $name. Like below:
  

• Debugging and Solution: It was important in this case to understand when does the issue happen. Does it happen when we submit a form and load xConnect contact information? or does it happen later when sync job runs. In our case, it was happening when pipeline batch that is responsible to sync data from Salesforce to Sitecore runs. Now that we know this, we have to investigate value mappings and value accessors responsible for this. Upon investigating in that path, below is what I see set on this item /sitecore/system/Data Exchange/my tenant/Value Mapping Sets/Salesforce to xConnect Contact Mappings/Salesforce Contact to xConnect Contact Emails Facet/Preferred Key. This definitely did not seem right, see screenshot below:
  

Swapped the above with a new reader created which will constantly return a string called ‘Preferred’ which is what we need. This did the trick and we did not see the ‘$name’ issue any more.