Hello! Every now and then, a JIRA ticket comes in form of learning angel. This bug was exactly that, it needed detective work, history exploration, research and that very special aha moment. I have love hate relationship with such pesky bugs. If I am successful, I get the bragging rights and if I am not I am signing up for sleepless nights. Luckily, this case was a successful one. 🙂
Here it goes. It started with the client email suggesting they are seeing drop in specific values of form submission data starting end of last year.
First step: Replication -> Nope, could not replicate this loss of data on dev, QA or UAT.
Second step: Data Analysis -> Asked client to provide the data from end of last year to now to see if I can find a pattern. Bingo – Yes, all the data complained about missing belonged to session. Okay, some direction.
Third step: Questions/All sort of them -> Why are session values missing? Actually, when does session values go missing. Answers below :
Session was never stored properly to begin with, so, when tried to access no values were present.
Session was stored properly, but, when tried to access it was empty. This usually happens when session is expired.
Fourth step: Debug -> Add tons of logs and see what is going on with every step. Session was always loaded fine when business requirement was met, so, ignore reason #1 noted above. When debugging, most of the time, session access was also good. But, weirdly it is missing in some cases. Bingo -> Replication is now successful. Only took me to do at least 15 trials before I hit the gold.
Fifth step: Look deeply on those scenarios of replication. Looked enough, spent almost half day, had some food to think better. But, nope, I am unable to see a reason. Failed!
Sixth step: Logs are your best friend! Took a deep dive on logs, checked all errors and warnings. Seemed harmless. Oh wait, what is this? A 404 on VisitorIdentificationCSS.aspx? oh wait, there is one more 404 on VIChecker.aspx. Could this do something to session? I found some stack overflow post (unable to find the link now) that suggests session does expire when visitor identification fails under 1 minute because Sitecore will think the end user is a bot based on default configuration here.
Seventh Step: Celebrate, but, with caution. First thing first, I ensured 404 is taken care of, it was due to a global IIS rewrite rule that was removing aspx extension on every incoming request. Poor visitor identification! This also sounds alarms as to why Sitecore still uses aspx stuff? we should truly make Sitecore internal stuff MVC style by default. Seems pretty outdated at the moment. Anyway, once 404’s are fixed and requests for visitor identification were happy. I was confident that my theory is correct, but, I have to prove it before hand off to QA. I tried 20 attempts, each min accounted for one trial. On those 20 trials, I did not see even one missing Session value. Hence proved!
I was sooo happy that I wanted to open a wine bottle, but, it was not dinner time just yet, so, adjusted with ice cold nitro brew from Starbucks. Enjoyed some bragging rights by sharing the hypothesis with clients, internal arch team and who ever pinged me that day or hopped on a call had to hear partial story lol, poor them. Work from home downside to not be able to share with some one in real. I miss people. 🙁
I took my first steps back in February this year with a mission to connect two amazing systems together – Sitecore and Salesforce. Little did I know back then that I will learn so much, get to break so many walls and reach the final(for now) data destination we had on our mind. It was a fun ride! Not only did I learn about the Sitecore Connect for Salesforce CRM and the scope of what we can do with it, but, also learned a ton of other things with Sitecore xConnect, Analytics, xDB SOLR indexes, custom contact facets, Sitecore Forms, Custom collections and models, Salesforce leads, objects and fields in Salesforce, Data Exchange Framework, custom readers and converters, pipeline batches and scheduled tasks. I mean I can go on and on and on. It was awesome and yet quite painful when things would not work. But, hey, I am here writing this last post on this series that covered how and why I did what I did. I made every effort to reference the blogs/links that inspired and helped me solve a problem on hand. But, I also managed to figure some kinks myself and I documented those to potentially help some one else who is amidst of implementing their data journey between the two systems.
Below are links to all of my blog posts that are part of this series. Hoping this would help any one facing similar challenges or potentially kicked off their journey stitching the two most powerful platforms out there – Sitecore and Salesforce.
I was also able to connect with Salesforce Marketing Cloud and do some extensions up there as well. I will talk about that side of things in my next blog post. Until then, take it easy. 🙂
We managed to ship the whole deal of data flow noted so far in my blog posts related to this journey. When you deploy a new system, the journey does not end so soon. Even if you did your best while implementing, testing and deploying, their could still be unknowns which we will need to keep an eye for and solve to ensure the data continues to flow smoothly between the systems. We had couple of such issues that we needed to battle. Gist of problems and corresponding solutions below:
Problem – ‘Contacts stopped syncing to Salesforce’: One not so fine day, the connector broke and contacts did not sync to Salesforce.
Debugging & Solution – This needed a lot of debugging as logs were not helpful. Below is the message on pipeline batch log that is responsible to sync contacts from Sitecore to Salesforce. Lets look at the error
ManagedPoolThread #4 20:05:04 DEBUG [Data Exchange] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Sitecore.DataExchange.Providers.Salesforce.Queues.BaseSubmitQueuePipelineProcessor.SubmitBatch(PipelineStep pipelineStep, PipelineContext pipelineContext, OperationType operationType, List`1 inputList, ILogger logger)
ManagedPoolThread #4 20:05:04 ERROR [Data Exchange] InvalidBatch
ManagedPoolThread #4 20:05:04 DEBUG [Data Exchange] at Salesforce.Common.XmlHttpClient.<HttpGetAsync>d__4`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Salesforce.Common.XmlHttpClient.<HttpGetAsync>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Salesforce.Force.ForceClient.<GetBatchResultAsync>d__52.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Salesforce.Force.ForceClient.<GetBatchResultAsync>d__4f.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Salesforce.Force.ForceClient.<RunJobAndPollAsync>d__1e`1.MoveNext()
First look at error, Second look at error, I mean looked at it for like two hundred times, I have no clue why it would break. Log entries were not helpful at all in this case. Since all was well couple days ago, it gives a subtle hint that it could be data. I made reports from Experience Analytics and exported all the data in to excel from the day contact syncing had stopped and started looking for anything offbeat. I found the below two entries with few special characters, I had a hunch that that might be causing issues with pipeline batch. I removed the specific field that had special characters from value mappings and ran the pipeline again and boom, the contacts started syncing back again. Also, identified the special character that the process had problem with is ‘&’.
Now, we can not remove the mapping permanently, we need to find a solution to avoid this situation on any other field mappings and data collected on xConnect contact in general. To do this, we need a new reader and converter that do exactly what you are guessing – replace our bad character with space. We would also need a template to make this happen. Below are key components needed for this solution:
Template that has place to enter the character and points to custom reader defined on code.
New Template for reader that accepts character of concernStandard Values of new template which has default character and Converter type that points to custom namespace where model resides and dll corresponding to the same
Next is to actually define the Converter and Reader. I included the code for both below
using Sitecore.DataExchange.DataAccess;
using Sitecore.DependencyInjection;
using Sitecore.Marketing.Definitions;
using Sitecore.XConnect;
using System;
using System.Globalization;
namespace yournamespace
{
public class ReplaceCharacterValueReader : IValueReader
{
public string SpecialCharacter { get; protected set; }
public ReplaceCharacterValueReader(string character)
{
if (string.IsNullOrEmpty(character))
throw new ArgumentOutOfRangeException(nameof(character), (object)character, "Empty Character");
this.SpecialCharacter = character;
}
public virtual ReadResult Read(object source, DataAccessContext context)
{
if (!(source is string str))
return ReadResult.NegativeResult(DateTime.Now);
return string.IsNullOrWhiteSpace(str) || !str.Contains(SpecialCharacter) ? ReadResult.PositiveResult(source, DateTime.Now) : ReadResult.PositiveResult((object)str.Replace(SpecialCharacter,string.Empty), DateTime.Now);
}
}
}
using Sitecore.DataExchange;
using Sitecore.DataExchange.Attributes;
using Sitecore.DataExchange.Converters;
using Sitecore.DataExchange.DataAccess;
using Sitecore.DataExchange.Repositories;
using Sitecore.Services.Core.Model;
namespace yournamespace
{
[SupportedIds(new string[] { "{7ABD90E0-5FC4-4547-8962-C0094F8A7CF7}" })]
public class ReplaceCharacterValueReaderConverter : BaseItemModelConverter<IValueReader>
{
public const string FieldNameCharacter = "Character";
public ReplaceCharacterValueReaderConverter(IItemModelRepository repository)
: base(repository)
{
}
protected override ConvertResult<IValueReader> ConvertSupportedItem(
ItemModel source)
{
return this.PositiveResult((IValueReader)new ReplaceCharacterValueReader(this.GetStringValue(source, FieldNameCharacter)));
}
}
}
Last piece to the puzzle is to actually use a reader created based on template above as source value transformer on value mappings which could potentially have this special character or where applicable. In our case ‘Title’ field could definitely lead to this character. So, we ensured we plugged above in for this value mapping for sure.
Source Value Transformer injected on mapping of concern
That is it, we finally do not have to worry about this pesky little character stopping the data flow.
Problem: I noticed an issue on Experience Profile where on contacts instead of seeing ‘Preferred’ on Contact Email address, it started showing $name. Like below:
Debugging and Solution: It was important in this case to understand when does the issue happen. Does it happen when we submit a form and load xConnect contact information? or does it happen later when sync job runs. In our case, it was happening when pipeline batch that is responsible to sync data from Salesforce to Sitecore runs. Now that we know this, we have to investigate value mappings and value accessors responsible for this. Upon investigating in that path, below is what I see set on this item /sitecore/system/Data Exchange/my tenant/Value Mapping Sets/Salesforce to xConnect Contact Mappings/Salesforce Contact to xConnect Contact Emails Facet/Preferred Key. This definitely did not seem right, see screenshot below:
Incorrect mapping on Preferred Key
Swapped the above with a new reader created which will constantly return a string called ‘Preferred’ which is what we need. This did the trick and we did not see the ‘$name’ issue any more.
Swapped with new constant value reader that will always return string prefferred
In my last blog, we talked about how we could extend xConnect contact facet with additional information and add details to it when a user submits a form for instance. We can check to ensure the data is saved properly by running few quick SQL commands like below.
SELECT TOP (1000) [ContactId]
,[FacetKey]
,[LastModified]
,[ConcurrencyToken]
,[FacetData]
FROM [notimpossible_Xdb.Collection.Shard0].[xdb_collection].[ContactFacets]
where FacetKey ='SalesforceAccount'
You should see a string in FacetData similar to below. It should match definition of custom facet that was deployed. In my case, it looked like:
Though through above you can check to see if the information you would like is being stored on contact. But, a more elegant way would be to actually see it on a special tab on Experience profile. There are couple blogs out there that can help you with this especially if the goal is to use Speak. But, if you do not care about using Speak, do check out this one. In my case, I wanted to stick with Speak to see how far I could go with this. It was quite a number of steps, but, a combination of resources and special steps helped me get there.
References that helped me:
https://www.konabos.com/blog/extending-sitecore-experience-profile-in-sitecore-9 -> This helped me get started and keep moving in right path till I reached this below. The link noted in there was a lot of information to digest and understand how to proceed next. I wish that area was expounded probably on another blog post. It would have probably helped me knock this down continuing in the same path.
Step where I stumbled and did not know how to proceed
Step #4 on this blog gave me an idea to duplicate the tab and adjust presentation using Sitecore rocks as needed instead of creating brand new items and scramble what should be added to presentation. One gap in this blog which was not mentioned is – when you duplicate the Tab Item, ensure you change references in presentation on new tab item to point to appropriate internal items within the new tab item. For example, below needed to be swapped to ensure I do not break existing details tab and to ensure I show values that I would need to show on my new tab.
Change the above highlighted to self reference the newly created TabChange ID here to new TabID. If you leave it to old one, your existing details tab could potentially not work. 🙂On LoadOnDemandPanel, ensure the itemid matches the ID of DetailsPanel under newly created tab. This should be swapped using Sitecore Rocks. Change Target Control ID as well to ID of Border given on presentation.
Remember to change ID’s on all presentation components on Tab item. This is to ensure no overlap with existing Details Tab item. If I did not do above, my original details tab was broken and was not working.
Now that we finished changing presentation of main Tab item, lets move to Details Panel Item which is also based on Tab Template -‘/sitecore/client/Business Component Library/version 1/Templates/Common/Tab’. Open this item up on Sitecore rocks and play with presentation to update/inject new items. Also, ensure to add new datasource items and swap existing references to newly created items under details panel. Finally, my presentation looked like below.
Highlighted ones are actually new additional fields that I would like to show on my Additional Information Tab on Experience Profile.
Do not forget to add new label data sources needed. I took inspiration as to where the existing ones are and dropped my news ones in the path below: ‘/sitecore/client/Applications/ExperienceProfile/Common/System/Texts’
I added additional css needed for this tab by adding a new style sheet item as shown below
Additional css needed for this tab and new elements
Finally, last step is to hone our js that is needed to make the right calls and read/display from JSON data. To configure which JS to use, edit the Subcode presentation component’s details.
Add file path to shiny new JS
//Here is JS Code for my new tab
define(["sitecore", "/-/speak/v1/experienceprofile/DataProviderHelper.js", "/-/speak/v1/experienceprofile/CintelUtl.js"], function (sc, providerHelper, cintelUtil) {
var intelPath = "/intel",
dataSetProperty = "dataSet";
var cidParam = "cid";
var intelPath = "/intel";
var getTypeValue = function (preffered, all) {
if (preffered.Key) {
return { Key: preffered.Key, Value: preffered.Value };
} else if (all.length > 0) {
return { Key: all[0].Key, Value: all[0].Value };
}
return null;
};
var app = sc.Definitions.App.extend({
initialized: function () {
var transformers = $.map(
[
"default"
], function (tableName) {
return { urlKey: intelPath + "/" + tableName + "?", headerValue: tableName };
});
providerHelper.setupHeaders(transformers);
providerHelper.addDefaultTransformerKey();
this.setupContactDetail();
this.setupContactAdditionalDetails();
},
setEmail: function (textControl, email) {
if (email && email.indexOf("@") > -1) {
cintelUtil.setText(textControl, "", true);
textControl.viewModel.$el.html('<a href="mailto:' + email + '">' + email + '</a>');
} else {
cintelUtil.setText(textControl, email, true);
}
},
setupContactAdditionalDetails: function () {
var contactId = cintelUtil.getQueryParam(cidParam);
var tableName = "additionalcontactinfo";
var baseUrl = "/sitecore/api/ao/v1/contacts/" + contactId + "/intel/" + tableName;
providerHelper.initProvider(this.ContactDetailsDataProvider,
tableName,
baseUrl,
this.DetailsTabMessageBar);
providerHelper.getData(this.ContactDetailsDataProvider,
$.proxy(function (jsonData) {
if (jsonData.data.dataSet != null && jsonData.data.dataSet.additionalcontactinfo.length > 0) {
// Data present set value content
var dataSet = jsonData.data.dataSet.additionalcontactinfo[0];
cintelUtil.setText(this.OrganizationValue, dataSet.Organization, false);
cintelUtil.setText(this.IndustryValue, dataSet.Industry, false);
cintelUtil.setText(this.WebsiteValue, dataSet.Website, false);
cintelUtil.setText(this.InvolvementValue, dataSet.Involvement, false);
cintelUtil.setText(this.SelectedCityValue, dataSet.SelectedCity, false);
cintelUtil.setText(this.ProposedCityValue, dataSet.ProposedCity, false);
cintelUtil.setText(this.AdditionalDetailsValue, dataSet.AdditionalDetails, false);
}
}, this));
},
setupContactDetail: function () {
var getFullAddress = function (data) {
var addressParts = [
data.streetLine1,
data.streetLine2,
data.streetLine3,
data.streetLine4,
data.city,
data.country,
data.postalCode
];
addressParts = $.map(addressParts, function (val) { return val ? val : null; });
return addressParts.join(", ");
};
providerHelper.initProvider(this.ContactDetailsDataProvider, "", sc.Contact.baseUrl, this.DetailsTabMessageBar);
providerHelper.getData(
this.ContactDetailsDataProvider,
$.proxy(function (jsonData) {
this.ContactDetailsDataProvider.set(dataSetProperty, jsonData);
var dataSet = this.ContactDetailsDataProvider.get(dataSetProperty);
var email = getTypeValue(jsonData.preferredEmailAddress, dataSet.emailAddresses);
if (jsonData.emailAddresses.length === 0 && email != null)
jsonData.emailAddresses.push(email);
var phone = getTypeValue(jsonData.preferredPhoneNumber, dataSet.phoneNumbers);
if (jsonData.phoneNumbers.length === 0 && phone != null)
jsonData.phoneNumbers.push(phone);
var address = getTypeValue(jsonData.preferredAddress, dataSet.addresses);
if (jsonData.addresses.length === 0 && address != null)
jsonData.addresses.push(address);
this.EmailColumnDataRepeater.viewModel.addData(jsonData.emailAddresses);
this.PhoneColumnDataRepeater.viewModel.addData(jsonData.phoneNumbers);
this.AddressColumnDataRepeater.viewModel.addData(jsonData.addresses);
cintelUtil.setText(this.FirstNameValue, jsonData.firstName, false);
cintelUtil.setText(this.MiddleNameValue, jsonData.middleName, false);
cintelUtil.setText(this.LastNameValue, jsonData.surName, false);
cintelUtil.setTitle(this.FirstNameValue, jsonData.firstName);
cintelUtil.setTitle(this.MiddleNameValue, jsonData.middleName);
cintelUtil.setTitle(this.LastNameValue, jsonData.surName);
cintelUtil.setText(this.TitleValue, jsonData.jobTitle, false);
cintelUtil.setText(this.GenderValue, jsonData.gender, false);
cintelUtil.setText(this.BirthdayValue, jsonData.formattedBirthDate, false);
if (email) {
cintelUtil.setText(this.PrimeEmailType, email.Key, true);
this.setEmail(this.PrimeEmailValue, email.Value.SmtpAddress);
cintelUtil.setTitle(this.PrimeEmailValue, email.Value.SmtpAddress);
}
if (phone) {
cintelUtil.setText(this.PrimePhoneType, phone.Key, true);
cintelUtil.setText(this.PrimePhoneValue, cintelUtil.getFullTelephone(phone.Value), true);
}
if (address) {
cintelUtil.setText(this.PrimeAddressType, address.Key, true);
cintelUtil.setText(this.PrimeAddressValue, getFullAddress(address.Value), true);
}
}, this)
);
this.EmailColumnDataRepeater.on("subAppLoaded", function (args) {
cintelUtil.setText(args.app.Type, args.data.Key, true);
this.setEmail(args.app.Value, args.data.Value.SmtpAddress);
cintelUtil.setTitle(args.app.Value, args.data.Value.SmtpAddress);
}, this);
this.PhoneColumnDataRepeater.on("subAppLoaded", function (args) {
cintelUtil.setText(args.app.Type, args.data.Key, true);
cintelUtil.setText(args.app.Value, cintelUtil.getFullTelephone(args.data.Value), true);
}, this);
this.AddressColumnDataRepeater.on("subAppLoaded", function (args) {
cintelUtil.setText(args.app.Type, args.data.Key, true);
cintelUtil.setText(args.app.Value, getFullAddress(args.data.Value), true);
}, this);
}
});
return app;
});
Only difference in code between the above and out of the box details tab is the method this.setupContactAdditionalDetails(). This function is wired up to call newly configured end point with tablename ‘additionalcontactinfo’ This is essentially the specific name you have given in your configuration element noted below.
That is it, if all goes well and you did the configuration and steps mentioned above correctly you should see the new tab loaded with information you need.
New Tab showing all the additional information
That is it, now I don’t have to open SQL Manager or run a query to check and ensure the data is updated fine in Sitecore when form is submitted with all data or when Salesforce is updated and synced back to Sitecore. Happy and Productive!
Next up, few hiccups post live and how we solved it.
By nature, I am extra cautious when I take my first steps on any project. This comes from experience, when a task is in front of you, many of us get tempted to jump and start working things out. I have a different take on this and even if I do burn couple of hours in the beginning, I would not change this style for no one or nothing! When you take your first steps right, when walls or falls happen, you are confident that you can get through them because you know that you are on right path.
So, on one of our projects, we decided that a tight integration with Salesforce would be a huge add on for this growing team. Now, the decision is clear that integration is needed, now, few key first decisions that should be made to ensure we will follow the right track to achieve our key goals.
Key Goal: We have a bunch of forms on the website that collect variety of information from end user based on purpose of the form. The stakeholders wanted to ship this collected information to Salesforce.
That’s a very abstract gist, I know. Not always would you have a dedicated Business Analyst that would document every single detail. Sometimes, you have to work with just a skeleton. Now, lets jot down the first set of questions:
What Connector should we use to ensure we have the tight integration between Sitecore and Salesforce?
What version of Connector should we be using?
What does this Connector do on a very abstract level and would it help us reach the key goal noted above?
Now, lets answer each one of those first before we jump deep and go crazy. Which we will trust me. I did run across quite a few walls and was able to successfully break or crack enough to get by, I will share that side of journey with you all as well.
But, first things first, answer to above key questions. So, for the first one, my boss, Liz Spranzani has a great collection of blogs which highlights Connectors that are available and which ones to use in which scenarios. Salesforce offers a whole stack of products and so does Sitecore, so, it is important to get your thoughts straightened out, so, you make wise and correct decisions.
Here are the blogs for your reference. I made my choice to be to use Sitecore Connect for Salesforce CRM because that fits all my boxes based on subscriptions the client has in regards to Salesforce. Now, your answer could be different, I am hoping its the same, so, my journey will help you. 🙂
Now, for the second one, folks who know Sitecore know that this question is very important to be answered because of just the way Sitecore product life cycle has been. To confirm this, I checked the compatibility KB article, but, unfortunately, it is not always up-to-date. So, I actually loaded a support ticket to understand what version of connector should I be using given my Sitecore version, got confirmation that I should be using: Sitecore Connect for Salesforce CRM 2.1.0 . Bottom line is latest and not always the right approach especially when you are dealing with multiple systems.
So, on a very abstract level the connector gives you ability to push various objects from Sitecore to Salesforce and vice versa. Most important one of course that most of us will be interested in would be Contacts. But, along side of Contacts, it can also ship Tasks, Events and Campaigns.
You nailed your first steps, Now, what to do next? Installation! Sitecore does have good documentation on Connector Installation and setup, but, like always there is few gaps and some helpful tips to get you over few humps. I will cover those in my next blog.
Do not underestimate the power of making right choices, take your own time and do not let anyone make them for you if you are the one who is going to own the implementation.
I saw a lot of blogs and bumped in to online links about an issue below, but, turns out every case is unique based on setup. The issue I was facing, I could not find a solution out there, but, yeah I cracked it. What does that mean? Yeah, you guessed it right, blog it, so, some one else can save their half day and enjoy a nature walk or something. 😉
So, I was trying to hook up Sitecore and Salesforce together, I will get another blog rolling soon on my learning, frustrations, happy moments noting that experience. But, for this one here, just important to know one of the prerequisites for shipping contacts from Sitecore to Salesforce, first I need to ensure I can see the contacts on Experience profile. All was well on my local Sitecore installation which for reference is version 9.1.1
But…. Experience profile was broken to say the least on upper environments. On Experience Profile UI, you will see something like below. Booooo, not helpful at all!!!!!! Sitecore really need to display some meaningful errors here especially if it is non production license.
Once I got in to servers and starting checking logs, I could see below errors
Exception System.NullReferenceException: Object reference not set to an instance of an object. at Sitecore.Cintel.Endpoint.Plumbing.NegotiateLanguageFilter.OnActionExecuted(HttpActionExecutedContext actionExecutedContext) at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutedAsync(HttpActionExecutedContext actionExecutedContext, CancellationToken cancellationToken) — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
Now, the above error had so many interpretations on web, but, led me no where , I then looked at XConnect logs to see if I can find deeper issue if any and I find the below error on XConnect logs
at Sitecore.Xdb.Collection.Search.Solr.SolrClient.EnsureSolrSuccessStatusCode(HttpResponseMessage response) at Sitecore.Xdb.Collection.Search.Solr.SolrReader.<ExecuteQuery>d__14`1.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.Search.Solr.SolrReader.<GetSearchResults>d__13`2.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.Search.Solr.SolrReader.<SearchContacts>d__10.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.Repository.<SearchContacts>d__11.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.RepositoryCountersDecorator.<SearchContacts>d__8.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.XConnect.Service.RepositorySearchInvoker.<Execute>d__7.MoveNext() 2020-01-29 13:51:40.269 -05:00 [Error] [“XdbContextLoggingPlugin”] XdbContext Batch Execution Exception Sitecore.Xdb.Collection.Search.Solr.Failures.SolrResponseException: { “responseHeader”:{ “status”:400, “QTime”:12, “params”:{ “fl”:”id”, “cursorMark”:”*”, “json”:”{\”query\”:\”(x_type_s:(\\\”ContactDataRecord\\\”) AND *:*)\”,\”sort\”:\”facets.engagementmeasures.mostrecentinteractionstartdatetime_dt desc,id asc\”}”, “rows”:”20″, “wt”:”json”}}, “error”:{ “metadata”:[ “error-class”,”org.apache.solr.common.SolrException”, “root-error-class”,”org.apache.solr.common.SolrException”], “msg”:”sort param field can’t be found: id”, “code”:400}}
at Sitecore.Xdb.Collection.Search.Solr.SolrClient.EnsureSolrSuccessStatusCode(HttpResponseMessage response) at Sitecore.Xdb.Collection.Search.Solr.SolrReader.<ExecuteQuery>d__14`1.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.Search.Solr.SolrReader.<GetSearchResults>d__13`2.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.Search.Solr.SolrReader.<SearchContacts>d__10.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.Repository.<SearchContacts>d__11.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.Xdb.Collection.RepositoryCountersDecorator.<SearchContacts>d__8.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Sitecore.XConnect.Service.RepositorySearchInvoker.<Execute>d__7.MoveNext()
Below is the order of things I did to see if that will fix my issue, obviously no luck!!!
Ensured XConnect url is working properly that was provided on Connection strings
Rebuilt all indexes from Control Panel , ran successfully
Ensured SOLR url given on connection strings config works
Ensured NewtonSoft.Json.dll has correct version per Sitecore expectation
Restarted all Sitecore Windows Services + App pool recycle on CM and XConnect site
Also tried deploying all my marketing panel stuff – why not right? lol
Almost gave up, that is when I usually create a support ticket and wait for couple days before re-working on the issue. I did do that, but, in this case, I was having some more energy left, so, I kept debugging and kept digging- Paid real close attention to solr error above, I saw that the json was: “json”:”{\”query\”:\”(x_type_s:(\\\”ContactDataRecord\\\”) AND *:*)\”,\”sort\”:\”facets.engagementmeasures.mostrecentinteractionstartdatetime_dt desc,id asc\”}”, I fired up my solr console and entered this breakdown on query window and yep I could replicate the error. Now, I changed the sort field which was the one Solr did not like to _uniqueid and it worked. Why would it work with _uniqueid and not with id? Ringing bells? It did for me and for refresh of memory, I pulled open Sitecore documentation for my version on Solr setup instructions here It says below – “repeat on all indexes”
But, XConnect code seems to fire Solr query with sort param as id, so, is xonnect UI incorrect? I was not sure and needed to confirm if Sitecore does not want us to do step #3 noted above on XDB indexes. So, I ran populate managed schema on my local and guess what all indexes have unique id modified to _uniqueid except for xdb related indexes. I mean this was still theory, I needed confirmation from Sitecore and my colleague bumped in to another set of documentation where there was NOTE, I could not believe my eyes when I saw that. So, yeah, my theory was correct and I had to login in to my not so favorite SSH Linux Solr server and hand edit the managed-schema file under both xdb and xdb_rebuild conf locations to have unique key as id and also replaced the settings line on schema to have id instead of _uniqueid to ensure all is swapped back. The NOTE that blew my mind and also comforted that I am on right path, screenshot below. May be Sitecore has to update documentation on the other spot to add this *****Important**** note to ensure folks do not change this on all indexes.
Now, once I did that and restarted Solr, Experience Profile is happy, there are no records just yet, but, no error either. yayyyy!!! On to my next challenge now. 🙂
First things first! Do we need a CDN at all when Sitecore is your platform of choice. Not really, you can actually do pretty awesome good in terms of performance with good caching, scheduled publishing, uploaded good imagery and rock solid code with few best practices followed. But, what if you have to use a CDN? There will be various situations in which you may have to due to business reasons or may be the audience that is very spread out across the globe or if the site is down right very media heavy.
Various questions pop up when folks bring up CDN. Where exactly is this going to sit in the bigger picture? What is the plan to use for CDN for? What do we do on Sitecore to make this happen? Are the assets on CDN? What happens behind the scenes? Now, it is more than likely that most of us already know answers for this. So, I will not bore you all explaining what are questions that need to addressed and understood when thinking about CDN with Sitecore.
Instead, what I would like to do is provide a quick comparison cheat sheet based on various features and capabilities when you have made your decision to use CDN with Sitecore. Recently, I was researching and finding information about what is the best way to integrate CDN with Sitecore in most efficient and easiest way possible. Couple years ago, I remember that integrating with CDN was still considered considerable amount of work. But, with Sitecore 9.1+ it should be real simple if of course the basic OOTB functionality that around CDN is enough for your business needs. I also explored one more option which is not bad either if you have time to make the legacy code work with newer versions of Sitecore.
Sitecore 9.1 Support – Unknown, but, some of our team suggested they did make this work on 8.2 and hence conceptually with that fix that we have to get this working on 8.x should probably and hypothetically work for 9.1 as well. I would recommend a POC if you are going to use the connector. As you can see from source url, the code base is very outdated almost 6 years ago. But, turns out with few fixes it may still work and does the job well. 🙂
Media Versioning – Yes. it does stamp the media with an incremental value based on date of publish which would help knock cache and have good versioning in play
File Versioning – yes. Based on documentation, it does support file versioning for static assets as well. But, this is something I highly recommend testing quickly before confirming that in fact it does this fine.
One important thing to note which was pointed out by my peer is that the connector actually checks to see when to load a resource from CDN vs when not to. For instance, if there are goals set up on a download of specific pdf for instance, it should be loaded from Sitecore media library and not from CDN. The connector handles this, but, obviously the code might be subtly outdated. Easy to fix this though if not working and I totally think this should be done if personalization is actively used in your sitecore instance. You can check that additional logic here if curious. https://github.com/NTTDATA/SitecoreCDN/blob/master/Code/Providers/CDNMediaProvider.cs
Option 2 – Sitecore Media Library CDN
Sitecore 9.1 actually comes with a special package to enable CDN
Multi Site Support – No. If you pay attention to config file in the package that Sitecore provides you will see couple of settings that enable CDN use on media library. But, these are not site specific and are global. Hence, this does not support different settings for different sites.
Sitecore 9.1 support : obviously, Yes. 🙂
Media Versioning – Yes
File Versioning – No. You can see an added note of no such capability here, scroll to the bottom of the page. https://doc.sitecore.com/developers/91/sitecore-experience-management/en/sitecore-media-library-cdn-overview.html
Screenshot of settings for reference in CDN.config which is the only thing on the package provided on downloads section
Now, there is no wrong or right way. It depends on your requirements and what would be most efficient path to choose. It is great that Sitecore actually has some option now to plug and play and use CDN OOTB in some level. Only hope is that OOTB we get more settings and flexibility to choose CDN configuration that comes OOTB than to implement some thing to fit needs that has become a new normal these days. I am sure we will get there soon. But, for now, pick and choose. If these do not fit your needs, feel free to make something custom. Custom solution actually takes me back to really old versions of Sitecore that needed custom solution to fit in CDN in-between media library, server and end user.
I know, I am super late! It has just been a super crazy month on my end due to typical deadlines and crunch times, but, would like to post this before we get on to more awesome things in the coming future. The speed at which we are racing, within next two months, I believe Sitecore 9 and anything to do with it would be called essentials. 🙂
I had made a note on my last blog here that I will update every one on one unique caveat and many more known devils I had faced while getting the taste of latest and greatest. Now, below are the my n different things I had to do to see the beautiful lady on my favorite browser and oh boy! I hopped with joy when I finally did see her. She looked more gorgeous than ever and yes, I knew I was in the game when I got a glimpse of her.
While I was hitting so many problems and walls, there were so many blogs and findings that helped me debug each and move forward. Beautiful community!
Except for one issue that stood strong between me and my new toy, just in case if you run in to this, hopefully it will help you and save few pesky hours of debugging.
I used the most favorite and easiest way to get Solr up and running on my local by running a pre-made script noted here – https://gist.github.com/jermdavis/8d8a79f680505f1074153f02f70b9105
Had to get my SQL 2016 as that was stated as a pre-requisite and wanted to ensure I do not miss any thing
Also, ensured I get latest version of powershell by installing updates and patches on my windows. Those things you put off because you simply do not want to restart your computer. lol, I repented it like always.
Also, ensure you have everything noted up on pre-requisites in installation guide. Sitecore is not being funny and yep everything noted here is indeed a pre-requisite and is needed including modification of any registry entries to avoid some super weird SQL issues.
Now, after I did all this and followed through a lot of blogs, I still could not overcome one final error on my end on running the powershell script. I failed and tried again like I do not remember how many times and I also kind of lost count on how many hours I had spent googling if I missed anything else.
Below, was the error I was seeing on the powershell.
“Xconnect windows service fails to start”
My first thought based on many reads was my license file is not legit and is not authorized to be used with Sitecore 9.0+. And that is what I saw all over the web for an error such as above. But, after I heard from some one else who used exact same license as I did was successful in setting up the site, kind of huge burst of a bubble there, I could literally hear a pop. I went to event viewer and checked all the errors, I could see a message that license was accepted, so, that was not it.
Now, I was at verge of giving up as I was totally numb and super freakishly tired. I am obviously doing something very silly, just could not point what.
While browsing the community, forums, blogs for similar errors, I did find one clue that helped me realize what is it that xconnect was not liking and hence not starting causing the powershell to crash on me.
On this thread here, something caught my eye, though this was not the issue I was having on hand, it was worth the shot of running service manually to see if the error on command prompt does give some insight on what the issue is. So, I went to the location where exe is located on my inetpub and gave it a whirl. I saw an error message relating to format exception on Solr URL. Crazy right! I did check to ensure my solr url I had configured quite easily on my step #1 is correct. But, why does it not like it, so, when you configure Solr and try to browse to the url, it actually appends a hash to it by inherent redirect. That was the URL i had copied to my install script. It looked like – https://solr:8983/solr/#/
Apparently, xconnect service does not like the url of such format though it is actually a working Solr url. So, pay attention to your url of Solr on PS script. It should look like
https://solr:8983/solr
Once, I did this all was well and I could spin up a Sitecore 9.0+ site on my local.
Hope this helps some one else for easy way of debugging such type of issues!
This has been long due and would like to share my hackathon experience before the memory gets a tad bit weak. First things first, I must say that I handled the competition with way more ease than first time around. Who ever said it, yes, it is absolutely correct that “Experience does makes you perfect”
I tried to remind myself and my comrade multiple times that hackathon is only an implementation of scratch of the surface stating the future to explore further. If you loose this thought, you will find yourself being extremely fatigued and will end up doing bunch of mistakes and miss prerequisites.
The goal is to shoot for stars, but, also, ensure you have a clear picture on what you would call a deliverable. All in all it was fun and calm for the most part with brain juices flowing pretty well.
You can read about what we did and how we did it – up here.
In my next post, I wanted to share my Sitecore 9.1 installation experience with SIF tools and all the pitfalls I bumped in to, luckily there was a blog for most of them. Of course, I had one unique one which might help some, so, will post that soon.
Cya later!
PS: If you do not know what hackathon I am talking about, see the details below.
