13 March, 2012

Umbraco No Node Exists / cmscontentxml out of sync


For some time now we have had a problem when modifying document type alias' in umbraco (4.x.x), because it causes a complete flush of the XML caches used in umbraco. During these flushes of the caches if unexpected errors occurs, like DB connection lost, IIS timeout, AppPool flush or similar the cmscontentxml database table could become out-of-sync. This would lead to strange errors on the frontend of the sites in the installation:
  • No node exists with id 'xxxx'
  • Missing menu items
  • Macros throwing out errors
  • etc.
The problem has become almost a show-stopper for our umbraco installation as the number of nodes has increased, we are currently having 6000+ nodes and hosting 200+ websites on a single umbraco install. 

The errors are not showing up right after a change of the doctype alias, but after roughly 10-15 minutes, due to the thread updateing of the caches, and that umbraco will automaticly flush the cmscontentxml table to disk and memory caches if any of the two caches becomes invalid. Having "random" errors happening on 6000+ nodes is not acceptable.

So far we have "fixed" the issue by calling the http://myumbraco.com/Umbraco/dialogs/republish.aspx?xml=true which will force a full republish of all pages/nodes to the cmscontentxml table, and after that flush the disk and memory cache. The normal "right-click > republish" on the root node is only flushing the cmscontentxml table to disk and memory, but is not updating the database table, which is out-of-sync.

We have found several posts about the issue, but so far no good solution:
So we decided to come up with a proposal for a solution, that would ensure that regardless of any connectivity issues, or AppPool recycles, the cmscontentxml table would not get out of sync (this was the main reason for errors). Also modifying a DocumentType alias should NOT be a 10-15+ minutes wait-while-saving operation on large node umbraco installs.

Our solution modifies the DocumentType save event/method, by adding each node id that is using the DocumentType into the cmstask database table bycreating a new umbraco Task. These tasks are then loaded (one-by-one) using the umbraco.presentation.publishingService, and a flush of the database, disk  and memory cache for each node is done, and only after it is completed for each node the Task is deleted.
This should ensure that even a AppPool flush will just cause the publishingService to publish the node one more time.

A umbraco work item allready exists for this, see:
Also our patch is uploaded to it. (download patch)

####### December 20 - 2012 Update. #######
Due to the high number of requests, both here and from other sources, about this bug in Umbraco, I have created some additional release builds of the Umbraco Source with this patch applied, just click on the version you need the patched files for to download it. (both patch, dll's and source is available)
Umbraco Version 4.7.2
Umbraco Version 4.9.1
Umbraco Version 4.10.1
Umbraco Version 4.11.1

####### July 20 - 2013 Update. #######
Friday I received a distress request to help a company with a site that was down, and I spent my vacation-day-off helping them patching the Umbraco 4.11.10 version with this fix. I reviewed the code done by Shannon at Umbraco HQ, and although his code was a HUGH improvement, it was not a fix for this issue.
So here is the patch bundle for Umbraco, the dll's are built in release mode, but with pdb files.
Umbraco Version 4.11.10

####### August 2 - 2013 Update. #######
Be very careful about the latest 4.11.10 version of Umbraco and republish xml, there is a bug which will cause the sortOrder property on the nodes to be set to "0" in the output/cache XML. I'm currently testing a new patch and will release it after some testing.
see: http://issues.umbraco.org/issue/U4-2527#comment=67-8808
A quick google search on "umbraco sort order 0" reveals that it is a old bug, and is affecting many areas of Umbraco, media, content, basicly all objects that inherit from CMSNode

22 comments:

  1. Hi Allan
    I've been experiencing exactly the same issues.
    Your patch seems to be a great solution.
    How are your long term experiences with running the patch?
    Have you had any feedback from Umbraco core developers?

    ReplyDelete
    Replies
    1. Hi Henrik

      We implemented the patch on our live setup in January 2012, and have been running with it ever since, without a single failure. And the out-of-sync issue has not shown it self ever since.

      Our system has grown to 300+ sites with 20.000+ nodes since I posted the patch. And we have many more editors + developers but dispite of this no content-out-sync has happend.

      Please contact me if you need advice or help applying the patch.

      Delete
  2. Hi Allan,

    I'm noticing something like this in 4.9.1 install, but just on images (so far). I have not verified for myself that it is a docType edit that causes all the image references in cmsContentXml to disappear. I have worked a lot with a 4.0.4.2 version with over 100,000 nodes and never noticed anything like it. How do I install this patch? I'm not sure how to do it, I tried searching.

    ReplyDelete
    Replies
    1. Hi

      I have not tried the patch on a 4.9.1 install, so I'm not sure if the same problem with the doctypes apply to the 4.9.1 version of Umbraco.

      But based on my knowledge with Umbraco, they have not yet fixed the problem. (But I have not verified this).

      We are currently still running on the 4.7 version, with the patch described in this post.

      I can try and download the 4.9.1 version and see if the patch still applies, but if you can send me some details about how your bug is appearing (steps to reproduce) then I can test it on the 4.9.1 version.

      If the patch still works (and fixes the issue) I can send you a guide + a patched 4.9.1 version.

      Delete
    2. Hi

      I have made a patched release of the Umbraco 4.9.1 sources, download it here: http://bitly.com/bundles/o_742jg6tlpm/3

      Best regards
      Allan

      Delete
    3. Thanks. Sorry for the multiple posts(I can't see how to delete them.) My posts kept timing out and giving an error.

      Delete
  3. Hi Allan,

    How do you apply the patch?

    ReplyDelete
  4. Hi Allan,

    I think I might need this on a new 4.9.1 version I'm setting up. All my images are disappearing out of cmsContentXml and breaking stuff.

    How do I install/apply the patch?

    ReplyDelete
  5. Hi Allan,

    I think I might need this on a new 4.9.1 version I'm setting up. All my images are disappearing out of cmsContentXml and breaking stuff.

    How do I install/apply the patch?

    ReplyDelete
  6. Hi Allan,
    I have the exact same problem, also after renaming a docmentType. (umb 4.7.2)
    Do you happen to have the patched sourcecode files in a zip? (I do not exactly know how to use the .patch extension).
    Also: the link to codeplex is not alive anymore.
    Nico

    ReplyDelete
    Replies
    1. Hi Nico

      I uploaded a patched version of the 4.7.2 Umbraco source here : http://bitly.com/bundles/o_742jg6tlpm/2
      Also included is a zip file with only the 2 dll's modified build in release mode.

      Your welcome to contact me again if you need more help.

      /Allan

      Delete
    2. Thanks Allan,

      I did also receive response from Sebastiaan on this ticket: http://issues.umbraco.org/issue/U4-493

      We are trying to reproduce this issue on our dev environment (no luck so far) so we can test again using your patch. Seems we just are going to deploy your patch anyway...

      Cheers Nico

      Delete
  7. Hi Allen, News: we were able to reproduce the bug in our dev without using the patch. When the patch is applied, it seems to work smoothly. I also notified sebastiaan of Umbraco with this update
    Cheers, Nico

    ReplyDelete
    Replies
    1. Super, I'm glad to help a fellow Umbraco developer/user.

      I have done quite a few performance and bug improvements to our own Umbraco install, to fix issues om large scale installs. I will try and post them once I have the time to describe them in details.

      But if you need it have a look at : http://allan-laustsen.blogspot.dk/2012/07/fix-sorting-thousands-of-nodes-in.html

      /Allan

      Delete
  8. I just added a new release build for the Umbraco version 4.11.10, see also my comment for Shannon here: http://issues.umbraco.org/issue/U4-493#comment=67-8567

    ReplyDelete
  9. Hi Allan,

    I'm experiencing the same issue and the version we use is 4.7. Can you provide a guide on how to install the patch you provided above? My email address: harshavarma at gmail dot com.

    ReplyDelete
  10. Hi Allan,

    Sorry if this happens to be a duplicate post. I posted a comment this morning but it never appeared in the threa.

    We have a website that uses Umbraco 4.7 and it has the same issue [missing node]. I really appreciate it if you can explain how to install the patch that you created.

    ReplyDelete
  11. Hi Allan,

    Can you please provide instructions for how to use the patch file?

    Much thanks.

    ReplyDelete
  12. Hi Harshavarma and dreeft

    The patch files are created using TortoiseSVN, and each patch file generated on a clean umbraco source code of the specific version number.
    And you need to have TortoiseSVN installed.
    We use VisualSVN as a server, and it can be run on localhost, so that you can use it without a remote SVN server (http://www.visualsvn.com/server/)

    So to apply the patch:

    Step 1. Get the umbraco sourcecode for the specific version of umbraco you are using (http://umbraco.com/download/get-the-latest-source-code.aspx)

    Step 2. Unzip the source code to a folder that is under source control by SVN, and commit it to SVN.

    Step 3. Download the patch file I have created for the same version of Umbraco.

    Step 4. Right click the root source code folder of Umbraco and select "TortoiseSVN > Apply Patch", in the new "Apply patch navigation window" navigate to the location where you put the patch file I created, and select open.

    Step 5. Now select patch all items

    Step 6. After the patch has been applied, rebuild in Visual studio in release mode. Do a double clean-rebuild of all projects, since there are some circular referencing that can be messed up.

    That was it, please write to me if you need more help :-)

    ReplyDelete
  13. Hi Allan,

    I just upgraded an Umbraco installation to v4.11.10 and I'm coming up against the issue with all the nodes having a sortOrder attribute of "0" in the XML cache after a document is published.

    Did you ever build a patch for that issue? Or would I be better off upgrading to the latest 6.1.x release instead?

    Thanks for your help.

    Steve

    ReplyDelete
  14. Hi Allan,

    I have two sites which are still running Umbraco 4.7.2 and have the same issue. Would do you like to email the patch for 4.7.2?

    My email: tloveje@gmail.com

    Thank you,
    Tony

    ReplyDelete
    Replies
    1. Hi Tony

      The patch file is available here (the first file in the list):
      https://bitly.com/bundles/o_742jg6tlpm/2

      Also in the same bitly bucket the patched dll's and source files are available.

      Please contact me again if you need help patching your installation.

      Best regards
      Allan

      Delete