Deleting elements in a dimension and rebuilding them in TM1

Is there a restriction for using DimensionDeleteAllElements() in TM1 wherein it can't work in tandem with a dimension update process that's called from the TI which houses DimensionDeleteAllElements()?

I've a TI which deletes all elements of a dimension using DimensionDeleteAllElements() and subsequently rebuilds it by calling another TI process which updates the dimension with elements from the database. This serves to weed out unnecessary elements.

After successful execution of this TI, I can find that the elements are wiped out in the dimension. But the dimension fails to get rebuilt. However, according to the tm1server log the secondary TI that updates the dimension with database elements completes its execution normally. Also, running the dimension update TI manually works fine and updates the dimension with elements from the database.

Should I use the contents of the dimension update process here in this TI instead of calling that?

Solution

Let me state this plainly... you should emphatically NOT, under any circumstances, be doing what you are doing.

The general consensus amongst TM1 experts is that except in very, very exceptional cases (such as creating a reference dimension which is not used in any cubes), DimensionDeleteAllElements() is too dangerous to be used. (Example 1, Example 2.) If the TI process fails part way through you can lose your elements. Lose your elements, and you lose your data.

You haven't specified the tab on which you're making that call but let me explain how a metadata update (currently) works. (It works a bit differently with the new functions like DimensionElementInsertDirect, or the new Restful API which is stateless, but for the purposes of this exercise it still applies.)

Any changes that you make to a dimension in the Prolog or Metadata tabs will cause a copy of the dimension to be made in memory.
After the last row of the datasource (if any) is processed on the Metadata tab or, if there is no datasource, after the execution of the code passes through the Metadata tab on its way to the Prolog, the copie(s) of the changed dimensions will be checked for integrity and, if they pass that check, will be registered as replacements for the original dimension objects.

Before the second of those things happen, however, the rest of the system does not know about the copy of the dimension. They are similar to private objects that only the TI process itself knows about.

So what is happening in your case is this:

Your first process executes the DimensionDeleteAllElements command. This causes a copy of the dimension to be created and all of the elements to be removed.
I would guess that you are calling the second process on your Prolog tab. (I would hope it's not the Metadata tab, otherwise you'll be executing the call once for every row in your record source, if any.)
When that process is called it will do the rebuild of the dimension. It will do this by creating its own copy of the dimension in memory, quite separate from the first process' one, updating that copy, then registering it as the new dimension once it passes its own Metadata tab.
Control will then return to the Prolog of the first process which, you may remember, still has its own copy of the dimension in memory, one which now has no elements. Once the first process passes the end of its own Metadata tab, it will do the integrity check (an absence of elements does not cause that check to fail) and register that dimension copy as the updated dimension, thus obliterating (or more precisely overwriting) the changes that the second process made.

The solution? If you are going to be calling DimensionDeleteAllElements (and you generally shouldn't) then you must do it in the Prolog of the same process that rebuilds the dimension. In that way the element deletion and the re-addition of the elements from the data source happens to the same copy of the dimension, and the resulting dimension is then registered.

You should not ever be removing N or S elements that contain data in cubes. These should never be "unnecessary elements" to be "weeded out". Doing so can cause hard to explain changes in cube values (since data vanishes with the elements) which is toxic from an auditing point of view.

C level elements are a different matter. If your intent is to remove all of those and allow the current hierarchy to be rebuilt from the source, it would be best to just iterate through the dimension elements (backwards) using the DimSiz and DimNm functions, and using the DType function to return the element type so that you can identify and delete consolidations. This is obviously done in the Prolog.