Zugg Software :: View topic - Changing how CMUD stores object properties internally

Posted: Mon May 04, 2009 7:32 pm

I've gotten sidetracked in the next version a bit, but I think it's important...

When I was working on adding a "textcolor" property to button states (separate from the main button text color), I ran into an issue with how CMUD is currently handling properties of various setting objects (aliases, triggers, macros, buttons, etc).

In the PKG database, each setting object has a couple of fields for properties. The "opt" field is an integer field for various boolean properties. Each boolean property (like the various checkboxes for triggers, like Trigger on Trigger) are represented by a particular "bit" in this integer field, allowing for 32 boolean properties for each setting. This works fine.

Then there are fields "userval" and "userint" which allow a single string and single integer property for the setting. For example, the "userval" string is used to store the unique UID property for modules and "userint" stores the color value for button states (the background color).

When additional properties are needed, the "options" field is used to store the XML for any number of additional properties. Buttons and Triggers have a lot of additional properties defined in the "options" XML data.

The problem with the "options" XML data is that CMUD wasn't handling it very efficiently. It was parsing the XML and storing the internal DOM tree for the XML for only the currently selected database record. So each time a different record was selected, it was re-parsing the XML in the Options field.

When I started adding the TextColor property for button states, I decided that I really wanted to revamp this structure to make it much easier for me to add and support more properties in the future (like a Style for each button state, etc). So while the database storage of these properties will remain the same (for compatibility), I am working on a new caching system that will actually cache *all* of the "options" XML properties within each setting record in memory. While this will use more memory for your settings, it should give a nice speed boost, especially to the button refresh code (where CMUD loops through all buttons and redraws them on the screen) that currently does a lot more XML parsing than it needs.

The new code abstracts "properties" of settings a bit, so that I don't actually need to worry about whether a particular property is using the "opt" boolean field, or the "userval" and "userint" fields, or the "options" XML field. The new code just creates some property arrays for each setting and the internal details of which database field is used and what format is needed (XML for "options") is now hidden behind the scenes where I don't need to deal with it all the time. This really cleans up the CMUD code and should make it easier to support and add new features in the future.

Hopefully I haven't broken anything, but I will probably do another beta release before I finish adding all of the other new features and doing all of the other bug fixes towards the end of this week. This is such a low-level change in CMUD that I need to make sure I haven't caused any new bugs because of it.

I've been sick for the past couple of days (and "no", I'm pretty sure it's not the swine-flu, so don't worry). But I still hope to have something for you to play with by the end of the week.

Posted: Mon May 04, 2009 9:52 pm

Was this change related to how the button foreground text color was flashing between white and its normal color in the latest version?

Posted: Mon May 04, 2009 11:08 pm

Well, kind of. The slowness of the button update code was definitely making the flicker more noticeable. But improving the property storage code doesn't remove the flicker...just makes it a bit quicker. Getting rid of the flicker is still really a separate bug that I still hope to completely fix.

Posted: Tue May 05, 2009 6:49 pm

This change has a nice side effect. In Delphi, properties have "setters" and "getters" which are routines to set and get the property value. With the new property array, many of the property routines now look like this:

Posted: Tue May 05, 2009 7:58 pm

Hmm, thats a very nice side affect. 9 lines to 1 code reduction, probably only a 1000 different properties to edit, I wouldn't look forward to doing that with a clear head.

Measuring speed increase could be hard, button colors/options not so much, but special characters, loops of 100000 instead of 1000, and some.... command dummy if and set/retrieve preferences I geuss.

hmmmm choc... errr bad chocolate, bad!

Posted: Mon May 11, 2009 5:12 pm

Feeling better today, so now I'm going to tackle the conversion of all of those properties.

Fortunately, the Preferences will be easy. I already have another application that I wrote which auto-generates the Delphi code for the Preferences based upon a database. That's how I can add new preferences so easily. So all I need to do is update the code generation application and have it generate the new code.

But all of the setting properties (aliases, triggers, buttons, etc) I will have to modify manually.

Posted: Tue May 12, 2009 10:56 pm

I got through all of the properties today. Now I'm debugging all of the new problems that I added. Since properties are handled more uniformly now, fixing these bugs should be pretty easy and will ensure that certain properties don't get handled differently in the future (like having a property that doesn't update a button when it's changed).

I also started looking at the button refresh issue, and found the current code for handling setting refresh (buttons and status bar items) was really bad. There were many situations that would cause multiple updates.

In debugging this, I found a very annoying problem with the in-memory database that I'm using: it marks a field as "modified" even if you overwrite the field with the same value that it currently holds. This causes all sorts of problems because it makes CMUD think that certain fields have changed (like the properties) even when they haven't. This causes a lot of extra button refreshes.

So now I'm trying to determine exactly how to fix this problem. Obviously I want CMUD to check the existing field value before assigning a new value (and not perform the assignment if the value hasn't changed). But this means I cannot access the database fields directly (which is what I mostly do everywhere in CMUD) and will need to go through some sort of "helper method" that performs this check. So the trick is figuring out how to implement this "helper method" so that it has the minimum impact on existing code and requires the fewest changes.

In theory, I should be able to accomplish this with writing my own "getter" and "setter" for database fields. Then I point the database fields to my own getter and setter methods to override the normal methods. And I think I can use the same idea that I used for the properties by using the "index" value to indicate which database field I am getting or setting.

In other words, right now I have code that looks like this:

Posted: Thu May 14, 2009 10:25 pm

Well, I'm making good progress. I've got all of the property and database access refactoring completed and have done the first debugging pass on it. The nice thing about the new code is that everything is handled more consistently with less code, so testing is easier. A bug in the new routines effects *all* properties or data access, so they are pretty easy to track down.

I've also done the first pass at the button/statusbar refresh optimization, ensuring that they are only refreshed once when needed. I added some code to peek at the Windows message queue and remove any duplicate refresh messages for the same object, so changing multiple properties doesn't cause multiple refreshes. The only time there is an extra refresh is when editing something via the Settings Editor...changing a property and clicking Save causes the object being edited to refresh itself (refresh #1) but then the settings editor itself forces a refresh of the edited object after the save has been successful in order to ensure that the tree view and other fields in the settings editor are also updated (refresh #2).

Looks like all of the button flicker is gone now. I did some tests with enabling and disabling different classes to ensure that buttons don't flicker, and also ran some loops like:

#LOOP 100 {hp=%random(100)}

with a button that has a caption of: @hp. It worked as desired where there was no flicker and the button only updated once at the end of the loop. If I changed the loop to this:

#LOOP 100 {hp=%random(100);#UPDATE Button}

then it properly updated the button after each loop iteration.

Did the same tests with status bar items. All looks good now. So hopefully this will speed up some scripts, especially if you use lots of buttons and status items.

It's harder to judge overall speed resulting from my property and database refactoring code. It will be heavily dependent on your exact scripts. In my big test script that I use to test the parser and all of the functions, there was about a 5% speed improvement vs the same script in v3.06. But your mileage might vary.

Now I'm going to start working on some other bug fixes for this version to see if I can get something worth releasing tomorrow. I've got a lot of other things on my schedule next week, and the week after that Chiara and I will be on vacation, so I might as well try to release something tomorrow or later this weekend to let people test all of these new low-level changes.

Posted: Fri May 15, 2009 5:23 am

Sounds really good. I always love speed enhancing changes.

Wizard Joined: 22 Mar 2007 Posts: 2320

Especially ones that actually clean up code! Nice work.

Posted: Fri May 15, 2009 4:35 pm

Yeah, now after all of this prep work, maybe I can finally get to adding additional properties to Button States, which was the original issue that led me down this path. All I want to do is add a Text Color for each button state, which requires using the XML format for the Options field, which required the new property code, which required the new database code!

Button States have always been a bit weird. Trigger States are handled differently and they already create a full "Settings" record for each state that can have properties. But Button States don't have their own object record...they are stored within the parent Button record as an array of record values, which do not have access to the XML properties.

Today I'm going to look into this and see if I can make Button States implemented more like Trigger States so that I don't have so many special cases in the code. It's another low-level change that might add new bugs, but it's one of the big remaining things in the settings code that I've been postponing for a long time. Now that the properties and database code is cleaned up, I think it's finally time to clean up the Button State kludges too.

Posted: Fri May 15, 2009 9:21 pm

Wow, my brain is full.

I *think* I've got the buttons converted. I was able to get rid of all of the special-case kludges for button states and treat them just like trigger states. In fact, the code for the buttons is even a bit better in terms of inherited classes compared to trigger states, so I might revisit trigger state code in the future too.

But for now I'm just looking at the buttons. I've tested all of the different button types and they seem to all work. In addition, all of the weird bugs with the button state color getting set to black and the button state transparent option not working properly are fixed. In addition, I added the new Text Color property to the button states and now each button state can have a unique text color. (Woot!)

So this all seems pretty good, but I need to take a break and clear my brain and do some more testing. My guess is that I'm going to release this as a "alpha" test version tomorrow (Saturday) and let really brave Beta Testers give it a shot. I'm guessing that there are still some serious bugs in this, but it's difficult to test everything thoroughly without help. So stay tuned tomorrow for more info.