Title: Broadvision - Part 3

In last issue's piece, I did my usual trick of going off on a tangent by looking at session 'state' and design issues but I promised to get back on track this time by looking at database access and some other aspects of the Broadvision framework.

I don't know what you know about databases but when I started this project, all I'd experienced were indexed flat files and hierarchical databases (where you searched on a key to get a set of results and then searched within them etc). Broadvision is built on top of Oracle, a relational database, about which I knew next to nothing. In a relational database, you have a set of tables with keys. You can search a table on some criteria and get back a result table containing rows that match. The relational element comes in when you have a column in one table that acts as a key on another table. Broadvision has several key tables which mainly deal with users (BV_USER indexed by user ID) and products (BV_PRODUCT indexed by product ID).

A typical Oracle query uses SQL (Structured Query Language) to retrieve the desired rows from a table:

SELECT *
FROM BV_PRODUCT
WHERE some_condition;

This retrieves all columns ('*') from the named table ('BV_PRODUCT') that match the specified 'some_condition' (e.g., DEPARTURE_DATE > SYSDATE - departure date is after today).

Broadvision expects you to put almost everything relating to what you're 'selling' over the Internet into its product table, BV_PRODUCT. Each row of the table has a unique Object ID which Broadvision generates as part of the data load process and a supposedly unique product ID which you supply. At first, we couldn't see how to force five unique product types (foreign currency, travel books, flights, holidays and 'late deals') into this flat structure so we fought against it. We developed a complex, multi-level data model where, for example, holidays were grouped by destination and within that resort and within that property and within that individual holidays (with departure dates and durations).

Since Broadvision is designed to deal with a single product table, this left us with rather a difficult problem: how to access the myriad tables we had created.

I've commented before on how the Broadvision application framework shapes your thinking and its approach to databases is no different. To explain how it does this, I need to explain how it allows you to search tables and display results. Broadvision provides some basic objects which search the product table (to return a table of results), iterate over the returned results and display individual columns from the current row. The basic structure is as follows:

<BVBlockObject Dyn_List accessor_name=results>
<BVObject Dyn_ContentDBAccessor name=results ...>
<bvloop>
  <!-- a row -->
  <BVObject Dyn_ContentField name=NAME>
  <BVObject Dyn_ContentField 
                          name=PRICE><BR>
</bvloop>
</BVBlockObject>

We'll start with the inner elements: a Dyn_ContentField object produces HTML for the named column within the current result record. <bvloop> .. </bvloop> iterates over all the rows in the result table. The Dyn_ContentDBAccessor object (named 'results') performs the actual search - the parameters controlling the search are omitted above. The enclosing Dyn_List indicates an intent to iterate over a result table.

Our problem seemed to be that since Dyn_ContentDBAccessor was built to access BV_PRODUCT we needed to write our own accessors for our own tables. Of course, Dyn_ContentDBAccessor is part of a class hierarchy so we tried to inherit, with varying degrees of success, from that, its superclass and that's superclass. The eventual framework we ended up with - after some input from a Broadvision consultant - was to inherit from Dyn_ContentDBAccessor and build instead an accessor that worked directly with Rogue Wave's DBTools.h++ (on which Broadvision itself is built). This allowed us to search any table and do things like sort the results and group them by specified fields. The actual code was rather complex as, due to Broadvision's strange architecture, we had to override whole methods when we usually only wanted to override part of them. We ended up with code that set various flags, called the base class method, and then reset the flags. Not pretty.

Each accessor has a few key methods that prepare the query and execute it (get_data()) and then retrieve elements from the current row (data_value()) as determined by the enclosing Dyn_List. It's not actually a bad architecture but we were still fighting against the application framework's structure so it seemed more painful than it needed to be.

As with many of our other problems with the application framework, our problem was that we were trying to fight against it rather than work with it. Unfortunately, we couldn't see a way to work within it at the time and, in our defence, Broadvision's consultants and technical support staff couldn't identify a way to do this more easily.

The Rogue Wave database tools are excellent, by the way. Building complex SQL queries is a breeze with self-describing objects for SELECT and WHERE expressions, allowing you to build accessors that are evaluated only when you need to read results. If you've ever read any of Todd Veldhuizen's writings about template expressions, you'll find DBTools.h++ very natural indeed. We used a series of lightweight 'search context' objects to maintain the details of the user's choices and from those built complex DBTools.h++ expression sequences that we then evaluated against various tables to retrieve the necessary rows from various database tables.

Very recently, after I'd spent many hours wrestling with the somewhat sparse Broadvision documentation, I spotted two API routines that appeared to do what we'd have liked to do with Broadvision's own objects: get_content_by_cond() and get_content_by_query(). The former takes an arbitrary WHERE clause and applies it to a SELECT on BV_PRODUCT, the latter allows you to specify an entire SQL query, therefore you can use it to select rows from arbitrary combinations of tables.

This meant that instead of writing layers of accessor objects built on Rogue Wave's DBTools.h++ and then adding custom accessors to search based on 'search context' objects, all we would need to do was have a single object that created the raw SQL query and store it in a Broadvision application dictionary variable (see part 2 for information about Broadvision's application dictionary).

With the travel site, we wrote custom accessors, based on our Rogue Wave accessor, for each and every search we needed. With the car manufacturers site, all we needed was to write one object that converted the 'search context' into a SQL query and stored it. Thereafter, we could use the standard Broadvision accessor to retrieve products based on that query: Dyn_ContentDBAccessor lets you specify that you want to perform searches using the two get_content_by_xxx() methods above in addition to the simple BV_PRODUCT searches.

Well, OK, it wasn't quite that simple! With the travel site, we built a logical data model and pretty much stuck to it, using custom accessors to fit that data model. This meant that we missed out on some of the built-in functionality that Broadvision provides to 'observe' choices the user makes - the standard Broadvision database accessor lets you observe which 'products' the user has been shown, and which ones they've clicked on - shown an interest in.

With the car manufacturer's site we have reworked the logical data model to fit Broadvision's single table view of the world, compressing about a dozen tables down into one, flat product table. This allows us to discover a lot more about what the user is seeing and choosing.

In terms of the tagged HTML, not much has changed - our custom accessors worked in the same way as Broadvision's standard accessor. The difference has been in terms of the amount of code we've had to write: we created about 100 customised Broadvision objects for the travel site (in addition to the search context objects) whereas for the car manufacturer's site we are only developing about a dozen customised objects (again, in addition to the search context objects).

As you may have gathered from my comments about Broadvision, I don't feel that it's been very well designed from the point of view of extensibility. The methods are not sufficiently granular to allow developers to easily override just the functionality they want to: not enough common functionality has been pushed up into base classes. In a similar way, Broadvision's objects often have remarkably similar functionality but with subtle differences: there are very similar objects that deal with 'products', 'adverts', 'editorials' etc instead of a generic object that has a parameter indicating which type of item they handle. Another example that we tripped over recently was to do with link objects - in Part 1, I showed this example:

<BVBlockObject Dyn_SmartLink receive_class=Dyn_SmartLinkReceive
destination=hol/hlb03f.t>Destination
</BVBlockObject>

Broadvision have a similar object called Dyn_RawSmartLink that also handles links:

<A HREF="
<BVObject Dyn_RawSmartLink
receive_class=Dyn_SmartLinkReceive
destination=hol/hlb03f.t>">Destination
</A>

This has the benefit of allowing arbitrary attributes to be specified in the HTML, outside the Broadvision tag, e.g., JavaScript to control rollover graphic effects (where graphics on a page change as you move your mouse over a link). Furthermore, this gets around a deficiency in the Dyn_SmartLink object - although it's a BVBlockObject, it isn't a Dyn_Container. What this means is that you cannot embed database lookup objects within the link name (e.g., the Dyn_ContentField object I discussed above).

However, neither of these essentially similar objects provides a way of recording that a user clicked on the link. Comprehensive user observations is a high priority for us so we produced a modified version of Dyn_SmartLink, called ISS_SmartLink, that produces observations on every click and also solves the Dyn_Container problem. Then we looked at the 'raw' smart link object: it seemed silly to have two separate objects doing such a similar job so we added a parameter to ISS_SmartLink to control whether it behaved like Dyn_SmartLink or like Dyn_RawSmartLink. That change involved adding handling for a new attribute (about 10 lines of C++) and some conditional code in handle_body() (see Part 1 for more information on the general structure of a Broadvision object).

Appropriate parameterisation is a very important issue in library frameworks and one that Broadvision falls down on badly - if you're building an application framework, please bear parameterisation in mind: if you can provide runtime parameters that help users write less code, do so!

In the final part of this series, I'll take a closer look at the customisation we did for the travel site and for the car manufacturer's site, again contrasting the work involved when you fight the system against when you go with the flow.

There'll probably be a brief postscript to the series too - I've realised that Part 4 will probably be published before the car manufacturer's site goes live. If I can get permission from my client, in the postscript, I shall identify the two sites I've talked about in this series, as well as one of the other sites we've worked on. In the meantime, check out Broadvision's web site (http://www.broadvision.com) and look at some of the web sites built with the framework.