Lately, I’ve seen software frameworks starting to mark up properties with validation attributes. These are all simple single-field affairs ¹, for example, rules such as

• Regular expressions (WCF RIA Service example)

  [RegularExpression("[A-Z][A-Za-z0-9]*")]

• String length (minimum, maximum)
• Required field
• Not nullable text fields

These are not much different to the In-Schema validation mentioned earlier.

Complex validation

We have so far dealt with simple validation on a single field. There are some practical situations which require validation of two fields together.

• Password confirmation boxes (both passwords must be the same)
• Date range (the “from” date must be less than the “to” date)

Highly complex validation

Sometimes an entire data structure may need to be validated. These are rare. For instance, when splitting payments on an accounting system, we need to validate that the totals balance. This type of validation span multiple records.

Another example is validation of addresses. One might wish to validate the zipcodes are consistent with the States provided. In addition, there may be third party services that provide address validation. These validations span multiple fields (Street Number, Street Name, City, Zip, State).

When not to validate

Some times, it is necessary to not validate data.

• The user is postponing data entry, but wishes to save a draft.
• The data has already been validated when the record was created. Two examples come to mind:
  • A task has to be scheduled for a future date (this is validated). However, when the task is completed, and marked as complete, the scheduled date should not be validated.
  • When a signed document is recorded in the database, we perform a validation that the signature is current, and has not expired nor revoked. However, if we record other metadata against the document (such as archival schedules), then we should not revalidate the signature, because the signature may no longer be valid.

  In other words, the fields should only be validated if they are updated, or when a record is created, and left alone otherwise.

• Increasingly, languages (such as XAML and JavaFX) which permit Two-Way databinding find that validation can get in the way of data entry. For instance, an email property field may require the entry of a properly formatted email address. If the user tabs away from a control with a mistyped email address, the email field needs to store the incorrect value so that the user can fix it later. One design pattern is to generate a ViewModel, which is a “Plain Old Object” or a struct that permits bad data, but will return a list of error messages if the data does not validate.
• It is also worthwhile mentioning Jon Udell, who suggested that users should be allowed to override certain types of rigid validation rules, subject to these events being logged and audited. It will certainly help with better customer service, especially in exceptional situations.

Footnotes

¹XForms, through XML Schemas, can also validate number of items in a list, using the minOccurs and the maxOccurs attribute.

While most of these do not deal with database operations, they have a certain place in database modelling. The following are must haves:

• Short Display Names – a sane toString() for tabular listings
• Simple Display Text – a sane toString() for summarizing complex displays

I’m sure there are more gems to be mined from ASP.Net MVC. If anyone who knows more, please e-mail me.

I’d love to interview these guys on the blog just for some insights they gained from the process.

Some data dictionary mark certain fields as being read-only. Why should fields ever be read-only? How do they get edited in the first place?

Possible justifications:

• The field represents a permanent record, i.e. it behaves more like a document than a database field. A field can be set at record creation time and no-other.
• The field represents an integration field. For instance, an account_details table may have it’s e-mail and telephone fields populated from an LDAP source. By marking up the metadata consistently, the user interface automatically renders a field as label, or as a locked text box.

Validation checks should not apply to read-only fields. A clever code generator should warn the developer that validation rules have been applied on read-only fields, and these rules should be turned off.

AribaWeb is a 4GL RAD for web applications running on a Java stack.

It uses an approach described in data dictionaries as a database pattern to generate metadata-driven UI.

The Ariba toolset seems reasonably mature from the screencasts, although one has to be wary because a lot of the AJAX work appears to be relatively new.

I’m watching the MetaUI screencast now, and the bubble style error messages on fields is quite cool. However, cooler still is the use of CSS-like language to mark up how fields are presented. This reduces impedance mismatch between binding data to controls, instead turns data into its representation.

Custom validation rules is specfied using a groovy string. I’m not sure if this is translated into Javascript on the client-side. The Meta UI specification decoupled from HTML (layout is in named zones), and there is no reason why this couldn’t be rendered in a RIA.

Actions in UI on a POJO is declared via an annotation (similar to Naked Objects). However, the actions seem to be more restrictive in the sense that the methods must not take any arguments.

e.g.

@Action(message="Account Suspended")
@Visible("! $object.isSuspended")
void suspendAccount() {
...
}

Overall, AribaWeb looks very polished. I haven’t had a look at the underlying libraries used¹, but I look forward to delve into it further.

Footnotes

¹There was a mention of JPA and Hibernate. (Is the display layer JSF?)

The interesting twist brought by Coda Server is in deployability.

In the countries table, … use the REF TABLE .. Ref tables are identical to regular tables, except that they are replicated automatically across the development, testing, and production environments. Any rows added, updated or deleted in those environments immediately percolate to all of the others. This is perfect for data such as countries which generally don’t change based on your application’s whims.

Here are some examples gleaned from existing frameworks:

• Compiere/ADempiere – Account, Amount, Assignment, Binary, Button, Color, Costs+Prices, Date, Date+Time, FileName, FilePath, ID, Image, List, Location (Address), Memo, Product Attribute, Quantity, Row ID, Table, TableDirect, Text, Text Long, Time, URL, Yes-No, Number, Printer Name, Product Attribute, Quantity, Search
• Plone Archetypes – CMFObjectField, ComputedField, ImageField, FilesField, LinesField
• CodeIgniter/Django – URL, Email
• UDEF – Amount, Code, Graphic, Measure (Unit of…), Percent, Picture, Rate, Document
• Miscellaneous – tag

There are several high level ways of describing how versioning/auditability is implemented:

• Versioned rows – history is mixed with current data in a single table. Results in complicated joins, archiving problems, performance trouble
• Naive history tables – a mirror table that adds three additional columns – action (Create, Update or Delete), timestamp, and user
• Delta tables – records table, primary key, old value, new value, action (Create, Update, or Delete), timestamp, and user

Implementation

Web applications tend to share database connections. This restriction means that we cannot implement auditing via database triggers, since the trigger does not have access to the current user. Instead, auditing is implemented at the database abstraction layer.

Oracle has a construct called SYS_CONTEXT (see: TechRepublic AMIS)

Users, Groups and Elevated Privilege

Any non-trivial business applications tend to have functionality beyond simple read/write permissions. Examples of this kind of use-cases can be seen in setuid in unix applications. Any audit-aware applications should take this into account, and log the real user id.

Examples

• Document-type artifacts – for instance, if an organisation was tracking licenses, the address of the licensee should be recorded permanently. If the address changes, then a new row has to be created, so that it maintains a faithful record as to what the original issued license represented.
• Auditing

See Also

Data Dictionaries

Examples of Search in RAD

• Adding Full Text Indexes to char columns – MySQL, SQLite, SQL Server
• Query By Example (QBE) in MS Access – is a special case where all fields are made searchable
• Autogenerated modal forms with search/filter capabilities. Used when look-up tables get large

• Reference Table – small number of columns, small number of rows, permanent. e.g. U.S. States. Also see Reference Table Deployability
• Small Master Table – small number of columns (but more than a simple reference table), small number of rows, permanent. Master tables hold data, rather than look up information. e.g.
• Large Master Table – transactional
• Cross Reference – describes has-a, many-to-many and other relationships

Ken goes on to suggest the types of primary keys for each type of table, e.g. Single-column CHAR primary keys for Reference and Small Master tables; Auto-assigned integers for Large Master tables; and Compound keys for Cross Reference.

I would like to add that tables should be explicitly annotated for their patterns and expected usage.

Applicability and Consequences

• Code generation tools can then intelligently cache slow changing patterns in-memory, during application start-up, or upon first-use.
• Character primary keys can be used as shortcuts for rapid data entry
• Look up controls can be auto-generated when reference tables are used in foreign keys