Preferisco - thoughts about Oracle, EAI/ETL and more

Managing Windows scheduled tasks - SCHTASKS output misleading

2012-01-26T13:04:00.000Z

Here's a little gem - found on Windows Server SP2 but still there on Windows Server 2008 R2 SP1 at least.

I wanted to write a little script to disable some scheduled tasks (for maintenance) then after a predetermined time to re0-enable them. This is a common support problem, and I find I often complete the maintenance and forget to re-enable the tasks which results in alarms going off - but maybe not until the start of the next working day.

Anyhow, Windows gives you (at least) two ways of interacting with scheduled tasks:

SCHTASKS
PowerShell and the PowerShellPack which includes a TaskScheduler module

Although Powershell is an attractive option for scripting, PowerShellPack is poorly documented and the TaskScheduler module is a bit lacking. You can create, start, stop, register or get a task but there doesn't seem to be a cmdlet for actually enabling or disabling a task.

So, back to using groovy as a wrapper to SCHTASKS. All fine, we can use execute to create a CMD process that calls SCHTASKS /query to get the task status. Here's an example using easy-to-parse CSV format:

C:\>schtasks /query /fo csv /tn "\Apple\AppleSoftwareUpdate"
"TaskName","Next Run Time","Status"
"\Apple\AppleSoftwareUpdate","31/01/2012 11:15:00","Ready"

We can see that "Status" is in field 3 on the heading line, and its value is "Ready" on the data line. That's great.
To disable the task, we can then:


C:\>schtasks /change /disable  /tn "\Apple\AppleSoftwareUpdate"
SUCCESS: The parameters of scheduled task "\Apple\AppleSoftwareUpdate" have been changed.

Now let's check the status again:


C:\>schtasks /query /fo csv /tn "\Apple\AppleSoftwareUpdate"

"TaskName","Next Run Time","Status"
"\Apple\AppleSoftwareUpdate","Disabled",""

Yay! The task is indeed disabled - but look how the status has swapped into field 2 - under "Next Run Time". Presumably because there is no next run time while the task is disabled. A blank 3rd field value has been provided, but it is in the wrong place. Whatever way you list out the data, the error is still there:


C:\>schtasks /query /fo table /tn "\Apple\AppleSoftwareUpdate"

Folder: \Apple
TaskName                                 Next Run Time          Status
======================================== ====================== ===============
AppleSoftwareUpdate                      Disabled


C:\>schtasks /query /fo list /tn "\Apple\AppleSoftwareUpdate"

Folder: \Apple
HostName:      PIERO
TaskName:      \Apple\AppleSoftwareUpdate
Next Run Time: Disabled
Status:
Logon Mode:    Interactive/Background

OK, now I know this, I can work around it. But another example of MS inconsistency (which no doubt is now firmly baked in for "backward compatibility" for ever and a day...

Breaking change in calling Groovy on 1.8 upgrade

2011-11-29T16:07:00.001Z

I've been bitten by this a couple of times now, so for anyone else's benefit:If you have a bat file that calls a groovy program, you may notice surprising behaviour after an upgrade from 1.7.x to 1.8.x (I went from 1.7.4 to 1.8.4).

If your bat file looks something like:

..some stuff..

groovy myGroovy
copy xyz abc

... more stuff ..

Then in 1.7.4 you would have called groovy.exe, executed the program, then continued to copying the file. But in 1.8.x groovy.exe is deprecated so instead you execute groovy.bat.Unfortunately, when a Windows bat script calls another in that way, it effectively jumps to the script (with no return) so the script finishes at the end of groovy.bat.To fix this, use the Windows CALL instruction:

..some stuff..

call groovy myGroovy
copy xyz abc

... more stuff ..

With the CALL, the groovy.bat script executes and then returns control to your script, and the copy and more stuff actually happens.

NOTE: I think the reason I have the problem is that I installed the generic groovy rather than using the specific windows installer (eg here). But codehaus seems to be down right now.

MySQL Group By is a little too indulgent

2011-07-28T13:54:00.003+01:00

After 30 years of Oracle, I've found myself using MySQL recently. I came across a little thing that surprised me. I'm by no means the first to trip over this - I found this 2006 post from Peter Zaitsey on the same topic.

MySQL lets you write a group by statement that references columns that aren't in the group by, and aren't aggregates. For example:


mysql> select table_name, column_name, count(*)
    -> from information_schema.columns
    -> where table_schema = 'information_schema'
    -> group by table_name
    -> limit 5;
+---------------------------------------+--------------------+----------+
| table_name                            | column_name        | count(*) |
+---------------------------------------+--------------------+----------+
| CHARACTER_SETS                        | CHARACTER_SET_NAME |        4 |
| COLLATIONS                            | COLLATION_NAME     |        6 |
| COLLATION_CHARACTER_SET_APPLICABILITY | COLLATION_NAME     |        2 |
| COLUMNS                               | TABLE_CATALOG      |       19 |
| COLUMN_PRIVILEGES                     | GRANTEE            |        7 |
+---------------------------------------+--------------------+----------+
5 rows in set (0.07 sec)

A similar query from any version of Oracle would fail:


SQL> select table_name, column_name, count(*)
  2  from dba_tab_columns
  3  group by table_name;
select table_name, column_name, count(*)
                   *
ERROR at line 1:
ORA-00979: not a GROUP BY expression

In effect MYSQL is doing the GROUP BY as requested, and giving you the first value it comes across for the un-aggregated columns (COLUMN_NAME in this example). A near equivalent Oracle query would be:


SQL> select table_name, min(column_name), count(*)
  2  from dba_tab_columns
  3* group by table_name


TABLE_NAME                     MIN(COLUMN_NAME)                 COUNT(*)
------------------------------ ------------------------------ ----------
ICOL$                          BO#                                    14
PROXY_ROLE_DATA$               CLIENT#                                 3
TS$                            AFFSTRENGTH                            32
ARGUMENT$                      ARGUMENT                               25
IDL_CHAR$                      LENGTH                                  6
TRIGGER$                       ACTION#                                19
TRIGGERCOL$                    COL#                                    5

But in the Oracle case we are explicitly selecting the MIN(column_name), whereas MySQL's laxer behaviour is just picking the first column name at random (or rather, dependent on the execution plan).

So: when grouping in MySQL, make double certain that your SQL is really returning the number of rows you expected. In our example it is possible that the intention was actually the very different:


mysql> select table_name, column_name, count(*)
    -> from information_schema.columns
    -> where table_schema = 'information_schema'
    -> group by table_name, column_name
    -> limit 20;
+---------------------------------------+--------------------------+----------+
| table_name                            | column_name              | count(*) |
+---------------------------------------+--------------------------+----------+
| CHARACTER_SETS                        | CHARACTER_SET_NAME       |        1 |
| CHARACTER_SETS                        | DEFAULT_COLLATE_NAME     |        1 |
| CHARACTER_SETS                        | DESCRIPTION              |        1 |
| CHARACTER_SETS                        | MAXLEN                   |        1 |
| COLLATIONS                            | CHARACTER_SET_NAME       |        1 |
| COLLATIONS                            | COLLATION_NAME           |        1 |
| COLLATIONS                            | ID                       |        1 |
| COLLATIONS                            | IS_COMPILED              |        1 |
| COLLATIONS                            | IS_DEFAULT               |        1 |
| COLLATIONS                            | SORTLEN                  |        1 |
| COLLATION_CHARACTER_SET_APPLICABILITY | CHARACTER_SET_NAME       |        1 |
| COLLATION_CHARACTER_SET_APPLICABILITY | COLLATION_NAME           |        1 |
| COLUMNS                               | CHARACTER_MAXIMUM_LENGTH |        1 |
| COLUMNS                               | CHARACTER_OCTET_LENGTH   |        1 |
| COLUMNS                               | CHARACTER_SET_NAME       |        1 |
| COLUMNS                               | COLLATION_NAME           |        1 |
| COLUMNS                               | COLUMN_COMMENT           |        1 |
| COLUMNS                               | COLUMN_DEFAULT           |        1 |
| COLUMNS                               | COLUMN_KEY               |        1 |
| COLUMNS                               | COLUMN_NAME              |        1 |
+---------------------------------------+--------------------------+----------+
20 rows in set (0.06 sec)

Happy debugging everyone!

OT: Good value first class fare?

2010-11-26T11:43:00.003Z

From the NationalRail website:

One million squids for a day return?

Using Groovy AntBuilder to zip / unzip files

2010-06-14T12:23:00.006+01:00

I've been quiet for quite a while - partly because I am not working with Oracle just at the moment. I have been building some automated workflow systems using Groovy as the scripting language. I've known about Groovy since James Strachan first invented it back in around 2002/3 - but this is the first time I've really been using it in earnest. It's a great for portable scripts, and for integration with Java (it runs in the JVM). It's much friendlier than Java for someone like me who comes from a PL/SQL and C (not C++) background.

Anyhow, I found out about using Groovy Antbuilder tasks, and have been using them to manage zipping / unzipping file sets:


def ant = new AntBuilder();   // create an antbuilder
ant.unzip(  src: planZipFile, dest:workingDirName,  overwrite:"true")

Then I found I wanted to flatten the output (ie don't reproduce the directory structure). The Apache Ant documentation for the unzip task shows the Ant XML:

<unzip src="apache-ant-bin.zip" dest="${tools.home}">
    <patternset>
        <include name="apache-ant/lib/ant.jar"/>
    </patternset>
    <mapper type="flatten"/>
</unzip>

How to add the mapper element?

Well, lots of googling later, I couldn't find an example but I did see the patternset being used. So thanks to that, I found that the Groovy way of expressing the mapper part of this is to add a closure after the call:

def ant = new AntBuilder();
ant.unzip(  src: planZipFile, dest:workingDirName,  overwrite:"true"){ mapper(type:"flatten")};

So I hope someone finds that useful.

Customer satisfaction - the Xerox Effect

2010-02-12T10:42:00.002Z

Thanks to Martin Widlake for pointing to this gem of a paper from Dennis Adams (pdf), pointing out that an increase in customer satisfaction can lead to an increase in negative feedback, and vice versa. Anyone who has worked in customer support (whether on an internal help desk or for external customers) will have gone through a "why don't they love us, we're doing such a great job for them?" period. This might explain why.

User friendly / supported monitoring of concurrent processes

2009-10-13T16:59:00.007+01:00

Yes, I know everyone else is having a great time at OOW, but some of us are back in the real world still.

I've asked a question on OTN (under EBS General Discussion) Best way to execute / monitor long running custom conc request with slave.

Can anyone help me with suggestions for an EBS-supported API (11.5.10 on Solaris 10 / Oracle 9iR2) that would enable the professional user who launched a (PL/SQL) concurrent process to monitor its progress over several hours from his/her application UI? To add to the fun, the process is going to spawn some slaves to make use of all the spare CPUs / cores / threads we have lying around.

As a developer, I would normally start with the DBMS_APPLICATION_INFO.SET_SESSION_LONGOPS procedure (and I'll build that in anyhow) - but in this case I'm struggling to find any documentation or ML notes. to point me at something that would actually appear on the apps UI.

Answers here - or better still, on the OTN thread. Thanks in advance!

ETL patent case: Constellar and DataMirror let off off the hook; DataStage still in dock

2009-07-06T20:27:00.006+01:00

Once again Vincent McBurney delivers a fantastic summary of the latest state of the Juxtacomm ETL patent case: SQL Server, DB2 and DataStage will fight out Data Integration Patent Infringement.

I'm most interested from the Constellar point of view - I first came across Constellar (then Information Junction) as a product on sale in late 1993 / early 1994 (before joining the company from Oracle in 1995), so it always seemed clear to me that it would qualify as prior art to Juxtacomm's 1998 patent. Oddly, it seems that the parties to the trial have agreed that Constellar Hub (and DataMirror Transformation Server) can be dropped from consideration; they won't be subject to damages - but equally they won't be considered as prior art. I don't understand that, but I guess the IBM lawyers must know what they are doing.

So, the case rumbles on, serving (if nothing else) to show how broken the US software patent system is.

SQLstream delivers instant data stream analysis of Mozilla 3.5 downloads

2009-07-01T19:52:00.004+01:00

Here are a couple of posts that describe the download monitor/dashboard which is giving up-to-the-second statistics for downloads by country of the latest Mozilla release 3.5 (just about to top 5.5 million downloads since yesterday's launch). The dashboard has been put together with the help of my friends at SQLstream. Just don't try looking at this with Internet Explorer, as it doesn't support HTML5.

Julian Hyde on Open Source OLAP. And stuff.: SQLstream powers ...
By Julian Hyde
SQLstream gathers data from Mozilla's download centers around the world, assigns each record a latitude and longitude, and summarizes the information in a continuously executing SQL query. Data is read with sub-second latencies, ...
Julian Hyde on Open Source OLAP.... - http://julianhyde.blogspot.com/

SQLstream the Sequel - RealTime Intelligence for Mozilla BI in Action
From ebizQ Presents BI in Action Virtual Conference ...

Be Alert!

2009-06-15T22:00:00.003+01:00

Here's a tale of woe from an organisation I know - anonymised to protect the guilty.

A couple of weeks after a major hardware and operating system upgrade, there was a major foul-up during a weekend batch process. What went wrong? What got missed in the (quite extensive) testing?

The symptom was that batch jobs run under concurrent manager were running late. Very late. In fact, they hadn't run. The external scheduling software had attempted to launch them, but failed. Worse than that, there had been no alerting over the weekend. Operators should have been notified of the failure of critical processes by an Enterprise Management (EM) tool.

Cut to the explanation:

As part of the O/S upgrade, user accounts on the server are now set to be locked out if three or more failed attempts to login are made. Someone in operations-land triggered a lockout on a unix account used to run the concurrent manager. And he didn't report it to anyone to reset it. So that explained the concurrent manager failures.

The EM software that should have woken up the operators also failed. Surprise, surprise: it was using the same (locked-out) unix account.

And finally, the alerting rules recognised all kinds of warnings and errors, but noone had considered the possibility that the EM system itself would fail.

Well, it's only a business system; though a couple of C-level execs got their end of month reports a couple of days late, and there were plenty of red faces, nobody actually died...

Just keep an eye out for those nasty corner cases!

Oracle Exadata posts #1 TCP-H result

2009-06-07T13:01:00.002+01:00

Grag Rahn's Structured Data blog provides the data that Kevin Closson had to remove from his own blog. From an HP/Oracle point of view, a very good performance, reducing cost/QphH by a factor of 4.

However, it is interesting to see that the HP/Oracle solution is still more than 4 times the cost/QphH of the #2 placed Exasol solution (running on Fujitsu Primergy, and reported a year ago) - while the absolute performance improvement is relatively slight (1.16M queries/hr against 1.02M).

Flutter is the new Twitter

2009-04-21T22:23:00.002+01:00

Worth a look, for those (like me) who find 140 characters too much to hande. Hat-tip to the BCS Oddit blog
.

Doubly dynamic SQL

2009-02-22T12:36:00.005Z

It is great to see a new post from Oracle WTF last week, after a quiet period. Which reminded me to post this example of a dynamic search.

I won't post the whole thing, and I have disguised the column names to protect the guilty. The basic problem is that the developer didn't quite understand that if you are going to generate a dynamic query, you don't have to include all the possibilities into the final SQL.

Let's say the example is based on books published in a given year. First, to decide whether to do a LIKE or an equality, he did this:

' WHERE' ||

' (('||p_exact||' = 1 AND pdc.title = '||chr(39)||p_title||chr(39)||')' ||

' OR ('||p_exact||' = 0 AND pdc.title LIKE '||chr(39)||l_title||'%'||chr(39)||')) ' ||

' AND ('||p_year||' = 0 OR pdc.year = '||p_year||')' ||

So at runtime you get both predicates coming through. Suppose you wanted an exact search (p_exact=1), for p_title='GOLDFINGER'. We don't know the year of publication so we supply 0. The generated predicates are:

WHERE ((1 = 1 AND pdc.title = 'GOLDFINGER')
OR (1 = 0 AND pdc.title LIKE 'GOLDFINGER%'))
AND (0 = 0 or pdc.year = 0)

Wouldn't the logically equivalent:

WHERE (pdc.title = 'GOLDFINGER')

have been much easier? Add a few of these together and a nice indexed query plan soon descends into a pile of full table scans and humungous hash joins. Oh, and no use of bind variables, so in a busy OLTP application this could sabotage the SQL cache quite quickly.

My favourite part though is with the sort order. The user can choose to order by a number of different columns, either ascending or descending:

l_order := ' ORDER BY ' ||

' case '||chr(39)||p_sort||chr(39)||' when ''publisher asc'' then publisher end asc, ' ||

' case '||chr(39)||p_sort||chr(39)||' when ''publisher desc'' then publisher end desc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''book_type asc'' then book_type end asc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''book_type desc'' then book_type end desc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''title asc'' then title end asc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''title desc'' then title end desc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''year asc'' then year end asc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''year desc'' then year end desc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''book_id asc'' then book_id end asc,' ||

' case '||chr(39)||p_sort||chr(39)||' when ''book_id desc'' then book_id end desc';

Yes, you got it; given that the variable p_sort has been picked from an LOV, the whole piece of PL/SQL can be replaced by:

l_order := ' ORDER BY ' ||p_sort;

That looks better, doesn't it?

Tibco RV in a box - would appliances help streaming SQL?

2009-02-05T10:23:00.005Z

Several others have posted on the new Tibco Messaging Appliance - apparently it's Tibco Rendezvous (RV) in a box OEM'd from Solace Systems. Paul Vincent ponders at the Tibco CEP blog:

It’s quite feasible that the same approach could be used for “basic” complex event processing operations, especially those that don’t require history (or much persistence)

Paul is certainly right that CEP and event/data stream processing engines could benefit from the appliance approach. However I think that it is precisely those applications that require history that could most benefit. A high-speed streaming engine with a co-located database (running with much of the data in cache) gives the perfect environment for answering a wider range of "right-time" BI requirements; the engine can control exactly what needs to be retained and for how long to provide the appropriate time-series data to support whatever aggregated outputs are desired. The database acts as a memory extender (allowing infrequently accessed data to be paged out), as a source of replay / recovery data, and as a history for newly defined streaming queries. The latter is most important - if you want a 1 hour moving average, most streaming queries won't give you the right answer until the end of the first hour.

Analytics as a Service

2009-02-02T10:23:00.003Z

With all this talk of SQLstream's recent v2.0 launch, I was interested to read Tim Bass's CEP blog posting on Analytics-as-a-Service. He calls it A3S - and rightly avoids calling it AaaS; apart from the fact the X-as-a-Service is almost as cliched as XYZ 2.0 (and equally meaningless), careless use of that sequence of As and Ss could cause spam filters round the world to get over-excited.

If we must have the as-a-service tag, I'd like to trademark BI-as-a-service: BIAS. Apart from being a proper acronym, it also gets across that BI often gives you the answers you think you want - not necessarily the ones you really need.

Shared Feeds

2009-02-01T16:07:00.003Z

My posting rate has been quite low recently, but I have been enjoying using Google Reader to keep up with what everyone else is saying.

I've been sharing items with some friends / colleagues, but it seems harmless to open this up to the world. The topics are eclectic, but mainly around product development, event stream processing, RDBMS and data warehouse. Maybe the very occasional Dilbert or xkcd. You're all very welcome to see what I've shared - the links are now on the right hand side of this blog. And if you too are using a reader, here is a direct link to my shared items and here is the atom feed for them. If Google could just pack up my daily shares as a single blog post, wouldn't that be great.

I'm still successfully avoiding Twitter... I've no idea how I would find the time, and having seen Philip Schofield mentioned as a user (on a low rent BBC Sunday afternoon show) - and apparently now he is London's #5 tw*t(terer), so it must be hopelessly uncool anyway.

SQLstream launches v2.0 of its Event Stream Processing engine

2009-02-01T14:55:00.005Z

As well as working as a freelance Oracle consultant, I have spent most of the last year working with SQLstream Inc on their Event Stream Processing engine, version 2.0 of which has now been launched.

My initial point of contact was Julian Hyde, with whom I worked in the Oracle CASE / Oracle Designer team in the early 90s. He went out to Oracle HQ and worked on bit-mapped indexes, then spent time at Broadbase before becoming best known as the founder-architect for the Mondrian OLAP server.

Event Stream Processing (ESP) is a development of what we might have called Active Database a few years ago. Rather than running queries against stored data as in an RDBMS, we register active queries which can be used to filter data arriving from any kind of stream. These active queries behave just like topic subscriptions in a message bus - except that unlike typical JMS implementations, the SQLstream query engine can do more than just filter. SQL queries against the data-in-flight can also:

join data between multiple streams, or between streams and tables
aggregate data within a stream to give rolling or periodic aggregates based on time or row counts
apply built-in and user-defined functions to columns within the stream's rows
build a network of streams and views to support complex transformation and routing requirements

Queries are defined using SQL:2003 with minimal extensions (there are some SQL examples on the web site); so developers used to working with an RDBMS will find it easy to grok the relational messaging approach. Unlike a typical message bus, most transformation and routing takes place entirely inside the SQLstream environment, rather than being delegated to external applications.

My experience working at JMS vendor SpiritSoft convinced me of the value of asynchronous message-based techniques, which in the last 10 years have spread out from high-end financial systems to ESB implementations all over the place. Since the early 80s we have seen how the RDBMS swept all other forms of structured data storage away. Now SQLstream's relational messaging approach removes the impedance mismatch between how we store data, and how we process it on the wire. In principle, this architecture can subsume both message-oriented (ESB style) and data-oriented (ETL style) integration architectures.

It should be said that "other ESP engines are available". Oracle itself has two projects on the go: its own CQL which I believe is still internal, and Oracle CEP (the rebranded BEA WebLogic Event Server - which itself is (or was) based on the open source Esper project). These two development threads will no doubt combine at some point (perhaps they already have?). IBM also has two or three independent CEP (complex event processing) projects on the go.

I think the same thing will happen to ESP / CEP as happened to ETL/EAI tools in the last ten or fifteen years. For sure, the database/application server vendors (especially Oracle and IBM) will sell plenty of this software within their respective client bases. An Oracle CEP system that was (or could be) tightly integrated with the RDBMS itself - maybe executing PL/SQL as well as Java functions - would be an easy sell. However multi-vendor sites will be interested in an agnostic / vendor-independent tool as a basis for their integration efforts. Just as Informatica has carved out a place for itself in competition with Oracle's ODI and OWB and IBM's DataStage, so SQLstream and other ESP vendors can fight for the common ground. It will be very interesting to see how it all turns out.

See the SQLstream product page for background, plus posts from Julian Hyde, CTO of SQLstream and Nicholas Goodman, Director of BI Solutions at Pentaho.

PS: here's another post from David Raab.

Unexpectedly honest job posting

2008-11-28T18:58:00.003Z

I recently joined the Oracle Connections group on Linked-In and I'm getting regular daily mails with job postings and searches for work. Mostly harmless, but I've just seen a great one:

Lead Oracle Confrigurators NEEDED

(You may need to be a member of Linked-In and/or the Oracle Connections group to follow the link).

I think we've all been there, confRigurating away to our heart's content, haven't we? It certainly explains a lot of the problems we see in production.

OT: 3 Mobile Broadband doesn't much like Gmail and Blogger

2008-11-02T14:36:00.002Z

I use 3 mobile broadband in the UK (and Italy) and I have only a couple of nags about it:

I can't use FireFox 3 and Gmail together over mobile broadband - I have to switch to IE7 and sometimes I have to downgrade Gmail to the simple HTML version. I have 2 gmail accounts (one work, one private) and the problem seems to be worse on the latter. The symptom is that the loading bar is followed by a blank screen and the status "Done".
I simply cannot seem login to blogger.com over mobile broadband - hence no posts during the week (probably a good thing as I'm supposed to be working). I get a 404.

These problems seem to persist whether I am in Italy with no bars on my reception, or in the west end with 5 bars. Does anyone have any idea what's going on?

Exadata - has it been in development for two years?

2008-09-25T20:09:00.006+01:00

Just my nit-picking mind, but why does the Exadata technical white paper say (at the time of writing at least) that it is "Copyright © 2006, Oracle"? I don't think they've been working on it that long - much more likely some soon-to-be-embarrassed technical writer has cut and paste the standard boilerplate from an out of date source.

What, no rule-driven content management?

Exadata and the Database Machine - the Oracle "kilopod"

2008-09-25T20:09:00.005+01:00

There has already been plenty of interesting posts about Oracle Exadata - notably of course from Kevin Closson here and here (update: and this analysis from Christo Kutrovsky)- but I just have one thing to say.

Larry Ellison was quoted in a number of reports saying the Oracle Database Machine "is 1,400 times larger than Apple’s largest iPod".

Larry, when you want to get over that something is big - really big that is, industrial scale even - just don't compare it with a (however wonderful) consumer toy. Not even with 1,400 of them. 1.4 kilopods is so not a useful measure.

By the way, can I trademark the word kilopod please? (presumably not - a quick google found a 2005 article using the same word, and there is some kind of science-in-society blog at kilopod.com).

Back to the future - or is that forward to the past

2008-09-04T19:30:00.003+01:00

I'm starting a new contract this coming Monday. I'd better keep the client confidential until I've found out how they feel about blogs, but the job revolves around data matching and data quality, using Oracle and SSA.

At the interview, I found myself less than 100m from the site of my first ever "proper" job (the old Scicon offices are now an upmarket West End hotel). So just 28 years and 1 month later, I will be sauntering along Oxford Street once again.

I'll be commuting daily at first, but I will try to stay down during the week at least some of the time if I can find somewhere cheap, clean and convenient. Any old colleagues around central London - sometime between now and Christmas we should meet up ...

Microsoft acquires DATAllegro

2008-07-25T08:19:00.005+01:00

Lots of interesting posts on this news, from:

Curt Monash's DBMS2 blog - all his DATAllegro links are here - as there are so many of them. He also links to other comments
Philip Howard at Bloor and IT-Director suggests that MS is assembling an entire DW stack with Zoomix and perhaps one day Kalido and Ab Initio as additional components
Mark Madsen from Intelligent Enterprise says what it means for customers, other vendors and BI
Kevin Closson with his welcome plain-speaking (and Oracle-centric) viewpoint refers back to earlier posts wondering about some specific details of this emperor's clothes
Seth Grimes thinks it's a mistake that will be slow to deliver (and may hurt existing DATAllegro customers); it should have been Dataupia
DATAllegro CEO Stuart Frost sounds happy with his new role

Meanwhile Seth Grimes also uses the news to point up a real difference in quality between Google and Microsoft search capabilities.

ESB consolidation - Progress buys Iona

2008-06-30T15:11:00.001+01:00

I guess Iona was one of the first well known Irish software companies, and now it must be one of the last. It has succumbed to Progress Software for $106M. Paul Freemantle beat me to the news that Progress now owns or is a major committer for at least 4 different ESBs - Sonic, Artix, C24 and ActiveMQ/Camel/ServiceMix (plus Actional if you like) - hey, that's more than Oracle isn't it? (or maybe not...).

First come first severed (sic)

2008-06-21T10:20:00.003+01:00

A little off topic perhaps, but sometimes the illiteracy of business communications just amazes me. This is from a Jobserve recruitment ad:

A large railway company are looking to hire a Oracle Developer to join there expanding company on a 3 month rolling contract.

Next comes a fairly detailed set of technical requirements; as it seems to make sense, it was no doubt supplied by the said railway company. Finally the ad ends with this appeal:

The is major role and candidates will be selected on matching skills and on first come first severed basis

Sounds painful! Seems unfair to be severed for being the first response. Perhaps I should wait a week before applying?

Vote now to open up Metalink!

2008-06-14T15:27:00.002+01:00

Richard Harding has proposed on Oracle Mix that Metalink should be opened up:

There is goldmine of useful information in Metalink and having access to it would optimize the efficiency of people using Oracle toolsets, in my opinion enhancing productivity and by inference Oracle adoption globally which would be win win for everyone.

I've expressed the same thoughts myself in the past. So all of you Oracle professionals who would benefit from access to Metalink but are not included in your employer's arrangements (especially for freelancers like me) - go and vote!

Another blogger

2008-06-11T08:30:00.003+01:00

Good to see Mark Bobak blogging - he's one of the stronger contributors to the Oracle-L mail list. Thanks to Doug Burns for spotting his blog.

BEA Aqualogic broken up

2008-06-06T10:29:00.003+01:00

The Register reports on how Oracle is going about the integration / breakup of BEA's Aqualogic SOA products (splitting them between ex-Stellent and Oracle Fusion product lines). Oracle's actions are described as "firm but fair".

Performance problems concatenating LOBs

2008-05-13T09:18:00.003+01:00

Greg Pike talks about some potential performance pitfalls with LOBs; I would comment on his blog, but it's a login only. Anyway, I posted on these problems last year (specifically with DBMS_LOB.APPEND), and posted some code to deal with it. Batching up appends makes a big difference - a 15-20 fold improvement, in the example shown (against 9.2.0.5).

MS subpoenas Oracle in Juxtacomm ETL patent case

2008-05-08T14:35:00.003+01:00

Vincent McBurney summarises the latest developments in the Juxtacomm "we invented ETL, yeah really, give us all your money" patent case, which I have mentioned before. Dang, don't these "obvious" cases take a lot of time and effort to clear up.

Latrz - for all those web pages you want to read, just not right now

2008-04-21T10:35:00.000+01:00

James Strachan blogged about Latrz here. It's a neat little Google app that lets you bookmark and tag pages you want to read later, then come back and read them (and check them off your list) when you've got a spare few minutes. Coming soon, you'll also be able to share the stuff you liked with your friends / work colleagues.

Kalido Business Information Modeler - everyone should have one

2008-04-07T10:40:00.000+01:00

I posted about Kalido Business Information Modeler back in February, but now it's gone GA and Philip Howard reviews it at IT-Director. He is pretty enthusiastic and says "everyone should have Kalido Dynamic Information Warehouse and they should certainly have Universal Information Director and Business Information Modeler too."

The only negative he comes up with is "the relatively limited number of platforms that Kalido's software runs on: one would like to see it on Netezza for example, or Teradata".

IBM FastTrack for Source To Target Mapping for DataStage

2008-03-31T15:03:00.001+01:00

Vincent McBurney was first to report IBM FastTrack for Source To Target Mapping, and he has been followed up by Philip Howard's note "the business face of data integration" at IT-Director.

I find it interesting that releasing a simple attribute mapping tool is seen as a major breakthrough for the DataStage/Information Server family; Constellar had more or less exactly that 12 years ago (no glossary though); the UI may not have been quite so business-user friendly, but certainly supported point and click, plus simple integration with metadata repositories.

Vitria brings Web 2.0 and high transaction rates to BPM with M3O

2008-03-13T17:00:00.000Z

Vitria recently announced its new M3O product, which is claims to be the convergence of BPM, Web 2.0 and Event Processing. Bloor's Simon Holloway reviews it here, where he quotes Vitria saying:

"BPM provides standards-based executable modelling (based on BPMN) on top of business knowledge Repository.
Web 2.0 provides the rich user experience with zero footprint to enable a collaborative design environment.
Event processing provides the support for rule and process definition and real-time runtime performance based on event driven architecture.
Only when you combine these together do you get a fundamentally new user experience with multilayer visualization, collaborative modelling environment, business level abstractions and event management"

Leaping shamelessly onto a passing bandwagon, Vitria explains M3O as "think iPhone meets dashboards" (quoted from ebizQ). The idea is that the "iPhone coolness" of the Web 2.0 interface will remove the gap between business and IT people. Well, as long as it doesn't (like the iPhone) lock users into an expensive long term relationship...

This looks like the first fruits from the return of JoMei Chang as CEO last July and the decision to go private, executed last March.

Cape Clear-out

2008-03-12T19:21:00.002Z

I just caught up with the month-old news that Irish ESB/SOA vendor Cape Clear has been bought by WorkDay, a supplier of ERP SaaS.

Of course, SaaS needs integration, and Cape Clear CEO Annrai O'Toole promises that they will be providing the necessary Integration-on-Demand - but it seems like one of the leading independent commercial vendors has now been marginalised.

Ronan Bradley, former CEO of PolarLake (also Irish - one of our partners at SpiritSoft, and later a client of mine) also worries where the ESB market is going.

Is this a general problem in the middleware market, or is it just 'cos they is Irish? Are small vendors being caught between the rock of large vendors and the hard place of open source? Or is this (as Steve Craggs suggests in a comment on Ronan's post) simply a result of Cape Clear's own hubris?

Change of scene

2008-03-09T09:38:00.001Z

My current project is coming to a sudden end, so it looks like my weekly commute from Suffolk to Lancashire will be done by Easter. I've already got two or three interesting (and very different) opportunities to consider over the weekend - and if they don't come off, there's plenty on jobserve. More news as it happens; I am looking forward to spending a little bit more time at home and a little less time on the M6.

Quantum dot memory

2008-03-08T17:10:00.000Z

Next time you have trouble with i/o performance, think about this: wouldn't it be great to fit a terabyte of non-volatile RAM onto a one square inch postage stamp, with write times of 6 nanoseconds? Soon quantum-dot memory may do just that for you. That's nearly as fast as regular DRAM, and around 1000 times faster than typical flash memory. Scientists even say they could eventually get the write time down to picoseconds. Scary or what?

H-Store - a new architectural era, or just a toy?

2008-03-01T12:06:00.000Z

Philip Howard's commentary Merchant relational databases: over-engineered and out-of-date? supports the idea that perhaps general purpose relational databases should now be treated as "legacy". He references a paper The End of an Architectural Era (It's time for a rewrite) by Michael Stonebraker and others from MIT.

My first thought was that RDBMS developers such as Oracle have seen off previous architectural challengers - most notably object oriented databases (OODBMS) - in the past 25-30 years. What makes Stonebraker's H-Store any different?

First, a quick summary of the paper:

RDBMSs were designed 30 years ago - since then memory and cpu have become faster, cheaper and bigger, changing the balance against magnetic (disc) storage
increasingly they are failing to meet today's complex challenges
niche solutions have overtaken "general purpose" RDBMS in many areas (eg data appliances for business intelligence; specialist text search engines; etc)
and now even OLTP, the "core competence" of the RDBMS, is no longer safe; a new approach (such as H-Store) can easily beat traditional RDBMS by cutting out non-functional architectural features (eg: redo logs stored on disc) and achieving the same goals (ACID transactions) in another way.

The paper claims that H-Store can beat a traditional RDBMS at TPC-C style benchmarks; an early version runs up to 80 times faster than "a very popular RDBMS" which itself underwent several days of tuning by a "professional DBA".

H-Store's secret sauce is that it is (in effect) single threaded. It makes the assumptions that all transactions are very fast; then it executes each transaction in turn. This gets rid of the need for complex read-consistency models. Other optimisations include keeping undo in memory (because transactions are short and sharp) and discarding it at the end of the transaction.

Well, go off and read the paper for the details, but here's what I think.

On the negative side:

As a comparative benchmark, this fails through insufficient disclosure. For a paper that passes as academic, there is remarkably little detail on what they actually did.
Stonebraker assumes that "most" OLTP systems can be represented by a hierarchical model - what he calls a "constrained tree application" (CTA). As a result these applications are relatively easy to partition over a shared-nothing architecture. I wonder whether this is really the case. Parts of your application may be like that, or they may be like that for some periods (during the online day, for example). But even OLTP applications need to manage longer transactions, complex reporting, and updates to the "read only" tables. In his example, he assumes (section 5.0) that the Items table is read only, so it doesn't break his tree and it can easily be replicated. But we know that new items will be added; others will be re-priced, re-categorised, phased out. Can that be handled without interrupting a 24/7 H-Store style application?
He also seems to assume that there is only one axis of partitioning - in his case, the warehouse. But over time, the main focus of interest changes. An order is taken at a shop; it is ordered from a warehouse; it is delivered to the customer. Different "CTAs" at each stage. How does the H-Store morph its representation through the course of the information lifecycle?

On the positive side, though, this represents a call to action for the traditional vendors.

Any performance specialist knows that the best way to tune something is to stop doing it. If we really can change the rules of the game, we can avoid all that expensive "insurance". We're used to making calculated design tradeoffs for performance; this could be just another one of those.
I suspect that the RDBMS vendors will (more or less rapidly) steal any really good ideas. Stonebraker states that "no [RDBMS] system has had a complete redesign since its
inception". But, like the proverbial axe, RDBMS internals have been refreshed, a piece at a time. Oracle's database kernel has had at least two major rehashes in its lifetime. It's well within Oracle's or IBM's capability to incorporate the more realistic ideas from this paper, and to find a way to blend them with current state of the art.
They can also learn from the approach MySQL has taken of supporting multiple storage engines - horses for courses. Oracle already merges XML, relational and OLAP data stores; building in a high performance OLTP kernel to address specific classes of OLTP application is not at all inconceivable. Although it will lead to all sorts of information lifecycle difficulties, we are already used to migrating data from OLTP to OLAP; with good tool support it should be possible to work round the constraints that allow H-Store to dispense with so much that we normally take for granted.

I may revisit this paper to tease out other issues - for example Stonebraker rants against SQL (perhaps he's never got over Ingres being forced by the market to provide SQL rather than the more academically respectable Quel). H-Store uses C++, and may move to Ruby; the implication is that applications will be object-oriented, making row-by-row navigations (like so many J2EE apps, and suspiciously like COBOL/Codasyl) rather than being set-oriented.

Let's watch this space.

SYS_CONTEXT versus V$ views for getting session information

2008-02-26T22:07:00.000Z

A thread on Oracle-L today sidetracked into the use of V$ views for getting hold of session information such as username, SID and module. The poster had a logon trigger that was supposed to record these items for each logon from a particular client.

I don't approve of granting access to V$ views willy nilly; best practice is always to grant the minimum privileges necessary to achieve an objective.

Another poster raised the issue of performance. In the past, SYS_CONTEXT was considered slower than direct access to the views.

So here is a test to compare the two:


set echo off feedback off
set timing on

set termout off

variable v_loops number;
exec :v_loops := 1000000;

set termout on

prompt Testing sys_context

declare
 l_user varchar2(30);
 l_action varchar2(32);
 l_module varchar2(48);
 l_sid number;
        l_loopcount pls_integer := :v_loops;
begin
    for i in 1..l_loopcount loop
 dbms_application_info.read_module(l_module, l_action);
 l_user := sys_context('userenv', 'session_user');
 l_sid := sys_context('userenv', 'sessionid');
    end loop;
end;
/


prompt Testing mystat

declare
 l_user varchar2(30);
 l_action varchar2(32);
 l_module varchar2(48);
 l_sid number;
        l_loopcount pls_integer := :v_loops;
begin
    -- note this only gets one of the three pieces of information
    for i in 1..l_loopcount loop
 select sid
 into   l_sid
 from   v$mystat
 where rownum = 1;
    end loop;
end;
/

And here are the results:


C:\sql>sqlplus testuser/testuser

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Feb 26 22:23:10 2008

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production

SQL> @sys_context_test
Testing sys_context
Elapsed: 00:00:07.31
Testing mystat
Elapsed: 00:00:38.57
SQL>

I suspect that in the past SYS_CONTEXT issued recursive SQL under the covers (just as the SYSDATE pl/sql functions used to, and the USER function and the 11g assignment from a sequence still do).

Now I assume SYS_CONTEXT gets its information directly.

The UML industry - as predicted

2008-02-20T21:51:00.000Z

The Register has been poking fun at UML (and why wouldn't you). They have dug up a 1997 term paper piss-take which was probably pretty funny at the time; now it seems inspired, prescient even.

Shared Business Vocabulary

2008-02-18T21:30:00.000Z

Following on from my active metadata post yesterday, Mike Ferguson at Dataflux says Shared Business Vocabulary (is) needed everywhere. By "shared business vocabulary" he means metadata, as this earlier post explains. I don't particularly like his phrase - metadata is fine by me - but maybe he's right and we need to use a less loaded (and misunderstood) term for it. And particularly in the warehousey world "metadata" also refers to the dynamics of data (when was it loaded, how long did it take) so there's scope for a better word or phrase.

Whatever that word or phrase is, please please can it not be ontology ....

Active Metadata

2008-02-17T12:29:00.000Z

Yesterday I moaned about poor use of metadata. Today I'd like to point to an example of the way metadata can be used actively. Andy Hayler describes Kalido's Business Information Modeller, which allows you actively to control and when necessary reshape the data warehouse by changing metadata; you can generate a BO universe, or deploy metadata to Cognos BI tools (more to come, apparently). You can (apparently) even undo the metadata changes, and see your warehouse as it would have been; kind of like Oracle's flashback recovery, but for data warehouse structures.

See Kalido's podcast, screenshots and other resources.

2008 Data Management Predictions from Dataflux

2008-02-16T16:35:00.001Z

Mike Ferguson posted these over a week ago. Key thoughts to take away from my point of view:

Information and data architects will continue to be in demand
Companies will need to invest again in data modeling tools and in data modeling skills
Holding this metadata in spreadsheets is no longer acceptable.

It is depressing to me that so many projects keep their business critical metadata in Word documents, Excel spreadsheets and Visio drawings. Some developers prefer to rely on their IDE - keeping DDL definitions as SQL in text files. That protects the definitions (at least you can find them quickly), but it doesn't get the maximum value out of it.

CASE tools have been around since the mid 80s or before - Oracle's SQL*CASE, now better known as Designer, was under development when I joined in '86 - so how come they are used less and less? They did get a bad name for encouraging complex, expensive and ultimately useless corporate data models - and we're glad to see the back of those, and the ivory towers they came from - but they can still be very helpful in defining and developing the metadata we need as a basis for system development.

I wonder whether the main problem is that many CASE tools are simply too expensive and/or too closed; they just can't cope with all the different kinds and layers of metadata we would like to throw at them, and they don't integrate well with all the other development tools around. Look at Designer - it's been more or less static for the last 10 years, and other Oracle products barely take any notice of it. No wonder it's slowly fading away.

Perhaps it's time for Oracle to get a grip and provide some common repository / metadata management for use across all its myriad of tools? Or for a small vendor or OSS project to take up the challenge? Let me know if you've already found the tool that can pull together ERDs, schema models, UML process diagrams, Discoverer EULs, a BO universe, Warehouse Builder or ODI transformation definitions and all the other kinds of development metadata that projects deal with every day.

Writing XML to a file

2008-02-15T18:31:00.002Z

Marco Gralike has posted a couple of items at the Amis blog and a later followup on his own blog about ways of writing XML to a file - comparing the use of CLOB2FILE with DBMS_XMLDOM.WRITETOFILE with DBMS_XSLPROCESSOR.CLOB2FILE - and finding the latter faster.

Surely this is simply because the input to the test is an XMLTYPE (the result of his test query).

The call to CLOB2FILE requires an implicit conversion from XMLTYPE to CLOB and then simply serialising the CLOB (a large string already in XML format)
WRITETOFILE is making an implicit conversion from XMLTYPE to DOMDOCUMENT - it is constructing a DOM tree and then walking it to produce the serialised output. Here's the signature for WRITETOFILE:


DBMS_XMLDOM.WRITETOFILE(
            doc IN DOMDOCUMENT,
            fileName IN VARCHAR2,
            charset IN VARCHAR2);

Building that tree is a major overhead - in this case - though obviously it wouldn't be if you actually needed to navigate around the tree adding/moving/updating or pruning nodes and branches.

Update 16-Feb:
Marco has continued the story here, in great detail and at great length (tkprof listings and all). He's particularly interested in file size; I'd be more interested in performance impact of the various steps in his operations. I will try to reproduce his tests and summarise those soon.

Nominative determinism

2008-02-15T18:22:00.000Z

Nominative determinism: when your job or hobby matches your name. For example, the famous Major Major, from Catch 22. Or the British Antarctic Survey scientist I heard on Radio 4 yesterday explaining how he was going to drill down through 3km of ice - a Mr Core (or was it Bore - they both work, and I was in the car at the time).

The latest to catch my eye was in this quote at El Reg: "The vice president in charge of Red Hat's JBoss middleware Craig Muzilla, said Red Hat is going to undercut giants IBM, Oracle and Microsoft in the saturated and expensive middleware sector."

With a name like Muzilla, I guess working for an OSS company was pretty much inevitable.

DATAllegro, DW appliances and the Oracle lock-in effect

2008-02-15T17:19:00.000Z

Andy Hayler has an interesting post on DATAllegro - talking about its sweet spot at the very high end of data volumes; apparently helped by a grid with up to 1TB/minute transfer speeds.

What caught my eye was Andy's observation that "Oracle customers can be a harder sell (than Teradata's, for example) not because of the technology, but because of the weight of stored procedures and triggers ... in Oracle's proprietary extension" (ie PL/SQL).

Just looking at the various technologies people talk about on OraNA blogs, I wonder whether the continuing growth in Java and .NET based applications over the last decade - where many of the business rules are executed outside the database - might hurt Oracle precisely by removing the PL/SQL "stickiness" Andy describes.

Admittedly Apex is a counter example - thoroughly PL/SQL, thoroughly Oracle; no chance of moving Apex clients to another RDBMS yet. And the tendency to provide management interfaces through PL/SQL (or very Oracle-specific SQL extensions) also helps to keep Oracle customers tied down. Which is good for us Oracle died-in-the-wool crowd. Whether the lock-in is good for the customers (or even Oracle) is another question; an existing customer lock-in can feel like a lock-out to new customers.

Any thoughts out there?

Regards Nigel

Schema Version Control

2008-02-05T20:09:00.000Z

An outbreak of synchronicity has produced a volley of blogs and questions about database version control in the last few days.

Tom Kyte adds his own twist to this story at Coding Horror.
JAson Hinds asked about database release management on one of the OTN forums.
And Syed Jaffar Hussein (the Human Fly) launched a thread on Oracle-L on the subject.

I seen so many projects take code control seriously but have no real idea how to control a database definition (let alone how to control changes to multiple instances of development, test and live data, both in house and on client sites).

Code control is a walk in the park compared to "schema control" (where the delta , like an ALTER TABLE needs to be developed alongside the new version of the database object creation script).

And "data control" adds even more challenges:

can you upgrade the data in place? or do you need to create a replacement object?
Is there enough space?
Do you need to suspend user access? How long will it take?
what are the chances of reversing the upgrade if something goes wrong?
Will you even know if something goes wrong?
Who decides go forward, go back or muddle through?

A complex application version upgrade should be treated with as much respect as a data migration project. There should be a strategy (sure, it may be a standard strategy). Use standard tools / scripts. The more the process is repeatable - and the more it is repeated - the better. Make sure that all likely error conditions (missing log files, out of DB storage, non-writeable working directories, no listener) are tested for - test the tests!

By the way, the "process" includes all the operating procedures. Make sure that upgrades aren't nannied through by developers. Script the whole thing; have the production DBAs try it out the procedures as early as possible in test to make sure that they understand what's supposed to be happening (and how to know if it worked OK; the best thing is if the script just says:

IT ALL WORKED OK

SOMETHING WENT WRONG IN STEP 13 - SEE LOGFILE xxxyyy13

By the time the schema-and-data upgrade is executed for real, nothing - not even the unexpected - should be unexpected.

Patently ridiculous

2008-02-04T20:48:00.002Z

SCO has had a pretty hostile press over the last few years as it morphed from a real software business into a patent mining outfit. Well, I've now come across another case of patent ambulance chasing over at Vincent McBurney's blog on ITToolbox. JuxtaComm and Teilhard versus IBM Microsoft Informatica SAP etc tells the story so far - basically that a small company is claiming it invented ETL, patented it in 1997, and can it now have loads-a-money if you please.

I followed Vince's link to Groklaw where InterSystems, one of the parties, is asking for help.

One of the parties is DataMirror (now part of IBM, and the purchaser of my former employer Constellar). As tne SQL Group, Constellar had an ETL system that seems to fit the key claims as early as 1994 (to my personal knowledge - it may have been on the market in 1993)

I'm stunned that (apparently) Oracle has already caved, handing over $2 million. One of the reasons I am so surprised is:

I was actually working for Oracle UK in 1994 - the Constellar Hub (then known as Information Junction) was evaluated for one of my clients (who ended up purchasing ETI instead).
Oracle UK was involved in quite a few pre-sales to Telecoms, Pharma and others over the period 94-2000.
Oracle US consultants helped support the product in at least a few US clients.
Constellar's US office was actually sublet from Oracle Corp, and I know many efforts were made to sell (the product and/or the company) to Oracle in around 1997 timescale.
Oracle's own WarehouseBuilder must date from around then if not earlier.

Of course, Constellar wasn't alone at the time. We competed over time against ETI, Prism, Vmark/Ardent/Ascential, Informatica, and all the others (as well as the ubiquitous roll-your-own, which is still probably the market leader).

Slipping the litigants $2 million may be cheaper than arguing the case for Oracle - but for the rest of us (especially any cash strapped open source projects) it's not an option. And I'm sure a sharp lawyer at IBM might like a bit of ammunition to see them off. Hey IBM, you're innocent!

So if any ETL pioneer (or their heirs and successors) wants my testimony to fight off JuxtaComm's malicious and meritricious suit - just let me know! I'm available for testing out the trans-atlantic business class beds any time you like...

Update 13 March 2008: more from Vincent here.

RAC Stretch Clusters - a survey

2008-01-21T20:46:00.000Z

There's been an interesting digression on the Oracle-L list today on the subject of RAC stretch clusters.

A stretch cluster is one in which nodes are separated - possibly by several miles - mainly as a precaution against a complete data-centre outage. I know of a couple of examples where separations in the region of 20-30 miles (30-50 km) are either in use or planned, and other posters on Oracle-L have mentioned "several" or "a handful" of implementations worldwide.

Obviously, the main performance issue for a stretch cluster is the latency and bandwidth of the interconnect. The bandwidth isn't affected by distance, but the latency certainly is.

I'd be very interested to hear from anyone who has implemented a stretch cluster (in test or production) with as much detail as you are free to pass on, particularly:

what is the distance between sites
how many nodes at each site
is the workload evenly distributed (active/active homogeneous), partitioned (active/active, heterogeneous) or uneven (active/passive)
some indication of database size and transaction rates (in whatever units are meaningful to you
Any performance issues?

Either email me (nigel at preferisco dot com) or comment below. I would like to publish the results but let me know if any part(s) of your information is too sensitive to be broadcast, even anonymously.

Thanks in advance...

TiddlyWiki

2008-01-18T15:02:00.000Z

I thought I'd seen all the permutations of Wikis out there: written in php, stored as files; written in PL/SQL and stored in Oracle; and with a range of different markup conventions. And recently I've successfully used wiki style markup to better decorate output from Oracle Designer (of which more later in a separate post).

Today, in one of those serendipitious chances that comes from following a couple of irrelevant but intriguing blog links, I tripped over TiddlyWiki. Rather than giving you a server-side wiki, the whole thing is packaged on the client side. A TiddlyWiki is a single HTML file that contains its own code; which self-edits as you add content. You save the file locally (and can then publish to a website if you like).

As a tool written by web designers it just looks great. You don't just click and get another page; the expanded content (called a Tiddler) zooms out at you. You can choose to expand or collapse whichever tiddlers you like - at the same time. That may be a big advantage over more boring wiki implementations.

For large communities, it's probably all wrong - personal changes in effect fork the entire wiki; but it may be a very effective personal tool - kind of mind mapping on the cheap, but much prettier than editors like treepad (and instantly portable, no software install required - keep it on your USB stick).

Basic usage instructions can be found here - written as a TiddlyWiki so you can try it out. I'm going to see if I can turn a rather unexciting Designer entity report into a thing of beauty...

Latest Bloor market research for Data Migration

2007-12-19T20:52:00.000Z

You can find this survey either at Informatica http://www.informatica.com/info/bloorwpq407/ or at Bloor http://www.bloor-research.com/research/survey/876/data_migration_survey.html.

Generally sensible throughout but - unusually for Philip Howard's research output -it leaves some unanswered questions.

Even as a statistical forecasting effort, it does leave something to be desired. Rather a lot of tables have been included that don't contribute anythingto the sum of human knowledge - in particular a single CAGR (compound annual growth rate) of 10% appears to have been applied willy-nilly to every single region and sector (confirmed by retrofitting the figures -allowing for rounding to the nearest million). What a shame the research doesn't build on either Bloor or external figures for:-

likely frequency of application replacement (by industry: Banking hasa more rapid cycle than Trading Companies, and perhaps by context/application type)
industry/regional growth forecasts (eg from OECD)
No mention of government / public sector projects This is the biggest area for cockups, especially given the fragmented natureof PFI/PPI in the UK, and similar huckster schemes abroad.

However, the headline figures are worth absorbing, to give an order of magnitude for the market, and more importantly an idea of the extent of project failure.

84% of data migration projects "fail" (not delivered on time and on budget)
Cost overruns average 30%; time overruns average 40%
Data migration is a $5 billion market this year (or maybe twice that if you throw in smaller projects that come in under the radar for this survery) growing to $12 billion by 2012 (sounds like a lot - but London is spending more than that on the 2012 Olympics...)

At a more detailed level, I'm not entirely comfortable with the attempt at separating "data migration" as a market in its own right from related markets (ETL, EAI, etc). I would prefer to see DM as a particular use case for ETL, EAI and DQ tools; many of the same tools will also have applications for data integration, application integration, business intelligence and other use cases - although mileage may vary.

Howard states that DM only aplies to "one off" migrations and is distinct from data integration; well, yes and no. DM may have to take place over months or years; during that period, "source" and "target" systems will have to coexist somehow. That may be achieved through "temporary" data integration/replication, or by partitioning the data and gradually handing over slices from source to target. Given the ever-changing nature of business, it is by no means inconceivable that the process never reaches its originally planned conclusion (the final decommissioning of the source system). Parts of the "source" may turn out to be worth retaining (ie the economic case for replacement is not viable).

Perhaps the biggest concern I have is that there is no mention of DM as a subset of business change. DM scoping is often imposed by (sometimes poorly considered) "business" considerations and decisions. There needs to be a feedback loop from DM processes into the overall project feasibility, scoping, planning, and costing. How many DM budgets are over budget simply because the budget was unrealistic to start with - because project costing was done using some rule of thumb [eg see John Morris's data migration blog] that happened to be inadequate in the circumstances?

Finally, I am amused (and not at all surprised) to find that hand coding is the market leader at 30%, beating ETL into second place with 28%. Given that ETL tools are often given away with applications, databases, and maybe even with leading brands of Cornflakes it is amazing that this situation hasn't much changed in the last 15-20 years since ETL tools first appeared. I always thought that DIY (roll-your-own for some american speakers) was our (Constellar's) biggest competitor back then - and here's yet more anecdotal evidence. Informatica and other tool developers - not to mention application/product architects - need to understand why that should be:

no tool is a magic bullet: even a market leading ETL tool may make some cases easier, but can make a few (important) cases much harder to manage (if all you have is a hammer, everything looks like a nail)
tools are expensive to own: they can be expensive to buy, but much more importantly skilled tool users are expensive to train or hire. The migration skill-base is fragmented (what works for PowerCenter is wrong for Oracle Data Integrator; Ascential does things differently from Ab Initio).
data migration projects are seen as "boring": so may not always attract the youngest, thrustingest, enterpriseyest architects. Like support, a critical area is treated like a leper in some organisations. No wonder the outcomes are sometimes suboptimal. Those cast into the outer darkness of a migration project eventually either transfer internally (and let their skills degrade), or hop off into a consultancy or frelance contract to rent out their newfound skills. Far too rarely does a competence centre emerge.
application architects (for in-house and packaged apps) often forget the requirement for data migration into their whizzy new systems. Nearly twenty years after it was first released, Oracle Apps still doesn't have a fully supported bulk migration solution for even the most basic data (eg Payables 11.5.10 finally added vendors, sites and contacts but still doesn't support their bank accounts...). What's unglamourous for the end user is equally unglamourous for the product developer, it seems.

Ho ho ho! Happy Christmas everyone

I cite, you borrow, he steals...

2007-12-05T18:26:00.000Z

Anyone who appreciates the hard work that goes into a decent presentation or book should read Doug Burns' tale of woe. Isaac Newton is widely quoted acknowledging his debt to other scientists: "If I see further, it is because I am standing on the shoulder of giants". Don't let's tolerate those who seek to take the credit for another's efforts without due courtesy or recompense.

I find it particularly annoying as Doug (whose blog I read regularly, but whom I have never met in person) seems to be particularly (even pathologically?) careful to acknowledge where he gets his ideas from.

Take care to read the comments as well as the blog itself. This one will run and run...

Happy UKOUG by the way (from darkest Lancashire at the moment)!

Tombstone, AZ

2007-11-07T10:21:00.000Z

Just back from a fortnight in Arizona, where we were visiting #1 daughter on exchange at UofA in Tucson. We took in Flagstaff, Williams, the Grand Canyon and Sedona as well as sights in and around Tucson itself.

A personal favourite was the Boothill Graveyard on the outskirts of Tombstone. It seems to have operated only for a very few years - and most of the occupants apparently died a sudden death - shot or hanged in the main (one is specifically shown as 'died of natural causes').

George Johnson was more unlucky than most. I wonder how they found out their mistake?

Got an issue with that?

2007-10-17T11:11:00.000+01:00

Here's a little summary of issue management systems I found in the Register for use with (agile) software products. It struck a chord with me - not least because I am right now working with a project team of around 30 trying to manage and organise a range of issues on an ever changing series of spreadsheets, using email as the repository.

I could add the example issue tracking system that comes with Oracle Apex - has anyone tried that?

Other wikis with issue management templates / recipes are available - including PmWiki PITS.

Latest Gartner Magic Quadrant for Data Integration

2007-10-15T10:18:00.000+01:00

Informatica has helpfully syndicated the latest data integration magic quadrant. Out goes Ab Initio - apparently due to their perennial problems with secrecy (ie they won't tell anyone anything). Only IBM and Informatica make it to poll position (IBM slightly bolstered by its recent purchase of DataMirror). Hummingbird is the back marker, both for vision and ability to execute. Anyway, read it yourselves; no major surprises in there (and why would there be in such a mature market...).

Is there WebLogic in Oracle's bid for BEA?

2007-10-13T09:57:00.000+01:00

Finally, Larry has got out his wallet and made a bid for BEA( which of course they have rejected). This one has been waiting to happen for several years. Oracle's experience with app servers hasn't been entirely happy - but after a slow start they have managed to steamroller their way to a decent market share (often on the back of Oracle Apps); meanwhile most of the (often less well funded) pioneers have fallen by the wayside. Who remembers Persistence? Gemstone? Whatever happened to Sun's variously named products?

So who's left now? Oracle, IBM, BEA; MS IIS for the Netties, and a handful of low/no cost options led by JBoss (aka Red Hat).

Buying BEA presents an interesting marketing challenge in the J2EE app server area - though at least Oracle only has one app server at a time. Thing are much more exciting (ie confused) in the integration / add-ons market; Oracle Fusion is already in a state of upheaval, digesting bits and pieces from Siebel, Sunopsis, Hyperion and others. Throwing in BEA AquaLogic - which itself combines organic development and some BEA acquisitions, I seem to remember - will be "interesting" to say the least.

And don't forget other jewels at BEA. Tuxedo - a real piece of engineering - something Oracle could usefully combine with the best of its own server technology; and JRockit - a high performance JVM.

For those who hate acquisitions, remember: BEA bought in both JRockit (from a startup) and Tuxedo (originally developed at AT&T, bought in 1995)... and come to that, WebLogic itself (in 1998).

Onwards and upwards

2007-10-11T18:46:00.000+01:00

I've recently started a new contract which should prove very interesting. Starting off with a data migration study, I'm expecting there will be lots of performance planning and assurance to follow.

The setting is a rather large and unusual Oracle Apps implementation - client confidential of course. RAC, stretch cluster, massive batch processing, impossible response and resilience targets? Lots for me to (re)learn, and hopefully some useful previous experience to share.

The 250 mile weekly commute is a bit of a pain...

Bottoms up!

Cognos partners with Informatica

2007-09-11T08:26:00.000+01:00

Cognos and Informatica have announced a strategic relationship; Cognos will sell and support Informatica Data Quality and Data Explorer products, and the two companies will "team to jointly provide customers with data integration capabilities" with a focus on performance management (rather than the wider data integration market?).

Interesting that Informatica's front page still highlights the May press release about being granted an injunction against Business Objects, while Cognos includes this release on its front page. A minor difference in marketing communications, or a subtle indication of who is the senior partner?

Analysis from The Street here, and from Intelligent Enterprise here.

Finished at last (well, almost)

2007-09-01T17:52:00.000+01:00

It's finally complete - we're just back from a lovely break in Sant'Angelo at the new house. Not entirely a holiday - there was lots of cleaning to do, and dozens of lights to buy and wire in (more to come later, along with a significant amount of Ikea furniture to be trucked in and assembled - in November we hope). And I spent most mornings working (those pesky deadlines again) proving at least that the EDGE service (no UMTS in the mountains) and a 20 euro for 500Mb web deal from TIM were enough to keep me going.

The garden remains - a couple of 80 year old olives should have gone in this week, and work will start on the rest in the autumn.

Nothing to do now but work as hard as possible to pay the bills!

No more mirror, mirror jokes... IBM snaffles DataMirror

2007-07-19T13:14:00.001+01:00

IBM bought my former employers DataMirror earlier this week. Obviously their main interests will be the HA product line (a direct value add for them), and then the Transformation Server (TS) real time data integration components. Others have written here and here, so I won't labour that except to say that as part of the package they have won Constellar Hub (formerly dear old Information Junction - there's a phrase that doesn't escape my lips too often).

DataMirror never really "got" Constellar. Sure, I think they sweated the customers enough to make a return on their CDN$10million investment (what a steal that was!), but they never made a serious effort (afaik) to bring together the best of the Hub and TS. A shame, because I think it could have been a winner compared to the still rather tired 1990s EAI products like Informatica, DataStage, Ab Initio etc. In fact, TS+Hub could have turned out not unlike the Oracle Data Integrator (but with added "publish / subscribe", and strong IBM and Oracle partnerships).

Now the boot is on the other foot. Will IBM "get" DataMirror and its various product lines? All we can be reasonably sure of is that the Hub is now officially past its sell by date...

Am I glad I left DataMirror in 2001, and so didn't get bought again 6 years later? On balance, I think so. But I know some who have stayed all along - will they stay with IBM too?

Iona FUSEs Celtix and LogicBlaze

2007-07-09T18:52:00.000+01:00

Iona says it is expanding customers' open SOA choices by bringing together under the FUSE banner products that were formerly part of Celtix ESB, or were acquired with the LogicBlaze purchase this April.

CBR points out that Iona seems to be favouring LogicBlaze supported components (Apache projects ActiveMQ, ServiceMix and Camel) over Celtix components (apart from CXF, which has moved over from ObjectWeb to Apache). The LogicBlaze purchase must be working out well, methinks...

Talend - another open source ETL tool

2007-07-06T17:12:00.001+01:00

Bloor's Philip Howard writes at IT-Director about Talend Open Studio - unusual among ETL solutions in that it is a code generator (of Java, SQL, Perl) rather than an "engine" type of product.

As well as the usual drag-n-drop transformation GUI, this apparently supports business process modelling - which gives Talend a feature that many "real" ETL/EAI tools don't have. There's also support for using a server grid to parallelise processing, and there's an "on demand" SaaS offering.

Version 2.0 is now available, as well as a 2.1 release candidate which is said to add features including:

further optimizations for performance increase
support of new databases (including bulk load)
transaction management (connection sharing, commit and rollback)
Slowly Changing Dimensions support
MOM (Message Oriented Middleware) support for real-time integration jobs
fuzzy logic data matching (using the Levenshtein and metaphone algorithms)
normalization, denormalization and flow merge
support of SSH remote connections
support of PGP file decryption (through GPG binary)
reinforced support of the XML standard: DTD validation, XSD, XSLT transformation, significant improvements in hierarchical XML file generation, support of the XMLRPC Web Services protocol…
improvements to the tMap component, to support input filters and new joins types such (the cartesian product, first match, last match…)

Sounds like it might be worth a closer look...

Zoning in to Sonos

2007-07-06T16:21:00.000+01:00

We've finally bitten the bullet and shelled out for a 2 zone Sonos music system. I see we're not alone; Rob Levy, BEA's CTO, says it is his favourite gadget - and Joel Spolsky has also endorsed it.

We started around Christmas buying 2 Squeezeboxes - because I'm cheap, and because the Sonos system needs at least one box on wired Ethernet - which we didn't have handy in the right rooms. Although the Squeezebox looks good, it didn't really deliver as I had hoped.

The worst things about the Squeezebox:

it's not very good at coping with poor wireless reception - which probably accounts for most of the other problems

although you can synchronise two boxes (say one in the kitchen, and one in the living room) they tend to drift apart; not a good sound!

The slimserver software is incredibly slow at (re)indexing

the whole solution relies on slimserver software installed (by an importer) on a linux NAS fileserver - that makes getting updates much harder.

Internet radio was listed, but I could never get it to work

Documentation and help was rather flaky

The boxes look great but the UI is crummy - I never managed to explain it to my wife (or even to myself...)

So we cut our losses and invested in two Sonos ZP80s - they're quite small boxes, which feed into your existing amplifiers (you can use ZP100 if you want a built in amp). Rather than recable our house, we got a couple of powernet adapters. Music sits on a (simple) NAS disc - no intelligence required there.

We had some difficulty getting the powernet working - in the end after three hours trying every feasible combination of sockets in our (1st floor) office and (ground floor) living room, the very helpful installer swapped the original Netgear products for Microlink and found a pair of sockets that worked. Fantastic! after that everything was very straightforward. There's a (pricey) i-Pod like controller - or you can use a desktop interface. Once you've sussed the Zones and Music buttons, it's a piece of cake to navigate.

It's using the same music library as Windows Media Player - so I can load music once, and use it either on my headphones from the laptop, or on the Sonos system. Having the same music in two zones is easy - and very reliable. Adding new zones is also very simple (I just know we'll need a third zone in the office...). And internet radio works a treat - though with a slightly robotic quality, from the selection of sites supported (plenty of UK ones - helpful for us as we get almost nothing on DAB or FM).

The Squeezeboxes won't be entirely wasted; they'll go out to Italy this summer, where they should benefit - eventually - from being wired in (assuming the builders have remembered the CAT-5...).

Graphic example

2007-07-01T11:00:00.000+01:00

Thanks to Oracle AppsLab for referencing this stunning presentation by Hans Rosling. Not just for the superbly informative animated graphics, but also for the moving and thoughtful content (and the sting in the tail - watch all 19 minutes for the climax).

Could open source BI close out incumbents?

2007-06-29T09:02:00.000+01:00

Bloor's Philip Howard writes in The Register that Jaspersoft 2.0 is now a serious threat to mainstream BI (and ETL) vendors; a full enterprise license for $35k will knock spots off most of the competition.

Perhaps that helps to explain why the fizz has gone out of Cognos's recent results.

And it doesn't help that according to a survey sponsored by Sybase and reported by CBR, only 13% of UK BI projects work as expected. So naturally "31% of firms surveyed, particularly small and medium sized businesses, [are] keeping an especially close eye on keeping costs down to a manageable level."

To stakeholder (verb, transitive)

2007-06-28T12:16:00.000+01:00

We've all done it at some point in our scribblings; we use horrible new words just because we know that you know what we really mean.

One particularly fashionable word at the moment is stakeholder. It's a useful short-hand for a person or group that has an investment, share, or interest in something, as a business or industry - and which can include employees, customers, the local community - and (if necessary) Uncle Tom Cobbley and all.

Here's a new usage I found in a consultant's report recently:

Customers should ideally not be adversely affected ... and if downstream impacts are identified the customer should be stakeholdered ahead of time and those impacts managed accordingly.

From the context, I think the author means that someone should explain what's happening to the customer, and manage his (understandable) wrath. But rather than being good for the customer, being stakeholdered sounds more like getting out the garlic and the silver stake. Hold your head in shame, anonymous author!

IBM loses 80% of its Informix customer base

2007-06-25T18:37:00.000+01:00

Computer Business Review reports that IBM admits to losing 4 out of every 5 customers for the Informix database it bought back in 2001. That's 80,000 gone out of 100,000.

Where did they go - to DB2 (in which case IBM is happy, even if the Informix rump isn't), or to Oracle, MS or the open source competition?

There's a contrary opinion here from ComputerWorld. Informix may have been "IBM's dirty little secret" but CW believes that the recent "Cheetah" release, improved marketing and changes to senior management in IBM will add up to a better future. Oh, and those 20,000 customers turn into 20,000 members of the Informix User Group - so there may be more out there that CBR didn't count.

IBM acquires Telelogic; a Rational move?

2007-06-18T19:13:00.000+01:00

Interesting. IBM just announced the acquisition of Telelogic for around $745M. Telelogic is a Swedish company specialising in software configuration management; obviously their products will come under the Rational brand within IBM.

I spent much of the the last year as a Telelogic CM/Synergy user - having previously been more familiar with IBM Rational's own ClearCase product. They each have their strengths and weaknesses; but both suffer most from the naivety of their users and administrators. I've now worked on several sites who have proudly handed over some seriously enterprisey sum of money to the vendor, and then used one or other of these products when CVS or Subversion would have worked just as well (or maybe better). Why buy an all-singing, all-dancing tool, and then tie its hands behind its back?

Anyway, read what the Register says here and here.

The new house is coming on nicely

2007-06-15T17:40:00.000+01:00

Here's a view of our new venture Casa dei due Mori - the house of the two mulberries - under construction. We're certainly looking forward to spending lots of time there this summer when it should be finished. It's been a long trek to get this far - nearly four years from the thought to the finished article. I think it will be well worth the wait.

Ciao!

Buffered LOB_APPEND

2007-06-12T12:03:00.000+01:00

Jens Ulrik asked to see the code for a buffered LOB_APPEND mentioned in an earlier post LOB_APPEND eats resources....

Here it is as part of a package. Apologies for the rush job at the time; inelegant, but it was worth it...



    gi_chunk_length integer := 0;
    gv_chunk varchar2(32767);

    -- add a character value to the CLOB being built up; catch null values
  -- NOTE - final flush is done elsewhere
  --        and we assume max(length(av_value))  < ki_maxchunksize

    PROCEDURE lob_append (at_doc_clob IN OUT NOCOPY CLOB, av_value IN VARCHAR2)
    IS
     li_value_length integer;
    li_new_chunk_length integer;
    ki_maxchunksize constant integer := 32000;
    BEGIN
        -- prt_common_pkg.print_xml_string(av_value);
        IF av_value IS NOT NULL THEN
        li_value_length := length(av_value);
    li_new_chunk_length := gi_chunk_length + li_value_length;
        
    if li_new_chunk_length >= ki_maxchunksize 
    then
      -- pad out the chunk as much as possible
      gv_chunk := gv_chunk || substr(av_value,1,ki_maxchunksize - gi_chunk_length);
      -- write out
      DBMS_LOB.append (at_doc_clob, gv_chunk);
      -- initialise remainder into next chunk
      gi_chunk_length := li_new_chunk_length - ki_maxchunksize;
      gv_chunk := substr(av_value,li_value_length +1 - gi_chunk_length);
    else
      gv_chunk := gv_chunk || av_value;
      gi_chunk_length := li_new_chunk_length;
    end if;
        ELSE
            prt_common_pkg.LOG ('lob_append null ignored',
                                prt_common_pkg.klog_ret
                               );
        END IF;
    END;

The final flush of the last chunk of a LOB is called like this:


    
    IF gi_chunk_length > 0 THEN
       -- flush out the last chunk of the CLOB
     dbms_lob.append(at_doc_clob, gv_chunk);
     gi_chunk_length := 0;
     gv_chunk := null;
    END IF;

ETI offers data integration projects as a built-to-order service

2007-06-12T10:00:00.000+01:00

(Disclosure: I worked for Constellar, a competitor, from 1995 to 2001, and worked with - or perhaps I should say against - ETI*Extract at an Oracle customer site as long ago as 1994).

Bloor's Philip Howard reports that ETI is offering a 'built to order' service for data migration / integration, based on ETI Solution (formerly ETI*Extract). Just enter your requirements into their integration portal, or send them a Word document, and they claim to be able to deliver a generated solution at an average price of $25,000 - and they claim to guarantee a delivery time (usually just 2-4 weeks). For more info, see ETI Built To Order Integration.

In my opinion, ETI's products suffered poor sales in the past from being pricey and (compared to products like Informatica) difficult to use. The template-based generation (by no means unique to ETI - Constellar took a similar approach, as does Oracle Data Integrator) was too hard for most clients to customise. However, the fact that it is a pure generator (there's no runtime engine) helped ETI establish a niche supporting all kinds of legacy/mainframe data sources and targets, and now makes it possible for them to deliver generated code without leaving their IP behind.

Offering BTO seems like a good way to exploit the tool's underlying capabilities, and the skills ETI's own consultants have accumulated, without having to make the tool itself more (dare I say) user friendly.

One question - most integration projects don't know what their requirements really are until they start development and testing. How will this service deal with iterative development? What about late 'clarifications' - "oh, did I mention that the NOTES field got used to distinguish customer types during the early 80s; in the 90s it was used to hold foreign currency information for certain services..."?

So now let's see how this works in real life!

Open Source BI and ETL - picking up pace?

2007-06-09T12:44:00.000+01:00

A couple of recent stories to ponder on: SnapLogic raises $2.5M (admittedly from its founder's very own VC boutique) to build its LAMP based data services, while according to CBR, Ingres is preparing a Jaspersoft based BI/ETL appliance.

Meanwhile Pentaho has released its Data Integration 2.5 (formerly known as Kettle) and is showing at ODTUG what it calls "the world's most popular open source BI suite".

How to build a calculator - not!

2007-06-07T19:14:00.000+01:00

The wonderful Worse Than Failure (formerly wtf.com) has been running The Olympiad of Misguided Geeks at Worse Than Failure Programming Contest. It's an inspired idea: just how bad can you make a deliverable, while at the same time meeting the requirement.

Alex set contestants the task of building a calculator. Responses have included:

Driving it all through optical character recognition - my personal favourite so far

collecting inputs and simply sending them to CALC.EXE to do the work

one built on an extensible framework that allows you to add new digits. You want to add apples and oranges? well now you can!

All too good to miss - you can find them here, so enjoy!

Open source BI vendors get busy

2007-05-20T12:54:00.000+01:00

A useful article from Computer Business Review. Has anyone got any experience with Pentaho or Jaspersoft data integration tools being used in anger that they'd like to share?

Celona Evolve - "progressive integration" competitor for Oracle Data Integrator?

2007-05-19T09:26:00.000+01:00

A piece of work I've been doing has involved taking a cursory look at Celona Evolve, though sadly I haven't been in a position actually to give it a go.

Coincidentally, this week Bloor's Philip Howard has written about it at IT-Director.

For the Oracle-ite, it's positioned in a similar market segment to Oracle Data Integrator (formerly Sunopsis); providing traditional ETL but also talking up continuous (or "progressive") integration (EAI / SOA / EII - take your pick of acronyms).

So far, successful references for Celona are hard to find - always a problem for startups. And it's a UK company; my personal experience says that tends to be a disadvantage too. You may get a few good early adopters over here, but going to the States can be a killer. At present the product seems to rely heavily on Celona's accompanying services; they won't be able to grow properly until either the product is easier to use, or there's a large independent base of Celona skills in the market (as there is for other tools like Informatica, OWB, etc).

They're obviously gearing up for a more public stance; they "launched" on 8th May (hence the analyst briefing). Good luck, we'll see how things progress

Is big software now like big pharma?

2007-05-08T22:52:00.000+01:00

An interesting summary at Barrons of a VC panel at the Software 2007 conference. The worry is that big companies like MSFT, ORCL and IBM have so much CIO mindshare that there's none left for the little guys; the richer get richer and the poor (or small) are shut out.

Even the SaaS players trailing behind Salesforce are tiny; as Ted Schlein says “They are all gnats. They are all tiny. Not one would be a small division in one of these larger software companies.” (Actually, even Salesforce's (Q407) revenue is just $144M; our Larry could buy it from petty cash...).

Moving swiftly on

2007-05-08T22:07:00.000+01:00

I've reached the end of a nine month engagement, where I've been helping to maintain a 12 year old system; originally developed I suspect on Oracle7, and still running on Forms/Reports 6i. It's been interesting mainly as an example of how to start to rebuild a team's understanding of a system a team (mainly contractors) has been churned, original documentation is hopelessly outdated, and the institution's understanding of its own systems has been undermined.

We started to replace a LAN full of (flakey, unindexed, heavyweight) documents with a wiki-based repository of (current, hyperlinked, searchable, lightweight) 'bits of knowledge' that can be combined in various useful ways. One or two other groups at the client had already started to use PmWiki, so we built on that. When I get a chance I'll post specifically on how I found PmWiki as a tool, and how it compares with other wikis (TWiki and MoinMoin) that I've used in the past.

Now I've started with a new client, where I am advancing my career by working on a data migration study for a 20+ year old system, based on a pre-relational database, whose name of course I can't mention here. The good news is that the 'new technology stack' is firmly Oracle based; but the most interesting part will be dealing with the political and operational issues involved in migrating what are literally the core systems for this business.

On the way, there should be some interesting comparisons of EAI/ETL/EII tools; I hope we see Oracle Data Integrator put through its paces on the way. Certainly I should be able to post more on 'integration' generally for the next few months.

Sonic's Dave Chappell joins Oracle

2007-05-02T14:23:00.000+01:00

Oracle must be well pleased to pick up Dave Chappell, formerly a VP and chief evangelist at Progress Software's Sonic division (he's still on the management team there, according to this link). When that gets taken down, you can find his O'Reilly author profile here.

Dave will be development VP for SOA/ESB. I wonder which products that includes - Oracle Data Integrator perhaps? Good luck Dave, enjoy the ride at Oracle - I hope we meet again at another JavaOne some day.

PS, don't confuse him with any other Dave Chappell (like this one who runs an eponymous IT consultancy and speaks (for example) on BPEL, or this one who (I think it's safe to say) has no IT connection at all.

Java Caching for Oracle Apps 11i

2007-05-02T13:53:00.000+01:00

Steve Chan's article is about the use of caching interesting; now how long will it take for Oracle to put him together with the guys from Tangosol that they acquired in March?

Who knows, they could even finish off JSR 107: JCACHE - Java Temporary Caching API which Oracle submitted in 2001, and for which Tangosol's Cameron Purdy has been tech lead since sometime around 2002/3. Heck, most of the companies listed in the expert group have gone to the retirement home long since (names like Gemstone, Blusestone, iPlanet and SeeBeyond). Not bad for a project that was expected to take 12 weeks...

Toodle pip!

Joins over histograms

2007-04-26T08:26:00.000+01:00

Here's an awesome paper from Alberto Dell'Era and Wolfgang Breitling looking at "the formula used by the Cost Based Optimizer to estimate the cardinality of a [single-column] equijoin, when both the columns referenced in the join predicate have histograms collected."

Awesome because of the thought given to the presentation, and the care taken to test the results. Alberto has followed the lead taken by Jonathan Lewis's CBO book to carefully deconstruct and reverse engineer the algorithms used - with one or two surprising results.

Well worth a read...

Iona picks up LogicBlaze and its open source SOA platform

2007-04-13T17:43:00.000+01:00

The team at LogicBlaze has been putting together open source systems including Apache ActiveMQ (a JMS implementation), Apache ServiceMix (an ESB) and LogicBlaze FUSE (their own SOA platform).

Iona and LogicBlaze have been working together for a while, so there shouldn't be too many integration problems.

I wish the team, especially James Strachan and Robert Davies, former founders of (and my colleagues at) SpiritSoft, good luck in their new home. It's interesting that Iona - which contributed several to the Spiritsoft team around the turn of the millenium - should be their next resting place.

WebMethods sells itself to Software AG

2007-04-06T13:12:00.000+01:00

I said a couple of months ago that webMethods needed to shake up a bit. Well, now they've sold out to Software AG - originally best known in the 1980s for its Adabas database and Natural reporting, more recently a player in the XML / SOA world with Tamino and EntireX.

WebMethods was founded in 1996 and its IPO in 200 marked a high point of the EAI boom; an offer price of $35.00 turned into an opening of $195 and a first day close of $212. The price reached over $300 quite quickly, but since then has trended fairly steadily down to today's sub $10. Not a good investment overall!

Software AG wants this to be "a major step in [their] plans to more than double revenue to EUR 1 billion (USD $1.3 billion)." WebMethods' $200 odd million of annual revenue is a big - if barely profitable - step in that direction.

An article at Barrons notes that the share price is now trading above the offer - a sign that some in the market are expecting a higher bid (the analyst doesn't think they will be in luck).

The deal has pushed up Tibco's price too - on hopes that they will be the next target, perhaps.

Cache in the bank - Oracle buys Tangosol

2007-03-24T12:15:00.000Z

Oracle has picked up Tangosol - and its caching product Coherence - for the traditional undisclosed sum.

Well done Cameron Purdy. For a couple of years I worked at SpiritSoft, and we competed with Tangosol. Our engineers always said we had the better architecture - but Tangosol whipped our ass good anyhow. Cameron's team delivered a robust product and made it easy for people to buy - and the fact that they now claim 1500 implementations is testimony to lots of hard work all round.

We competed around the JCache JSR 107, which must be the slowest JSR to get to a review draft (and ironically, it was originally sponsored by Oracle). This 2002 TSS thread was already talking about excessive delays; since then very little seems to have happened - mind you the JCP (Java Community Process) site is having an off day, so perhaps it came out while I wasn't looking.

Soon after that TSS exchange Cameron joined the JSR 107 expert group, and IIRC he became the spec lead. But still no delivery. Perhaps now Oracle will push the standard some more? Anyway, all that competition is water long under the bridge, so let's recognise a winner, and as Cameron would say:

Peace

John Backus - father of FORTRAN - RIP

2007-03-22T13:04:00.000Z

As well as being the father of the first high level language suitable for numerical work - and of Algol 60 which can be taken as the root of most of today's widely used languages like C, C++ and Java - John Backus lives on as coauthor of Backus Naur Form (BNF) which we still use in the Oracle docs - for example:


relational_table ::= CREATE [GLOBAL TEMPORARY] TABLE [schema.]table
                       [(relational_properties)] 
                       [ON COMMIT {DELETE | PRESERVE} ROWS]
                       physical_properties
                       table_properties;

Vitria is now private; Iona buys C24; Informatica Integration on demand

2007-03-12T19:24:00.000Z

As I noted in October, Vitria is being taken private by its founders. Shareholders have now approved, and the transaction closed on 7th March.

This marks the end of a rollercoaster ride on NASDAQ (and by rollercoaster I mean there was a sharp climb at the beginning, but after a series of humps, bumps and loops you end up right back on the ground). Vitria's results over the last three years show everything gradually declining - revenue, assets, license sales. One positive - losses have also been reduced. Can Chang and Skeen turn the ship around, or is this just another step towards the sunset retirement home for distressed software companies?

Meanwhile Iona has picked up (London) City specialist integration boutique C24 for an undisclosed price. Given C24's small size - 12 employees - there should be no major digestion problem for Iona as long as C24's customers are kept sweet. Good luck to the C24 guys, a couple of whom I met while I was at SpiritSoft.

Finally, Informatica has launched the "first and only" on-demand data integration service. The "On Demand Data Replicator" - yours free for 30 days, and $1500/month from then on - is initially aimed at Salesforce.com customers; SaaS vendors like RightNow and NetSuite are next in line. The integration is (I guess) intended to be from your internal apps to your hosted apps, and vice versa.

Oracle captures Hyperion - a well planned thrust at SAP?

2007-03-01T20:31:00.000Z

Lots of buzz today about Orace's rumoured - then confirmed - acquisition of Hyperion (formerly known as, and still largely known for, the eponymous Essbase multi-dimensional database). Oracle bloggers like Mark Rittman have concentrated on the BI side of things, but it is worth remembering that Hyperion has also assembled a set of financial applications (mainly planning and modelling, as you’d expect) - not to mention a BPM product line. That could be very interesting as an add-on to Oracle Apps (not to mention PeopleSoft and Siebel).

Looking at their latest Q2 results, there’s no breakdown of revenue between product lines - but it’s interesting that the headcount is 1745 in Americas, 632 in EMEA and only 212 in APAC. Oracle’s wider/deeper international network could give a big boost to sales in Asia (as well as making admin savings at home). They'll keep the salesforce but dump the top-heavy administration.

The financial market seems reasonably positive about the news (see for example Barron's Eric Savitz. As well as providing a sell-up for Oracle Apps, Peoplesoft and Siebel, this can also get Oracle's foot further into the door at SAP sites.

"Hyperion is the latest move in our strategy to expand Oracle’s offerings to SAP customers,” said Oracle President Charles Phillips. "... Now Oracle's Hyperion software will be the lens through which SAP's most important customers view and analyze their underlying SAP ERP data."

There is also a feeling that Oracle got a good price, catching Hyperion with its share price down. The impact on other BI players like Business Objects and Cognos is mixed: M&A activity might be expected to push up their price - but their share prices already factored in a bit of a punt on Larry Ellison coming round to tea; now he's spent his money on Hyperion, the others may well fall back.

Good luck to the Oracle BPM, BI and Apps product managers trying to make sense of it all!

PS: Another good take here from Curt Monash at DBMS2

WebMethods/Infravio X-Registry - free 45 day trial download available

2007-02-28T13:08:00.000Z

Back in December, I asked would you pay $99000 for a starter pack.

Well, it looks like everyone else must have agreed, and not paid up. So webMethods has now announced a free trial download which "can be deployed for 45 days of testing and evaluation within a non-production environment".

After their recent results (see yesterday's post) I guess someone in marketing has decided it's about time to shake things up and create a few more leads for the salesforce to work on.

EAI results - a mixed bag

2007-02-27T20:40:00.000Z

Back to looking at financials, a note on Motley Fool that touts Tibco as a "king of cash" reminded me to poke around some of the EAI vendors' financial results.

Tibco's Q4 (ending November 06) was pretty spectacular, with revenue up 20% to $160 million - license revenue growth being a very healthy 32% to $88 million. But earnings over the entire year were barely up; Tibco seems to have a pattern of a huge Q4 after flat Q1/2/3.

WebMethods earnings show license income down more than 10% in Q3 to $19.7 million (total revenues $53.1 million). Motley Fool's article New Product, Same Problem suggests that digesting recent acquisitions (eg Infravio) and restructuring the salesforce are affecting sales.

Vitria's results show a startling spike in license revenue up to $6.7 million up from $1.8 million in the same quarter to Dec 2005 - around half of that accounted for by two customers. Encouragingly, Vitria is in the black.

Finally, BEA takes a bit of a beating - shares down 10% even though revenues were up 15% on same quarter last year. Why? because forecasts are down for next quarter (the analysts wanted $385 million, but management expects only $350-364 million).

A Morningstar analyst rounds it all off in the same news item by suggesting that BEA will get tough competition in the SOA space from Oracle, IBM and Tibco.

Unique IDs for multi-master replication - Sequence or SYS_GUID?

2007-02-26T20:15:00.000Z

Oracle-L is proving to be a good source of inspiration at the moment. Oracle ACE Syed Jaffar Hussain asked the question Is it possible to share a database sequence between multiple databases?

Using remote sequences
A couple of the replies took the question a bit too literally, and said yes, you can define a sequence on one database, and use it from another:


Select SequenceOwner.RemoteSeqName.NextVal@DBLinkName From dual

This works - and it can be made 'location transparent' by creating a local synonym for the remote sequence. But it is asymmetric (one database has to own the sequence) and it introduces a point of failure. If database 1 owns the sequence, database 2 can only insert rows if database 1 is actually available. If you could guarantee availability, why would you bother with replication?

Using a sequence local to each database
Several replies (including mine) suggested using carefully defined sequences which will deliver discrete values on two or more master databases. There are two basic patterns:

Partitioned: Suggested by several posters, the number space is divided up in blocks:


on db 1: create sequence myseq start with 1000000 max 1999999 
on db 2: create sequence myseq start with 2000000 max 2999999

A variant on this is to use positive numbers for database 1, and negative numbers for database 2.

Interleaved: The more popular option is to use the old trick of assigning odd numbers to database 1, and even numbers to database 2:


on db 1: create sequence myseq start with 1 increment by 2
on db 2: create sequence myseq start with 2 increment by 2

This mechanism is much easier to manage (essentially, it doesn't need any further management). It is also easy to extend to 3, 4, 27 or 127 masters - just set the "start with" to the database number, and "increment by" to the maximum anticipated number of databases required.

A third option was also proposed by Mark D Powell: use SYS_GUID(). That has some disadvantages:

SYS_GUID() is bigger (16 bytes on 9iR2) than a NUMBER (eg a 13 digit integer needs around 8 bytes). Obviously the extra space follows through to indexes, foreign keys etc.

Another is simply that it is a RAW, which has some possibly undesirable implications; for example several common tools (including SQL*Developer 1.0) can't directly display RAW values; you have to explicitly select RAWTOHEX(id_column).

On the other hand, SYS_GUID can be defined as the default value for a column - unlike NEXTVAL which is normally set in a trigger or directly as part of an insert into/select from statement. Worse, it is very common to see a separate SELECT seq.NEXTVAL FROM DUAL for every ID generated. I've never investigated the relative performance of SYS_GUID() against getting a sequence number - anyone else like to share that? That may well be the most important consideration of all.

So I think I'll be coming back to this subject in future.

Update - later the same day:

I've quickly timed a million iterations of select sys_guid() from dual, and a million iterations of select sequence.nextval from dual (Oracle XE 10.2.0.1, HP dv1665 Centrino Duo running Windows XP)


    select sys_guid()  into variable from dual: 89 seconds
    select seq.nextval into variable from dual: 40 seconds

For good measure, I've added a variant - one million calls of sys_guid() without a select:


    variable := sys_guid()                    : 95 seconds

One of the potential advantages of sys_guid() I had anticipated is that it could be called directly from PL/SQL - but it looks like the implementation is less efficient than it might be; tracing shows that the PL/SQL function recursively selects SYS_GUID() from dual.

So I think I'll stick to traditional and more convenient sequence.nextval for now.

Chin chin!

Avoiding application suicide by session pool

2007-02-25T16:18:00.000Z

A recent post on Oracle-L mentioned:

We had an issue affecting one of our production DBs. The middle tier application for some reason, went crazy spawning processes chewing up the DBs process parameter. DB started throwing errors indicating max process exceeded.

He went on to ask how to wake up PMON to clean up these processes - he'd had to bounce the database to get things cleaned up.

My response was to ask about the root cause, rather than the symptom. The poster may have been solving the wrong problem (or rather, after putting out the fire, he needed to find the cause and stop it happening again).

"Mid tier went crazy spawning processes" is often a symptom of session pool
madness. In such an application, X number of users share a smaller Y number of Oracle sessions. Everything tootles along happily; Users (midtier threads if you like) loop around:

get a session from the pool

issue one or two SQL

commit/rollback

give the session back

As long as the users spend less time in Oracle than they do in the rest of the
application (and waiting for user input etc), no problem.

Then something goes wrong; maybe a session sits on a lock that everyone needs; maybe a sequence cache isn't big enough (or is ordeed) and/or you forgot that We Don't Use RAC; maybe you had an SGA problem like ORA-4031.

What happens next:

all the Oracle sessions in the pool are busy

next midtier thread asks for an Oracle session

midtier pool manager says "no problem", launches a new Oracle session and adds it to the pool

that session becomes busy

and the next thread, and the next thread, and the next thread...

Soon instead of sharing say 100 Oracle sessions across 1000 processing threads,
your mid tier has responded to the blockage by adding 900 new sessions to the
load. That's probably made the problem worse, not better - kind of like slamming your foot on the accelerator when you see brakelights ahead in the fog.

I had exactly this problem performance last year, testing a J2EE app, using OC4J. We hit a 4031 problem (no bind variables in one part of the system) and then fairly immediately the application server did its lemming impersonation as described above.

Things to consider:
1) reduce the upper limit on the session pool size (definitely to below your Oracle processes level!)
2) if possible, slow down the rate of session starts (eg set a delay in the mid-tier session manager)
3) find out what caused the problem in the first case.

The good news is that if you dampen down this suicidal behaviour, you probably have a better chance of diagnosing the root cause next time.

Data warehouse appliance pros and cons

2007-02-21T07:50:00.000Z

Andy Hayler writes here about the marketing hype surrounding data warehouse appliances like Netezza, and warns that buyers should not just consider the 20% of a project budget that is typically spent on hardware and software. The other 80% goes on - well, people like us.

However the economic case for the Netezza model can still be made - rather easily. I’m aware of a POC where the capital costs of a 50TB Netezza solution are considerably less than the business is being charged back each year just for the cpus and discs supporting the equivalent Oracle DW. The clincher is that Netezza doesn’t require the same level of DBA expertise - it just hasn’t got any tuning knobs for us to fiddle with. Oh, and critical queries run up to 150 times faster ...

If these appliances result in a hefty capital saving and a substantial reduction in DBA/performance tuning overheads, then they will certainly continue to gain market share.

Google Oracle - Good start, now what about Metalink?

2007-02-14T20:41:00.000Z

The news is out - here from Eye on Oracle - that Oracle has finally opened the door to Google indexing of Oracle's documentation.

That's marvellous, but it's not exactly the fall of the Berlin wall. Many Oracle docs have been online and available for Google search for years - certainly for the database. Some of these have been (accidentally?) made available on customer websites (eg http://www.lc.leidenuniv.nl/awcourse/oracle/nav/docindex.htm - it's often universities) and others by Oracle themselves (eg http://download-east.oracle.com/docs/cd/B14117_01/nav/portal_3.htm). Just Google for key phrases like Browse the list of books, including PDF for printing for 9i or ADM link goes to the Administrator's Guide which gets you the 10g book list - and you'll see what I mean. I've included the 'repeat the search with omitted the results included' option.

What would be really useful is to open up Metalink itself (only customer SRs excepted). Too many Oracle developers need access, but can't get it because of short sighted officiousness. We developers (especially freelancers like me) often get the short straw - we're expected to be have all Oracle knowledge at our fingertips and yet we are effectively prevented from using one of the best available resources for it. Our employers just won't give us access to the support contract information necessary to get connected. BTW, I stress that the main fault is not Oracle's; it lies more with jobsworths (Brit expression I think - those people who say "it's more than my job's worth" as an excuse for anything).

Oracle Corp can and has made valid arguments for secrecy - mainly around its intellectual property rights and its competitors. But anyone who wants to indulge in industrial espionage can probably afford to buy a support license and get into Metalink anyway. The net effect is that the security defeats exactly the wrong group.

So go on Redwood - open up a bit more; you know you want to!

Schema models, FK constraints and Oracle Apps - a matter of entropy?

2007-02-12T20:31:00.000Z

There's been an interesting series of posts on Oracle-L recently, which started with a request for a schema model of Oracle Apps. Jared Still posted this response, which included the observation:

Oracle Apps - Not quite as sure about it, but I believe its origins
predate the use of referential integrity in the database.

I worked on Oracle (UK) Accounting v3 during mid-late 80s, some ideas from which (code combinations for example) were 'borrowed' for what became Oracle Financials. AIRC that was around 1987-8. HR development started a little later, and being UK based was a lot more CASE (Oracle Designer) inclined (they shared the same building). Manufacturing originated in the US consulting organisation, and again they were somewhat more methodological than the original Apps team in Redwood. In both cases I think initial development still pre-dated the implementation of effective foreign keys in Oracle 7. For Financials, even the option of FKs in the dictionary was not yet on the menu.

The (defensible) strategy to 'disintegrate' the applications (eg having separate GL, AP, AR modules in Financials) made it easier to get early releases out of the door - but at the cost of hiding relationships. And of course the whole Application Foundation ethos of configurable code combinations and flexfields means that many relationships are impossible to implement as Oracle serverside constraints out of the box.

Since then, I guess the normal product development priorities have reigned: functionality/saleability first, customer bug fixes second, and engineering/non-functional improvements last. Performance fixing priority goes up and down the scale according to the level of pain being felt by critical customers (Cary Millsap will remember performance testing of release 9, for example).

Just try to explain to a VP of Applications that you want to spend (managers prefer 'invest') tens of man-years documenting and 'improving' internals. It's like painting the Golden Gate bridge - once you start, you never stop. That budget has to come from somewhere - and the other priorities always seem more attractive. So there is a tendency to maximise entropy (btw that's one of the main reasons why startups can beat gorillas).

All the large ERPs seem to suffer the same problems - it's not just Oracle. But as one poster said - the more gotchas there are in the Apps, the more work for us... in the short term at least.

Happy obfuscating!

The more the meta - 'Intentional Software' meets reality?

2007-02-06T20:19:00.002Z

This quite long Technology Review article catches up with Charles Simonyi - Microsoft's former Chief Architect and the alleged inventor of Hungarian variable naming convention - who will be rocketing up to the International Space Station in April.

In his day job, he has been developing a 'new' concept now known as 'intentional software'; for every problem domain, developers will create generic tools which users can employ to "guide the software's future evolution".

Code generators, CASE tools and IDEs could all be seen as primitive examples of what he's on about. They also point to the possible fly in the ointment. As you approach the problem boundaries, any preconceived tool inevitably becomes increasingly inefficient; and the abstraction often makes it difficult or impossible for the user/developer to interfere productively. This is the law of leaky abstractions.

That's not to say that domain-specific generators are a bad thing per se; the template driven generator underlying Constellar Hub was a key component in its success, allowing us to quickly improve and extend the range of objects it could generate. When the tool runs out of steam, code developers can simply jump out of the tool and modify the generated code (sadly, that seems to be the permanent situation for most J2EE - more code and less declarative than any 1980s 4GL) or use a different abstraction.

But as far as Simonyi's work goes, you can count me in with the sceptics. Changing the representation of a problem (from textual code to a graphical parse tree, for example) does not in itself make the problem easier to solve. I saw my first 'visual' coding system in about 1981 - on a VT100 at that - and it was both brilliant and (without a mouse) completely **** useless at the same time. Still, it will be fun watching him throw all his money at it...

Ciao!

Kalido in a box?

2007-01-14T14:40:00.000Z

Bloor's Philip Howard writes in IT-Director.com that Kalido gets a competitor.

The problem Kalido addresses is dealing with change in a datawarehouse; for nearly 10 years Kalido has been the only company trying to The new kid on the block aiming to manage changing/evolving business models without having to completely rebuild your datawarehouse is BIReady, a small Dutch company (hmm, Kalido came out of Royal Dutch Shell ... do those Netherlanders know something we don't?) which claims to deliver 'fully model driven' data warehouse with support for history of change.

This is an area which has traditionally been dealt with by large numbers of staff and consultants. Kalido has managed to make some sales in large multi-national companies who value the benefits of managing not just evolving models, but multiple concurrently active models. BIReady looks too small to bite off these very large organisations (like Shell, Unilever etc) - but they may help to commoditise this part of the DW market, and perhaps make people think about model-driven warehousing further down the food chain.

One to watch.

Listen to the music

2007-01-06T20:03:00.000Z

I love listening to music. At work, I use my own laptop like a (rather large) iPod - its a great way to zone out of the open plan hubbub. Currently I'm just about to hit 6000 tracks, which I mostly have on permanent shuffle. We're looking into sound systems for our house in Italy and although I think the Sonos system (described here by Joel Spolsky) looks fab, I also tripped over the Slim Devices Squeezebox and wanted to have a go with that first, as it's a lot cheaper.

Essentially, we've moved our music collection onto a network attached file server (from Qnap, which conveniently comes with the Slimserver software pre-installed on Linux). You can continue to rip CDs using Windows Media Player (just pointing it at the new network directory).

Slimserver indexes the collection (very slowly - what's that all about?) and then you're ready to go.

Setting up the Squeezebox was easy - apart from revealing yet again what a poor wireless network we have. Eventually I convinced myself to replace the ADSL/wireless router, replacing a Belkin which has never been very reliable with a Netgear Rangemax, which was a doddle to set up and so far seems solid as a rock. The Squeezebox is in the living room, plugged into the home cinema's spare input (DVD, Sky+ and now this).

The Squeezebox looks fabulous - sleek and white and not too small (or large). The music reproduction is fine as far as my aging ears nnd cheap home cinema speakers can tell; and I easily can re-rip my CDs at a higher bit rate. However needing a separate remote is a bit of a pain (that now makes FIVE remotes to lose). That's where Sonos wins - they have a beautiful (massively expensive) wireless remote that can control all your Sonos Zone Players using an iPod like panel. The Squeezebox has a complete phonepad of function keys, and the interaction with the display is not entirely intuitive.

Anyway, it all works fine (as long as we balance the Squeezebox on a box of chocolates - it must be in the worst place in the house for wireless, as far as possible from the base station and practically behind the TV). So now I have to work out how to build playlists, and then decide if I want to fork out double or treble for Sonos at the house in Italy.

Choices, choices!

Would you pay $99,000 for a starter pack?

2006-12-13T21:55:00.000Z

WebMethods has launched a starter pack for Infravio X-Registry, which combines a UDDI registry, a JAXR repository, and what they coyly describe as integrated governance [which] ensures that these policies can be effectively applied across design-time, run-time and change-time environments.

For the bargain basement price of just $99,000 you get to minimize the upfront cost of initial adoption. Or to put it another way, you get 2 cpus, 5 named users and a UDDI registry that will allow you to define just 25 web services.

Why on earth would anyone want to pay that much to adopt a vendor-ized version of open standard technologies? Now if they had said this was a free community edition, they might be in with a chance of luring unsuspecting punters in and hitting them with the upgrade later. But this way round? Seems daft to me...

Informatica buys Itemfield, gets a sprinkle of Gartner's magic dust

2006-12-12T23:21:00.001Z

Informatica has agreed to buy Itemfield - an Isreali outfit specialising in integration of unstructured and semi-structured data. That adds a useful extra string to their bow.

Coincidentally Gartner's latest Magic Quadrant for Data Integration Tools has recently materialised. Informatica gets a positive write up - its stands alone with IBM in the leaders quadrant, head and shoulders above the pack (including Oracle).

Sadly my former Constellar colleagues at DataMirror don't even get on the map - DM just gets a rather dismissive one line mention for Transformation Server in the also-rans section. Is DataMirror under-marketing its product range - including Constellar Hub, or is Gartner just not too impressed with it?

Problems with Statspack

2006-12-08T08:38:00.000Z

Everyone's talking about it - statspack isn't perfect. Driven from Jonathan Lewis to Doug Burns to Daniel Fink's OptimalDBA I thought it was worth pointing out that as a snapshot technique, your knowledge of the value of the increment is uncertain.

Suppose snapshot N of metric has value s(N)
lets call the increment N to N+1 x(N)

All we can say for sure is:
A) if f(N+1) <>= f(N+1)
B) if f(N+1) >= f(N) then x(N) >= f(N+1) - f(N)

In normal use (when the shared pool is stable), we only see the case B, and we replace the >= with =. But if your shared pool is being trashed (see earlier posts) then remember uncertainty, and consider building case (A) into your reports.

Vitria goes private

2006-10-15T12:52:00.000+01:00

Once one of the great hopes of the EAI boom of the late 90s, Vitria has struggled over the last five years. Now I just noticed that they recently announced that they are going private.

The purchasers are Vitria founders Dale Skeen (current CEO) and his wife Jomei Chang (founding president and CEO). When Reuters bought Teknekron (which soon morphed into Tibco) they used their $10m to start up Vitria. As this 2001 story explains, Vitria's 1999 float was one of the more ebullient - priced at $16, hitting $273 just three months later.

Vitria was perhaps the first EAI company really to 'get' the idea of process level (rather than data) integration. Perhaps their biggest problem has been that their proprietary approach had hardly started to gain acceptance by the time industry standards - web services, service oriented architecture, BPEL and the rest - pulled the rug from under their feet.

Adjusted for stock splits, a peak price of $400 compares to a sale price of $2.75 per share. It will be very interesting to see what Chang and Skeen manage to make of the rump of their creation.