ETL patent case: Constellar and DataMirror let off off the hook; DataStage still in dock

Once again Vincent McBurney delivers a fantastic summary of the latest state of the Juxtacomm ETL patent case: SQL Server, DB2 and DataStage will fight out Data Integration Patent Infringement.

I'm most interested from the Constellar point of view - I first came across Constellar (then Information Junction) as a product on sale in late 1993 / early 1994 (before joining the company from Oracle in 1995), so it always seemed clear to me that it would qualify as prior art to Juxtacomm's 1998 patent. Oddly, it seems that the parties to the trial have agreed that Constellar Hub (and DataMirror Transformation Server) can be dropped from consideration; they won't be subject to damages - but equally they won't be considered as prior art. I don't understand that, but I guess the IBM lawyers must know what they are doing.

So, the case rumbles on, serving (if nothing else) to show how broken the US software patent system is.

Tibco RV in a box - would appliances help streaming SQL?

Several others have posted on the new Tibco Messaging Appliance - apparently it's Tibco Rendezvous (RV) in a box OEM'd from Solace Systems. Paul Vincent ponders at the Tibco CEP blog:

It’s quite feasible that the same approach could be used for “basic” complex event processing operations, especially those that don’t require history (or much persistence)

Paul is certainly right that CEP and event/data stream processing engines could benefit from the appliance approach. However I think that it is precisely those applications that require history that could most benefit. A high-speed streaming engine with a co-located database (running with much of the data in cache) gives the perfect environment for answering a wider range of "right-time" BI requirements; the engine can control exactly what needs to be retained and for how long to provide the appropriate time-series data to support whatever aggregated outputs are desired. The database acts as a memory extender (allowing infrequently accessed data to be paged out), as a source of replay / recovery data, and as a history for newly defined streaming queries. The latter is most important - if you want a 1 hour moving average, most streaming queries won't give you the right answer until the end of the first hour.

SQLstream launches v2.0 of its Event Stream Processing engine

As well as working as a freelance Oracle consultant, I have spent most of the last year working with SQLstream Inc on their Event Stream Processing engine, version 2.0 of which has now been launched.

My initial point of contact was Julian Hyde, with whom I worked in the Oracle CASE / Oracle Designer team in the early 90s. He went out to Oracle HQ and worked on bit-mapped indexes, then spent time at Broadbase before becoming best known as the founder-architect for the Mondrian OLAP server.

Event Stream Processing (ESP) is a development of what we might have called Active Database a few years ago. Rather than running queries against stored data as in an RDBMS, we register active queries which can be used to filter data arriving from any kind of stream. These active queries behave just like topic subscriptions in a message bus - except that unlike typical JMS implementations, the SQLstream query engine can do more than just filter. SQL queries against the data-in-flight can also:

• join data between multiple streams, or between streams and tables
• aggregate data within a stream to give rolling or periodic aggregates based on time or row counts
• apply built-in and user-defined functions to columns within the stream's rows
• build a network of streams and views to support complex transformation and routing requirements
  

Queries are defined using SQL:2003 with minimal extensions (there are some SQL examples on the web site); so developers used to working with an RDBMS will find it easy to grok the relational messaging approach. Unlike a typical message bus, most transformation and routing takes place entirely inside the SQLstream environment, rather than being delegated to external applications.

My experience working at JMS vendor SpiritSoft convinced me of the value of asynchronous message-based techniques, which in the last 10 years have spread out from high-end financial systems to ESB implementations all over the place. Since the early 80s we have seen how the RDBMS swept all other forms of structured data storage away. Now SQLstream's relational messaging approach removes the impedance mismatch between how we store data, and how we process it on the wire. In principle, this architecture can subsume both message-oriented (ESB style) and data-oriented (ETL style) integration architectures.

It should be said that "other ESP engines are available". Oracle itself has two projects on the go: its own CQL which I believe is still internal, and Oracle CEP (the rebranded BEA WebLogic Event Server - which itself is (or was) based on the open source Esper project). These two development threads will no doubt combine at some point (perhaps they already have?). IBM also has two or three independent CEP (complex event processing) projects on the go.

I think the same thing will happen to ESP / CEP as happened to ETL/EAI tools in the last ten or fifteen years. For sure, the database/application server vendors (especially Oracle and IBM) will sell plenty of this software within their respective client bases. An Oracle CEP system that was (or could be) tightly integrated with the RDBMS itself - maybe executing PL/SQL as well as Java functions - would be an easy sell. However multi-vendor sites will be interested in an agnostic / vendor-independent tool as a basis for their integration efforts. Just as Informatica has carved out a place for itself in competition with Oracle's ODI and OWB and IBM's DataStage, so SQLstream and other ESP vendors can fight for the common ground. It will be very interesting to see how it all turns out.

See the SQLstream product page for background, plus posts from Julian Hyde, CTO of SQLstream and Nicholas Goodman, Director of BI Solutions at Pentaho.

PS: here's another post from David Raab.

ESB consolidation - Progress buys Iona

I guess Iona was one of the first well known Irish software companies, and now it must be one of the last. It has succumbed to Progress Software for $106M. Paul Freemantle beat me to the news that Progress now owns or is a major committer for at least 4 different ESBs - Sonic, Artix, C24 and ActiveMQ/Camel/ServiceMix (plus Actional if you like) - hey, that's more than Oracle isn't it? (or maybe not...).

BEA Aqualogic broken up

The Register reports on how Oracle is going about the integration / breakup of BEA's Aqualogic SOA products (splitting them between ex-Stellent and Oracle Fusion product lines). Oracle's actions are described as "firm but fair".

MS subpoenas Oracle in Juxtacomm ETL patent case

Vincent McBurney summarises the latest developments in the Juxtacomm "we invented ETL, yeah really, give us all your money" patent case, which I have mentioned before. Dang, don't these "obvious" cases take a lot of time and effort to clear up.

Vitria brings Web 2.0 and high transaction rates to BPM with M3O

Vitria recently announced its new M3O product, which is claims to be the convergence of BPM, Web 2.0 and Event Processing. Bloor's Simon Holloway reviews it here, where he quotes Vitria saying:

• "BPM provides standards-based executable modelling (based on BPMN) on top of business knowledge Repository.
• Web 2.0 provides the rich user experience with zero footprint to enable a collaborative design environment.
• Event processing provides the support for rule and process definition and real-time runtime performance based on event driven architecture.
• Only when you combine these together do you get a fundamentally new user experience with multilayer visualization, collaborative modelling environment, business level abstractions and event management"

Leaping shamelessly onto a passing bandwagon, Vitria explains M3O as "think iPhone meets dashboards" (quoted from ebizQ). The idea is that the "iPhone coolness" of the Web 2.0 interface will remove the gap between business and IT people. Well, as long as it doesn't (like the iPhone) lock users into an expensive long term relationship...

This looks like the first fruits from the return of JoMei Chang as CEO last July and the decision to go private, executed last March.

Cape Clear-out

I just caught up with the month-old news that Irish ESB/SOA vendor Cape Clear has been bought by WorkDay, a supplier of ERP SaaS.

Of course, SaaS needs integration, and Cape Clear CEO Annrai O'Toole promises that they will be providing the necessary Integration-on-Demand - but it seems like one of the leading independent commercial vendors has now been marginalised.

Ronan Bradley, former CEO of PolarLake (also Irish - one of our partners at SpiritSoft, and later a client of mine) also worries where the ESB market is going.

Is this a general problem in the middleware market, or is it just 'cos they is Irish? Are small vendors being caught between the rock of large vendors and the hard place of open source? Or is this (as Steve Craggs suggests in a comment on Ronan's post) simply a result of Cape Clear's own hubris?

Patently ridiculous

SCO has had a pretty hostile press over the last few years as it morphed from a real software business into a patent mining outfit. Well, I've now come across another case of patent ambulance chasing over at Vincent McBurney's blog on ITToolbox. JuxtaComm and Teilhard versus IBM Microsoft Informatica SAP etc tells the story so far - basically that a small company is claiming it invented ETL, patented it in 1997, and can it now have loads-a-money if you please.

I followed Vince's link to Groklaw where InterSystems, one of the parties, is asking for help.

One of the parties is DataMirror (now part of IBM, and the purchaser of my former employer Constellar). As tne SQL Group, Constellar had an ETL system that seems to fit the key claims as early as 1994 (to my personal knowledge - it may have been on the market in 1993)

I'm stunned that (apparently) Oracle has already caved, handing over $2 million. One of the reasons I am so surprised is:

• I was actually working for Oracle UK in 1994 - the Constellar Hub (then known as Information Junction) was evaluated for one of my clients (who ended up purchasing ETI instead).
• Oracle UK was involved in quite a few pre-sales to Telecoms, Pharma and others over the period 94-2000.
• Oracle US consultants helped support the product in at least a few US clients.
• Constellar's US office was actually sublet from Oracle Corp, and I know many efforts were made to sell (the product and/or the company) to Oracle in around 1997 timescale.
• Oracle's own WarehouseBuilder must date from around then if not earlier.

Of course, Constellar wasn't alone at the time. We competed over time against ETI, Prism, Vmark/Ardent/Ascential, Informatica, and all the others (as well as the ubiquitous roll-your-own, which is still probably the market leader).

Slipping the litigants $2 million may be cheaper than arguing the case for Oracle - but for the rest of us (especially any cash strapped open source projects) it's not an option. And I'm sure a sharp lawyer at IBM might like a bit of ammunition to see them off. Hey IBM, you're innocent!

So if any ETL pioneer (or their heirs and successors) wants my testimony to fight off JuxtaComm's malicious and meritricious suit - just let me know! I'm available for testing out the trans-atlantic business class beds any time you like...

Update 13 March 2008: more from Vincent here.

Latest Gartner Magic Quadrant for Data Integration

Informatica has helpfully syndicated the latest data integration magic quadrant. Out goes Ab Initio - apparently due to their perennial problems with secrecy (ie they won't tell anyone anything). Only IBM and Informatica make it to poll position (IBM slightly bolstered by its recent purchase of DataMirror). Hummingbird is the back marker, both for vision and ability to execute. Anyway, read it yourselves; no major surprises in there (and why would there be in such a mature market...).

Is there WebLogic in Oracle's bid for BEA?

Finally, Larry has got out his wallet and made a bid for BEA( which of course they have rejected). This one has been waiting to happen for several years. Oracle's experience with app servers hasn't been entirely happy - but after a slow start they have managed to steamroller their way to a decent market share (often on the back of Oracle Apps); meanwhile most of the (often less well funded) pioneers have fallen by the wayside. Who remembers Persistence? Gemstone? Whatever happened to Sun's variously named products?

So who's left now? Oracle, IBM, BEA; MS IIS for the Netties, and a handful of low/no cost options led by JBoss (aka Red Hat).

Buying BEA presents an interesting marketing challenge in the J2EE app server area - though at least Oracle only has one app server at a time. Thing are much more exciting (ie confused) in the integration / add-ons market; Oracle Fusion is already in a state of upheaval, digesting bits and pieces from Siebel, Sunopsis, Hyperion and others. Throwing in BEA AquaLogic - which itself combines organic development and some BEA acquisitions, I seem to remember - will be "interesting" to say the least.

And don't forget other jewels at BEA. Tuxedo - a real piece of engineering - something Oracle could usefully combine with the best of its own server technology; and JRockit - a high performance JVM.

For those who hate acquisitions, remember: BEA bought in both JRockit (from a startup) and Tuxedo (originally developed at AT&T, bought in 1995)... and come to that, WebLogic itself (in 1998).

Cognos partners with Informatica

Cognos and Informatica have announced a strategic relationship; Cognos will sell and support Informatica Data Quality and Data Explorer products, and the two companies will "team to jointly provide customers with data integration capabilities" with a focus on performance management (rather than the wider data integration market?).

Interesting that Informatica's front page still highlights the May press release about being granted an injunction against Business Objects, while Cognos includes this release on its front page. A minor difference in marketing communications, or a subtle indication of who is the senior partner?

Analysis from The Street here, and from Intelligent Enterprise here.

No more mirror, mirror jokes... IBM snaffles DataMirror

IBM bought my former employers DataMirror earlier this week. Obviously their main interests will be the HA product line (a direct value add for them), and then the Transformation Server (TS) real time data integration components. Others have written here and here, so I won't labour that except to say that as part of the package they have won Constellar Hub (formerly dear old Information Junction - there's a phrase that doesn't escape my lips too often).

DataMirror never really "got" Constellar. Sure, I think they sweated the customers enough to make a return on their CDN$10million investment (what a steal that was!), but they never made a serious effort (afaik) to bring together the best of the Hub and TS. A shame, because I think it could have been a winner compared to the still rather tired 1990s EAI products like Informatica, DataStage, Ab Initio etc. In fact, TS+Hub could have turned out not unlike the Oracle Data Integrator (but with added "publish / subscribe", and strong IBM and Oracle partnerships).

Now the boot is on the other foot. Will IBM "get" DataMirror and its various product lines? All we can be reasonably sure of is that the Hub is now officially past its sell by date...

Am I glad I left DataMirror in 2001, and so didn't get bought again 6 years later? On balance, I think so. But I know some who have stayed all along - will they stay with IBM too?

Iona FUSEs Celtix and LogicBlaze

Iona says it is expanding customers' open SOA choices by bringing together under the FUSE banner products that were formerly part of Celtix ESB, or were acquired with the LogicBlaze purchase this April.

CBR points out that Iona seems to be favouring LogicBlaze supported components (Apache projects ActiveMQ, ServiceMix and Camel) over Celtix components (apart from CXF, which has moved over from ObjectWeb to Apache). The LogicBlaze purchase must be working out well, methinks...

Talend - another open source ETL tool

Bloor's Philip Howard writes at IT-Director about Talend Open Studio - unusual among ETL solutions in that it is a code generator (of Java, SQL, Perl) rather than an "engine" type of product.

As well as the usual drag-n-drop transformation GUI, this apparently supports business process modelling - which gives Talend a feature that many "real" ETL/EAI tools don't have. There's also support for using a server grid to parallelise processing, and there's an "on demand" SaaS offering.

Version 2.0 is now available, as well as a 2.1 release candidate which is said to add features including:

• further optimizations for performance increase
• support of new databases (including bulk load)
• transaction management (connection sharing, commit and rollback)
• Slowly Changing Dimensions support
• MOM (Message Oriented Middleware) support for real-time integration jobs
• fuzzy logic data matching (using the Levenshtein and metaphone algorithms)
• normalization, denormalization and flow merge
• support of SSH remote connections
• support of PGP file decryption (through GPG binary)
• reinforced support of the XML standard: DTD validation, XSD, XSLT transformation, significant improvements in hierarchical XML file generation, support of the XMLRPC Web Services protocol…
• improvements to the tMap component, to support input filters and new joins types such (the cartesian product, first match, last match…)

Sounds like it might be worth a closer look...

Could open source BI close out incumbents?

Bloor's Philip Howard writes in The Register that Jaspersoft 2.0 is now a serious threat to mainstream BI (and ETL) vendors; a full enterprise license for $35k will knock spots off most of the competition.

Perhaps that helps to explain why the fizz has gone out of Cognos's recent results.

And it doesn't help that according to a survey sponsored by Sybase and reported by CBR, only 13% of UK BI projects work as expected. So naturally "31% of firms surveyed, particularly small and medium sized businesses, [are] keeping an especially close eye on keeping costs down to a manageable level."

ETI offers data integration projects as a built-to-order service

(Disclosure: I worked for Constellar, a competitor, from 1995 to 2001, and worked with - or perhaps I should say against - ETI*Extract at an Oracle customer site as long ago as 1994).

Bloor's Philip Howard reports that ETI is offering a 'built to order' service for data migration / integration, based on ETI Solution (formerly ETI*Extract). Just enter your requirements into their integration portal, or send them a Word document, and they claim to be able to deliver a generated solution at an average price of $25,000 - and they claim to guarantee a delivery time (usually just 2-4 weeks). For more info, see ETI Built To Order Integration.

In my opinion, ETI's products suffered poor sales in the past from being pricey and (compared to products like Informatica) difficult to use. The template-based generation (by no means unique to ETI - Constellar took a similar approach, as does Oracle Data Integrator) was too hard for most clients to customise. However, the fact that it is a pure generator (there's no runtime engine) helped ETI establish a niche supporting all kinds of legacy/mainframe data sources and targets, and now makes it possible for them to deliver generated code without leaving their IP behind.

Offering BTO seems like a good way to exploit the tool's underlying capabilities, and the skills ETI's own consultants have accumulated, without having to make the tool itself more (dare I say) user friendly.

One question - most integration projects don't know what their requirements really are until they start development and testing. How will this service deal with iterative development? What about late 'clarifications' - "oh, did I mention that the NOTES field got used to distinguish customer types during the early 80s; in the 90s it was used to hold foreign currency information for certain services..."?

So now let's see how this works in real life!

Open Source BI and ETL - picking up pace?

A couple of recent stories to ponder on: SnapLogic raises $2.5M (admittedly from its founder's very own VC boutique) to build its LAMP based data services, while according to CBR, Ingres is preparing a Jaspersoft based BI/ETL appliance.

Meanwhile Pentaho has released its Data Integration 2.5 (formerly known as Kettle) and is showing at ODTUG what it calls "the world's most popular open source BI suite".

Open source BI vendors get busy

A useful article from Computer Business Review. Has anyone got any experience with Pentaho or Jaspersoft data integration tools being used in anger that they'd like to share?

Celona Evolve - "progressive integration" competitor for Oracle Data Integrator?

A piece of work I've been doing has involved taking a cursory look at Celona Evolve, though sadly I haven't been in a position actually to give it a go.

Coincidentally, this week Bloor's Philip Howard has written about it at IT-Director.

For the Oracle-ite, it's positioned in a similar market segment to Oracle Data Integrator (formerly Sunopsis); providing traditional ETL but also talking up continuous (or "progressive") integration (EAI / SOA / EII - take your pick of acronyms).

So far, successful references for Celona are hard to find - always a problem for startups. And it's a UK company; my personal experience says that tends to be a disadvantage too. You may get a few good early adopters over here, but going to the States can be a killer. At present the product seems to rely heavily on Celona's accompanying services; they won't be able to grow properly until either the product is easier to use, or there's a large independent base of Celona skills in the market (as there is for other tools like Informatica, OWB, etc).

They're obviously gearing up for a more public stance; they "launched" on 8th May (hence the analyst briefing). Good luck, we'll see how things progress