Open source BI vendors get busy

A useful article from Computer Business Review. Has anyone got any experience with Pentaho or Jaspersoft data integration tools being used in anger that they'd like to share?

Celona Evolve - "progressive integration" competitor for Oracle Data Integrator?

A piece of work I've been doing has involved taking a cursory look at Celona Evolve, though sadly I haven't been in a position actually to give it a go.

Coincidentally, this week Bloor's Philip Howard has written about it at IT-Director.

For the Oracle-ite, it's positioned in a similar market segment to Oracle Data Integrator (formerly Sunopsis); providing traditional ETL but also talking up continuous (or "progressive") integration (EAI / SOA / EII - take your pick of acronyms).

So far, successful references for Celona are hard to find - always a problem for startups. And it's a UK company; my personal experience says that tends to be a disadvantage too. You may get a few good early adopters over here, but going to the States can be a killer. At present the product seems to rely heavily on Celona's accompanying services; they won't be able to grow properly until either the product is easier to use, or there's a large independent base of Celona skills in the market (as there is for other tools like Informatica, OWB, etc).

They're obviously gearing up for a more public stance; they "launched" on 8th May (hence the analyst briefing). Good luck, we'll see how things progress

Is big software now like big pharma?

An interesting summary at Barrons of a VC panel at the Software 2007 conference. The worry is that big companies like MSFT, ORCL and IBM have so much CIO mindshare that there's none left for the little guys; the richer get richer and the poor (or small) are shut out.

Even the SaaS players trailing behind Salesforce are tiny; as Ted Schlein says “They are all gnats. They are all tiny. Not one would be a small division in one of these larger software companies.” (Actually, even Salesforce's (Q407) revenue is just $144M; our Larry could buy it from petty cash...).

Moving swiftly on

I've reached the end of a nine month engagement, where I've been helping to maintain a 12 year old system; originally developed I suspect on Oracle7, and still running on Forms/Reports 6i. It's been interesting mainly as an example of how to start to rebuild a team's understanding of a system a team (mainly contractors) has been churned, original documentation is hopelessly outdated, and the institution's understanding of its own systems has been undermined.

We started to replace a LAN full of (flakey, unindexed, heavyweight) documents with a wiki-based repository of (current, hyperlinked, searchable, lightweight) 'bits of knowledge' that can be combined in various useful ways. One or two other groups at the client had already started to use PmWiki, so we built on that. When I get a chance I'll post specifically on how I found PmWiki as a tool, and how it compares with other wikis (TWiki and MoinMoin) that I've used in the past.

Now I've started with a new client, where I am advancing my career by working on a data migration study for a 20+ year old system, based on a pre-relational database, whose name of course I can't mention here. The good news is that the 'new technology stack' is firmly Oracle based; but the most interesting part will be dealing with the political and operational issues involved in migrating what are literally the core systems for this business.

On the way, there should be some interesting comparisons of EAI/ETL/EII tools; I hope we see Oracle Data Integrator put through its paces on the way. Certainly I should be able to post more on 'integration' generally for the next few months.

Sonic's Dave Chappell joins Oracle

Oracle must be well pleased to pick up Dave Chappell, formerly a VP and chief evangelist at Progress Software's Sonic division (he's still on the management team there, according to this link). When that gets taken down, you can find his O'Reilly author profile here.

Dave will be development VP for SOA/ESB. I wonder which products that includes - Oracle Data Integrator perhaps? Good luck Dave, enjoy the ride at Oracle - I hope we meet again at another JavaOne some day.

PS, don't confuse him with any other Dave Chappell (like this one who runs an eponymous IT consultancy and speaks (for example) on BPEL, or this one who (I think it's safe to say) has no IT connection at all.

Java Caching for Oracle Apps 11i

Steve Chan's article is about the use of caching interesting; now how long will it take for Oracle to put him together with the guys from Tangosol that they acquired in March?

Who knows, they could even finish off JSR 107: JCACHE - Java Temporary Caching API which Oracle submitted in 2001, and for which Tangosol's Cameron Purdy has been tech lead since sometime around 2002/3. Heck, most of the companies listed in the expert group have gone to the retirement home long since (names like Gemstone, Blusestone, iPlanet and SeeBeyond). Not bad for a project that was expected to take 12 weeks...

Toodle pip!

Joins over histograms

Here's an awesome paper from Alberto Dell'Era and Wolfgang Breitling looking at "the formula used by the Cost Based Optimizer to estimate the cardinality of a [single-column] equijoin, when both the columns referenced in the join predicate have histograms collected."

Awesome because of the thought given to the presentation, and the care taken to test the results. Alberto has followed the lead taken by Jonathan Lewis's CBO book to carefully deconstruct and reverse engineer the algorithms used - with one or two surprising results.

Well worth a read...

Iona picks up LogicBlaze and its open source SOA platform

The team at LogicBlaze has been putting together open source systems including Apache ActiveMQ (a JMS implementation), Apache ServiceMix (an ESB) and LogicBlaze FUSE (their own SOA platform).

Iona and LogicBlaze have been working together for a while, so there shouldn't be too many integration problems.

I wish the team, especially James Strachan and Robert Davies, former founders of (and my colleagues at) SpiritSoft, good luck in their new home. It's interesting that Iona - which contributed several to the Spiritsoft team around the turn of the millenium - should be their next resting place.

WebMethods sells itself to Software AG

I said a couple of months ago that webMethods needed to shake up a bit. Well, now they've sold out to Software AG - originally best known in the 1980s for its Adabas database and Natural reporting, more recently a player in the XML / SOA world with Tamino and EntireX.

WebMethods was founded in 1996 and its IPO in 200 marked a high point of the EAI boom; an offer price of $35.00 turned into an opening of $195 and a first day close of $212. The price reached over $300 quite quickly, but since then has trended fairly steadily down to today's sub $10. Not a good investment overall!

Software AG wants this to be "a major step in [their] plans to more than double revenue to EUR 1 billion (USD $1.3 billion)." WebMethods' $200 odd million of annual revenue is a big - if barely profitable - step in that direction.

An article at Barrons notes that the share price is now trading above the offer - a sign that some in the market are expecting a higher bid (the analyst doesn't think they will be in luck).

The deal has pushed up Tibco's price too - on hopes that they will be the next target, perhaps.

Cache in the bank - Oracle buys Tangosol

Oracle has picked up Tangosol - and its caching product Coherence - for the traditional undisclosed sum.

Well done Cameron Purdy. For a couple of years I worked at SpiritSoft, and we competed with Tangosol. Our engineers always said we had the better architecture - but Tangosol whipped our ass good anyhow. Cameron's team delivered a robust product and made it easy for people to buy - and the fact that they now claim 1500 implementations is testimony to lots of hard work all round.

We competed around the JCache JSR 107, which must be the slowest JSR to get to a review draft (and ironically, it was originally sponsored by Oracle). This 2002 TSS thread was already talking about excessive delays; since then very little seems to have happened - mind you the JCP (Java Community Process) site is having an off day, so perhaps it came out while I wasn't looking.

Soon after that TSS exchange Cameron joined the JSR 107 expert group, and IIRC he became the spec lead. But still no delivery. Perhaps now Oracle will push the standard some more? Anyway, all that competition is water long under the bridge, so let's recognise a winner, and as Cameron would say:

Peace

John Backus - father of FORTRAN - RIP

As well as being the father of the first high level language suitable for numerical work - and of Algol 60 which can be taken as the root of most of today's widely used languages like C, C++ and Java - John Backus lives on as coauthor of Backus Naur Form (BNF) which we still use in the Oracle docs - for example:

relational_table ::= CREATE [GLOBAL TEMPORARY] TABLE [schema.]table
                       [(relational_properties)] 
                       [ON COMMIT {DELETE | PRESERVE} ROWS]
                       physical_properties
                       table_properties;

Vitria is now private; Iona buys C24; Informatica Integration on demand

As I noted in October, Vitria is being taken private by its founders. Shareholders have now approved, and the transaction closed on 7th March.

This marks the end of a rollercoaster ride on NASDAQ (and by rollercoaster I mean there was a sharp climb at the beginning, but after a series of humps, bumps and loops you end up right back on the ground). Vitria's results over the last three years show everything gradually declining - revenue, assets, license sales. One positive - losses have also been reduced. Can Chang and Skeen turn the ship around, or is this just another step towards the sunset retirement home for distressed software companies?

Meanwhile Iona has picked up (London) City specialist integration boutique C24 for an undisclosed price. Given C24's small size - 12 employees - there should be no major digestion problem for Iona as long as C24's customers are kept sweet. Good luck to the C24 guys, a couple of whom I met while I was at SpiritSoft.

Finally, Informatica has launched the "first and only" on-demand data integration service. The "On Demand Data Replicator" - yours free for 30 days, and $1500/month from then on - is initially aimed at Salesforce.com customers; SaaS vendors like RightNow and NetSuite are next in line. The integration is (I guess) intended to be from your internal apps to your hosted apps, and vice versa.

Oracle captures Hyperion - a well planned thrust at SAP?

Lots of buzz today about Orace's rumoured - then confirmed - acquisition of Hyperion (formerly known as, and still largely known for, the eponymous Essbase multi-dimensional database). Oracle bloggers like Mark Rittman have concentrated on the BI side of things, but it is worth remembering that Hyperion has also assembled a set of financial applications (mainly planning and modelling, as you’d expect) - not to mention a BPM product line. That could be very interesting as an add-on to Oracle Apps (not to mention PeopleSoft and Siebel).

Looking at their latest Q2 results, there’s no breakdown of revenue between product lines - but it’s interesting that the headcount is 1745 in Americas, 632 in EMEA and only 212 in APAC. Oracle’s wider/deeper international network could give a big boost to sales in Asia (as well as making admin savings at home). They'll keep the salesforce but dump the top-heavy administration.

The financial market seems reasonably positive about the news (see for example Barron's Eric Savitz. As well as providing a sell-up for Oracle Apps, Peoplesoft and Siebel, this can also get Oracle's foot further into the door at SAP sites.

"Hyperion is the latest move in our strategy to expand Oracle’s offerings to SAP customers,” said Oracle President Charles Phillips. "... Now Oracle's Hyperion software will be the lens through which SAP's most important customers view and analyze their underlying SAP ERP data."

There is also a feeling that Oracle got a good price, catching Hyperion with its share price down. The impact on other BI players like Business Objects and Cognos is mixed: M&A activity might be expected to push up their price - but their share prices already factored in a bit of a punt on Larry Ellison coming round to tea; now he's spent his money on Hyperion, the others may well fall back.

Good luck to the Oracle BPM, BI and Apps product managers trying to make sense of it all!

PS: Another good take here from Curt Monash at DBMS2

WebMethods/Infravio X-Registry - free 45 day trial download available

Back in December, I asked would you pay $99000 for a starter pack.

Well, it looks like everyone else must have agreed, and not paid up. So webMethods has now announced a free trial download which "can be deployed for 45 days of testing and evaluation within a non-production environment".

After their recent results (see yesterday's post) I guess someone in marketing has decided it's about time to shake things up and create a few more leads for the salesforce to work on.

EAI results - a mixed bag

Back to looking at financials, a note on Motley Fool that touts Tibco as a "king of cash" reminded me to poke around some of the EAI vendors' financial results.

Tibco's Q4 (ending November 06) was pretty spectacular, with revenue up 20% to $160 million - license revenue growth being a very healthy 32% to $88 million. But earnings over the entire year were barely up; Tibco seems to have a pattern of a huge Q4 after flat Q1/2/3.

WebMethods earnings show license income down more than 10% in Q3 to $19.7 million (total revenues $53.1 million). Motley Fool's article New Product, Same Problem suggests that digesting recent acquisitions (eg Infravio) and restructuring the salesforce are affecting sales.

Vitria's results show a startling spike in license revenue up to $6.7 million up from $1.8 million in the same quarter to Dec 2005 - around half of that accounted for by two customers. Encouragingly, Vitria is in the black.

Finally, BEA takes a bit of a beating - shares down 10% even though revenues were up 15% on same quarter last year. Why? because forecasts are down for next quarter (the analysts wanted $385 million, but management expects only $350-364 million).

A Morningstar analyst rounds it all off in the same news item by suggesting that BEA will get tough competition in the SOA space from Oracle, IBM and Tibco.

Unique IDs for multi-master replication - Sequence or SYS_GUID?

Oracle-L is proving to be a good source of inspiration at the moment. Oracle ACE Syed Jaffar Hussain asked the question Is it possible to share a database sequence between multiple databases?

Using remote sequences
A couple of the replies took the question a bit too literally, and said yes, you can define a sequence on one database, and use it from another:

Select SequenceOwner.RemoteSeqName.NextVal@DBLinkName From dual

This works - and it can be made 'location transparent' by creating a local synonym for the remote sequence. But it is asymmetric (one database has to own the sequence) and it introduces a point of failure. If database 1 owns the sequence, database 2 can only insert rows if database 1 is actually available. If you could guarantee availability, why would you bother with replication?

Using a sequence local to each database
Several replies (including mine) suggested using carefully defined sequences which will deliver discrete values on two or more master databases. There are two basic patterns:

Partitioned: Suggested by several posters, the number space is divided up in blocks:

on db 1: create sequence myseq start with 1000000 max 1999999 
on db 2: create sequence myseq start with 2000000 max 2999999

A variant on this is to use positive numbers for database 1, and negative numbers for database 2.

Interleaved: The more popular option is to use the old trick of assigning odd numbers to database 1, and even numbers to database 2:

on db 1: create sequence myseq start with 1 increment by 2
on db 2: create sequence myseq start with 2 increment by 2

This mechanism is much easier to manage (essentially, it doesn't need any further management). It is also easy to extend to 3, 4, 27 or 127 masters - just set the "start with" to the database number, and "increment by" to the maximum anticipated number of databases required.

A third option was also proposed by Mark D Powell: use SYS_GUID(). That has some disadvantages:

• SYS_GUID() is bigger (16 bytes on 9iR2) than a NUMBER (eg a 13 digit integer needs around 8 bytes). Obviously the extra space follows through to indexes, foreign keys etc.


• Another is simply that it is a RAW, which has some possibly undesirable implications; for example several common tools (including SQL*Developer 1.0) can't directly display RAW values; you have to explicitly select RAWTOHEX(id_column).

On the other hand, SYS_GUID can be defined as the default value for a column - unlike NEXTVAL which is normally set in a trigger or directly as part of an insert into/select from statement. Worse, it is very common to see a separate SELECT seq.NEXTVAL FROM DUAL for every ID generated. I've never investigated the relative performance of SYS_GUID() against getting a sequence number - anyone else like to share that? That may well be the most important consideration of all.

So I think I'll be coming back to this subject in future.

Update - later the same day:

I've quickly timed a million iterations of select sys_guid() from dual, and a million iterations of select sequence.nextval from dual (Oracle XE 10.2.0.1, HP dv1665 Centrino Duo running Windows XP)

    select sys_guid()  into variable from dual: 89 seconds
    select seq.nextval into variable from dual: 40 seconds

For good measure, I've added a variant - one million calls of sys_guid() without a select:

    variable := sys_guid()                    : 95 seconds 

One of the potential advantages of sys_guid() I had anticipated is that it could be called directly from PL/SQL - but it looks like the implementation is less efficient than it might be; tracing shows that the PL/SQL function recursively selects SYS_GUID() from dual.

So I think I'll stick to traditional and more convenient sequence.nextval for now.

Chin chin!

Avoiding application suicide by session pool

A recent post on Oracle-L mentioned:

We had an issue affecting one of our production DBs. The middle tier application for some reason, went crazy spawning processes chewing up the DBs process parameter. DB started throwing errors indicating max process exceeded.

He went on to ask how to wake up PMON to clean up these processes - he'd had to bounce the database to get things cleaned up.

My response was to ask about the root cause, rather than the symptom. The poster may have been solving the wrong problem (or rather, after putting out the fire, he needed to find the cause and stop it happening again).

"Mid tier went crazy spawning processes" is often a symptom of session pool
madness. In such an application, X number of users share a smaller Y number of Oracle sessions. Everything tootles along happily; Users (midtier threads if you like) loop around:

• get a session from the pool


• issue one or two SQL


• commit/rollback


• give the session back

As long as the users spend less time in Oracle than they do in the rest of the
application (and waiting for user input etc), no problem.

Then something goes wrong; maybe a session sits on a lock that everyone needs; maybe a sequence cache isn't big enough (or is ordeed) and/or you forgot that We Don't Use RAC; maybe you had an SGA problem like ORA-4031.

What happens next:

• all the Oracle sessions in the pool are busy


• next midtier thread asks for an Oracle session


• midtier pool manager says "no problem", launches a new Oracle session and adds it to the pool


• that session becomes busy


• and the next thread, and the next thread, and the next thread...

Soon instead of sharing say 100 Oracle sessions across 1000 processing threads,
your mid tier has responded to the blockage by adding 900 new sessions to the
load. That's probably made the problem worse, not better - kind of like slamming your foot on the accelerator when you see brakelights ahead in the fog.

I had exactly this problem performance last year, testing a J2EE app, using OC4J. We hit a 4031 problem (no bind variables in one part of the system) and then fairly immediately the application server did its lemming impersonation as described above.

Things to consider:
1) reduce the upper limit on the session pool size (definitely to below your Oracle processes level!)
2) if possible, slow down the rate of session starts (eg set a delay in the mid-tier session manager)
3) find out what caused the problem in the first case.

The good news is that if you dampen down this suicidal behaviour, you probably have a better chance of diagnosing the root cause next time.

Data warehouse appliance pros and cons

Andy Hayler writes here about the marketing hype surrounding data warehouse appliances like Netezza, and warns that buyers should not just consider the 20% of a project budget that is typically spent on hardware and software. The other 80% goes on - well, people like us.

However the economic case for the Netezza model can still be made - rather easily. I’m aware of a POC where the capital costs of a 50TB Netezza solution are considerably less than the business is being charged back each year just for the cpus and discs supporting the equivalent Oracle DW. The clincher is that Netezza doesn’t require the same level of DBA expertise - it just hasn’t got any tuning knobs for us to fiddle with. Oh, and critical queries run up to 150 times faster ...

If these appliances result in a hefty capital saving and a substantial reduction in DBA/performance tuning overheads, then they will certainly continue to gain market share.

Google Oracle - Good start, now what about Metalink?

The news is out - here from Eye on Oracle - that Oracle has finally opened the door to Google indexing of Oracle's documentation.

That's marvellous, but it's not exactly the fall of the Berlin wall. Many Oracle docs have been online and available for Google search for years - certainly for the database. Some of these have been (accidentally?) made available on customer websites (eg http://www.lc.leidenuniv.nl/awcourse/oracle/nav/docindex.htm - it's often universities) and others by Oracle themselves (eg http://download-east.oracle.com/docs/cd/B14117_01/nav/portal_3.htm). Just Google for key phrases like Browse the list of books, including PDF for printing for 9i or ADM link goes to the Administrator's Guide which gets you the 10g book list - and you'll see what I mean. I've included the 'repeat the search with omitted the results included' option.

What would be really useful is to open up Metalink itself (only customer SRs excepted). Too many Oracle developers need access, but can't get it because of short sighted officiousness. We developers (especially freelancers like me) often get the short straw - we're expected to be have all Oracle knowledge at our fingertips and yet we are effectively prevented from using one of the best available resources for it. Our employers just won't give us access to the support contract information necessary to get connected. BTW, I stress that the main fault is not Oracle's; it lies more with jobsworths (Brit expression I think - those people who say "it's more than my job's worth" as an excuse for anything).

Oracle Corp can and has made valid arguments for secrecy - mainly around its intellectual property rights and its competitors. But anyone who wants to indulge in industrial espionage can probably afford to buy a support license and get into Metalink anyway. The net effect is that the security defeats exactly the wrong group.

So go on Redwood - open up a bit more; you know you want to!

Schema models, FK constraints and Oracle Apps - a matter of entropy?

There's been an interesting series of posts on Oracle-L recently, which started with a request for a schema model of Oracle Apps. Jared Still posted this response, which included the observation:

Oracle Apps - Not quite as sure about it, but I believe its origins
predate the use of referential integrity in the database.

I worked on Oracle (UK) Accounting v3 during mid-late 80s, some ideas from which (code combinations for example) were 'borrowed' for what became Oracle Financials. AIRC that was around 1987-8. HR development started a little later, and being UK based was a lot more CASE (Oracle Designer) inclined (they shared the same building). Manufacturing originated in the US consulting organisation, and again they were somewhat more methodological than the original Apps team in Redwood. In both cases I think initial development still pre-dated the implementation of effective foreign keys in Oracle 7. For Financials, even the option of FKs in the dictionary was not yet on the menu.

The (defensible) strategy to 'disintegrate' the applications (eg having separate GL, AP, AR modules in Financials) made it easier to get early releases out of the door - but at the cost of hiding relationships. And of course the whole Application Foundation ethos of configurable code combinations and flexfields means that many relationships are impossible to implement as Oracle serverside constraints out of the box.

Since then, I guess the normal product development priorities have reigned: functionality/saleability first, customer bug fixes second, and engineering/non-functional improvements last. Performance fixing priority goes up and down the scale according to the level of pain being felt by critical customers (Cary Millsap will remember performance testing of release 9, for example).

Just try to explain to a VP of Applications that you want to spend (managers prefer 'invest') tens of man-years documenting and 'improving' internals. It's like painting the Golden Gate bridge - once you start, you never stop. That budget has to come from somewhere - and the other priorities always seem more attractive. So there is a tendency to maximise entropy (btw that's one of the main reasons why startups can beat gorillas).

All the large ERPs seem to suffer the same problems - it's not just Oracle. But as one poster said - the more gotchas there are in the Apps, the more work for us... in the short term at least.

Happy obfuscating!