Friday, February 15, 2008

Writing XML to a file

Marco Gralike has posted a couple of items at the Amis blog and a later followup on his own blog about ways of writing XML to a file - comparing the use of CLOB2FILE with DBMS_XMLDOM.WRITETOFILE with DBMS_XSLPROCESSOR.CLOB2FILE - and finding the latter faster.

Surely this is simply because the input to the test is an XMLTYPE (the result of his test query).
  • The call to CLOB2FILE requires an implicit conversion from XMLTYPE to CLOB and then simply serialising the CLOB (a large string already in XML format)
  • WRITETOFILE is making an implicit conversion from XMLTYPE to DOMDOCUMENT - it is constructing a DOM tree and then walking it to produce the serialised output. Here's the signature for WRITETOFILE:

DBMS_XMLDOM.WRITETOFILE(
doc IN DOMDOCUMENT,
fileName IN VARCHAR2,
charset IN VARCHAR2);
Building that tree is a major overhead - in this case - though obviously it wouldn't be if you actually needed to navigate around the tree adding/moving/updating or pruning nodes and branches.

Update 16-Feb:
Marco has continued the story here, in great detail and at great length (tkprof listings and all). He's particularly interested in file size; I'd be more interested in performance impact of the various steps in his operations. I will try to reproduce his tests and summarise those soon.

Nominative determinism

Nominative determinism: when your job or hobby matches your name. For example, the famous Major Major, from Catch 22. Or the British Antarctic Survey scientist I heard on Radio 4 yesterday explaining how he was going to drill down through 3km of ice - a Mr Core (or was it Bore - they both work, and I was in the car at the time).

The latest to catch my eye was in this quote at El Reg: "The vice president in charge of Red Hat's JBoss middleware Craig Muzilla, said Red Hat is going to undercut giants IBM, Oracle and Microsoft in the saturated and expensive middleware sector."

With a name like Muzilla, I guess working for an OSS company was pretty much inevitable.

DATAllegro, DW appliances and the Oracle lock-in effect

Andy Hayler has an interesting post on DATAllegro - talking about its sweet spot at the very high end of data volumes; apparently helped by a grid with up to 1TB/minute transfer speeds.

What caught my eye was Andy's observation that "Oracle customers can be a harder sell (than Teradata's, for example) not because of the technology, but because of the weight of stored procedures and triggers ... in Oracle's proprietary extension" (ie PL/SQL).

Just looking at the various technologies people talk about on OraNA blogs, I wonder whether the continuing growth in Java and .NET based applications over the last decade - where many of the business rules are executed outside the database - might hurt Oracle precisely by removing the PL/SQL "stickiness" Andy describes.

Admittedly Apex is a counter example - thoroughly PL/SQL, thoroughly Oracle; no chance of moving Apex clients to another RDBMS yet. And the tendency to provide management interfaces through PL/SQL (or very Oracle-specific SQL extensions) also helps to keep Oracle customers tied down. Which is good for us Oracle died-in-the-wool crowd. Whether the lock-in is good for the customers (or even Oracle) is another question; an existing customer lock-in can feel like a lock-out to new customers.

Any thoughts out there?

Regards Nigel

Tuesday, February 05, 2008

Schema Version Control

An outbreak of synchronicity has produced a volley of blogs and questions about database version control in the last few days.
I seen so many projects take code control seriously but have no real idea how to control a database definition (let alone how to control changes to multiple instances of development, test and live data, both in house and on client sites).

Code control is a walk in the park compared to "schema control" (where the delta , like an ALTER TABLE needs to be developed alongside the new version of the database object creation script).

And "data control" adds even more challenges:
  • can you upgrade the data in place? or do you need to create a replacement object?
  • Is there enough space?
  • Do you need to suspend user access? How long will it take?
  • what are the chances of reversing the upgrade if something goes wrong?
  • Will you even know if something goes wrong?
  • Who decides go forward, go back or muddle through?

A complex application version upgrade should be treated with as much respect as a data migration project. There should be a strategy (sure, it may be a standard strategy). Use standard tools / scripts. The more the process is repeatable - and the more it is repeated - the better. Make sure that all likely error conditions (missing log files, out of DB storage, non-writeable working directories, no listener) are tested for - test the tests!

By the way, the "process" includes all the operating procedures. Make sure that upgrades aren't nannied through by developers. Script the whole thing; have the production DBAs try it out the procedures as early as possible in test to make sure that they understand what's supposed to be happening (and how to know if it worked OK; the best thing is if the script just says:

IT ALL WORKED OK

or

SOMETHING WENT WRONG IN STEP 13 - SEE LOGFILE xxxyyy13

By the time the schema-and-data upgrade is executed for real, nothing - not even the unexpected - should be unexpected.

Monday, February 04, 2008

Patently ridiculous

SCO has had a pretty hostile press over the last few years as it morphed from a real software business into a patent mining outfit. Well, I've now come across another case of patent ambulance chasing over at Vincent McBurney's blog on ITToolbox. JuxtaComm and Teilhard versus IBM Microsoft Informatica SAP etc tells the story so far - basically that a small company is claiming it invented ETL, patented it in 1997, and can it now have loads-a-money if you please.

I followed Vince's link to Groklaw where InterSystems, one of the parties, is asking for help.

One of the parties is DataMirror (now part of IBM, and the purchaser of my former employer Constellar). As tne SQL Group, Constellar had an ETL system that seems to fit the key claims as early as 1994 (to my personal knowledge - it may have been on the market in 1993)

I'm stunned that (apparently) Oracle has already caved, handing over $2 million. One of the reasons I am so surprised is:
  • I was actually working for Oracle UK in 1994 - the Constellar Hub (then known as Information Junction) was evaluated for one of my clients (who ended up purchasing ETI instead).
  • Oracle UK was involved in quite a few pre-sales to Telecoms, Pharma and others over the period 94-2000.
  • Oracle US consultants helped support the product in at least a few US clients.
  • Constellar's US office was actually sublet from Oracle Corp, and I know many efforts were made to sell (the product and/or the company) to Oracle in around 1997 timescale.
  • Oracle's own WarehouseBuilder must date from around then if not earlier.

Of course, Constellar wasn't alone at the time. We competed over time against ETI, Prism, Vmark/Ardent/Ascential, Informatica, and all the others (as well as the ubiquitous roll-your-own, which is still probably the market leader).

Slipping the litigants $2 million may be cheaper than arguing the case for Oracle - but for the rest of us (especially any cash strapped open source projects) it's not an option. And I'm sure a sharp lawyer at IBM might like a bit of ammunition to see them off. Hey IBM, you're innocent!

So if any ETL pioneer (or their heirs and successors) wants my testimony to fight off JuxtaComm's malicious and meritricious suit - just let me know! I'm available for testing out the trans-atlantic business class beds any time you like...

Update 13 March 2008: more from Vincent here.

Monday, January 21, 2008

RAC Stretch Clusters - a survey

There's been an interesting digression on the Oracle-L list today on the subject of RAC stretch clusters.

A stretch cluster is one in which nodes are separated - possibly by several miles - mainly as a precaution against a complete data-centre outage. I know of a couple of examples where separations in the region of 20-30 miles (30-50 km) are either in use or planned, and other posters on Oracle-L have mentioned "several" or "a handful" of implementations worldwide.

Obviously, the main performance issue for a stretch cluster is the latency and bandwidth of the interconnect. The bandwidth isn't affected by distance, but the latency certainly is.

I'd be very interested to hear from anyone who has implemented a stretch cluster (in test or production) with as much detail as you are free to pass on, particularly:
  • what is the distance between sites
  • how many nodes at each site
  • is the workload evenly distributed (active/active homogeneous), partitioned (active/active, heterogeneous) or uneven (active/passive)
  • some indication of database size and transaction rates (in whatever units are meaningful to you
  • Any performance issues?

Either email me (nigel at preferisco dot com) or comment below. I would like to publish the results but let me know if any part(s) of your information is too sensitive to be broadcast, even anonymously.

Thanks in advance...

Friday, January 18, 2008

TiddlyWiki

I thought I'd seen all the permutations of Wikis out there: written in php, stored as files; written in PL/SQL and stored in Oracle; and with a range of different markup conventions. And recently I've successfully used wiki style markup to better decorate output from Oracle Designer (of which more later in a separate post).

Today, in one of those serendipitious chances that comes from following a couple of irrelevant but intriguing blog links, I tripped over TiddlyWiki. Rather than giving you a server-side wiki, the whole thing is packaged on the client side. A TiddlyWiki is a single HTML file that contains its own code; which self-edits as you add content. You save the file locally (and can then publish to a website if you like).

As a tool written by web designers it just looks great. You don't just click and get another page; the expanded content (called a Tiddler) zooms out at you. You can choose to expand or collapse whichever tiddlers you like - at the same time. That may be a big advantage over more boring wiki implementations.

For large communities, it's probably all wrong - personal changes in effect fork the entire wiki; but it may be a very effective personal tool - kind of mind mapping on the cheap, but much prettier than editors like treepad (and instantly portable, no software install required - keep it on your USB stick).

Basic usage instructions can be found here - written as a TiddlyWiki so you can try it out. I'm going to see if I can turn a rather unexciting Designer entity report into a thing of beauty...

Wednesday, December 19, 2007

Latest Bloor market research for Data Migration

You can find this survey either at Informatica http://www.informatica.com/info/bloorwpq407/ or at Bloor http://www.bloor-research.com/research/survey/876/data_migration_survey.html.

Generally sensible throughout but - unusually for Philip Howard's research output -it leaves some unanswered questions.

Even as a statistical forecasting effort, it does leave something to be desired. Rather a lot of tables have been included that don't contribute anythingto the sum of human knowledge - in particular a single CAGR (compound annual growth rate) of 10% appears to have been applied willy-nilly to every single region and sector (confirmed by retrofitting the figures -allowing for rounding to the nearest million). What a shame the research doesn't build on either Bloor or external figures for:-
  • likely frequency of application replacement (by industry: Banking hasa more rapid cycle than Trading Companies, and perhaps by context/application type)
  • industry/regional growth forecasts (eg from OECD)
  • No mention of government / public sector projects This is the biggest area for cockups, especially given the fragmented natureof PFI/PPI in the UK, and similar huckster schemes abroad.

However, the headline figures are worth absorbing, to give an order of magnitude for the market, and more importantly an idea of the extent of project failure.

  • 84% of data migration projects "fail" (not delivered on time and on budget)
  • Cost overruns average 30%; time overruns average 40%
  • Data migration is a $5 billion market this year (or maybe twice that if you throw in smaller projects that come in under the radar for this survery) growing to $12 billion by 2012 (sounds like a lot - but London is spending more than that on the 2012 Olympics...)

At a more detailed level, I'm not entirely comfortable with the attempt at separating "data migration" as a market in its own right from related markets (ETL, EAI, etc). I would prefer to see DM as a particular use case for ETL, EAI and DQ tools; many of the same tools will also have applications for data integration, application integration, business intelligence and other use cases - although mileage may vary.

Howard states that DM only aplies to "one off" migrations and is distinct from data integration; well, yes and no. DM may have to take place over months or years; during that period, "source" and "target" systems will have to coexist somehow. That may be achieved through "temporary" data integration/replication, or by partitioning the data and gradually handing over slices from source to target. Given the ever-changing nature of business, it is by no means inconceivable that the process never reaches its originally planned conclusion (the final decommissioning of the source system). Parts of the "source" may turn out to be worth retaining (ie the economic case for replacement is not viable).

Perhaps the biggest concern I have is that there is no mention of DM as a subset of business change. DM scoping is often imposed by (sometimes poorly considered) "business" considerations and decisions. There needs to be a feedback loop from DM processes into the overall project feasibility, scoping, planning, and costing. How many DM budgets are over budget simply because the budget was unrealistic to start with - because project costing was done using some rule of thumb [eg see John Morris's data migration blog] that happened to be inadequate in the circumstances?

Finally, I am amused (and not at all surprised) to find that hand coding is the market leader at 30%, beating ETL into second place with 28%. Given that ETL tools are often given away with applications, databases, and maybe even with leading brands of Cornflakes it is amazing that this situation hasn't much changed in the last 15-20 years since ETL tools first appeared. I always thought that DIY (roll-your-own for some american speakers) was our (Constellar's) biggest competitor back then - and here's yet more anecdotal evidence. Informatica and other tool developers - not to mention application/product architects - need to understand why that should be:

  • no tool is a magic bullet: even a market leading ETL tool may make some cases easier, but can make a few (important) cases much harder to manage (if all you have is a hammer, everything looks like a nail)
  • tools are expensive to own: they can be expensive to buy, but much more importantly skilled tool users are expensive to train or hire. The migration skill-base is fragmented (what works for PowerCenter is wrong for Oracle Data Integrator; Ascential does things differently from Ab Initio).
  • data migration projects are seen as "boring": so may not always attract the youngest, thrustingest, enterpriseyest architects. Like support, a critical area is treated like a leper in some organisations. No wonder the outcomes are sometimes suboptimal. Those cast into the outer darkness of a migration project eventually either transfer internally (and let their skills degrade), or hop off into a consultancy or frelance contract to rent out their newfound skills. Far too rarely does a competence centre emerge.
  • application architects (for in-house and packaged apps) often forget the requirement for data migration into their whizzy new systems. Nearly twenty years after it was first released, Oracle Apps still doesn't have a fully supported bulk migration solution for even the most basic data (eg Payables 11.5.10 finally added vendors, sites and contacts but still doesn't support their bank accounts...). What's unglamourous for the end user is equally unglamourous for the product developer, it seems.

Ho ho ho! Happy Christmas everyone

Wednesday, December 05, 2007

I cite, you borrow, he steals...

Anyone who appreciates the hard work that goes into a decent presentation or book should read Doug Burns' tale of woe. Isaac Newton is widely quoted acknowledging his debt to other scientists: "If I see further, it is because I am standing on the shoulder of giants". Don't let's tolerate those who seek to take the credit for another's efforts without due courtesy or recompense.

I find it particularly annoying as Doug (whose blog I read regularly, but whom I have never met in person) seems to be particularly (even pathologically?) careful to acknowledge where he gets his ideas from.

Take care to read the comments as well as the blog itself. This one will run and run...

Happy UKOUG by the way (from darkest Lancashire at the moment)!

Wednesday, November 07, 2007

Tombstone, AZ

Just back from a fortnight in Arizona, where we were visiting #1 daughter on exchange at UofA in Tucson. We took in Flagstaff, Williams, the Grand Canyon and Sedona as well as sights in and around Tucson itself.

A personal favourite was the Boothill Graveyard on the outskirts of Tombstone. It seems to have operated only for a very few years - and most of the occupants apparently died a sudden death - shot or hanged in the main (one is specifically shown as 'died of natural causes').

George Johnson was more unlucky than most. I wonder how they found out their mistake?
Posted by Picasa

Wednesday, October 17, 2007

Got an issue with that?

Here's a little summary of issue management systems I found in the Register for use with (agile) software products. It struck a chord with me - not least because I am right now working with a project team of around 30 trying to manage and organise a range of issues on an ever changing series of spreadsheets, using email as the repository.

I could add the example issue tracking system that comes with Oracle Apex - has anyone tried that?

Other wikis with issue management templates / recipes are available - including PmWiki PITS.

Monday, October 15, 2007

Latest Gartner Magic Quadrant for Data Integration

Informatica has helpfully syndicated the latest data integration magic quadrant. Out goes Ab Initio - apparently due to their perennial problems with secrecy (ie they won't tell anyone anything). Only IBM and Informatica make it to poll position (IBM slightly bolstered by its recent purchase of DataMirror). Hummingbird is the back marker, both for vision and ability to execute. Anyway, read it yourselves; no major surprises in there (and why would there be in such a mature market...).

Saturday, October 13, 2007

Is there WebLogic in Oracle's bid for BEA?

Finally, Larry has got out his wallet and made a bid for BEA( which of course they have rejected). This one has been waiting to happen for several years. Oracle's experience with app servers hasn't been entirely happy - but after a slow start they have managed to steamroller their way to a decent market share (often on the back of Oracle Apps); meanwhile most of the (often less well funded) pioneers have fallen by the wayside. Who remembers Persistence? Gemstone? Whatever happened to Sun's variously named products?

So who's left now? Oracle, IBM, BEA; MS IIS for the Netties, and a handful of low/no cost options led by JBoss (aka Red Hat).

Buying BEA presents an interesting marketing challenge in the J2EE app server area - though at least Oracle only has one app server at a time. Thing are much more exciting (ie confused) in the integration / add-ons market; Oracle Fusion is already in a state of upheaval, digesting bits and pieces from Siebel, Sunopsis, Hyperion and others. Throwing in BEA AquaLogic - which itself combines organic development and some BEA acquisitions, I seem to remember - will be "interesting" to say the least.

And don't forget other jewels at BEA. Tuxedo - a real piece of engineering - something Oracle could usefully combine with the best of its own server technology; and JRockit - a high performance JVM.

For those who hate acquisitions, remember: BEA bought in both JRockit (from a startup) and Tuxedo (originally developed at AT&T, bought in 1995)... and come to that, WebLogic itself (in 1998).

Thursday, October 11, 2007

Onwards and upwards

I've recently started a new contract which should prove very interesting. Starting off with a data migration study, I'm expecting there will be lots of performance planning and assurance to follow.

The setting is a rather large and unusual Oracle Apps implementation - client confidential of course. RAC, stretch cluster, massive batch processing, impossible response and resilience targets? Lots for me to (re)learn, and hopefully some useful previous experience to share.

The 250 mile weekly commute is a bit of a pain...

Bottoms up!

Tuesday, September 11, 2007

Cognos partners with Informatica

Cognos and Informatica have announced a strategic relationship; Cognos will sell and support Informatica Data Quality and Data Explorer products, and the two companies will "team to jointly provide customers with data integration capabilities" with a focus on performance management (rather than the wider data integration market?).

Interesting that Informatica's front page still highlights the May press release about being granted an injunction against Business Objects, while Cognos includes this release on its front page. A minor difference in marketing communications, or a subtle indication of who is the senior partner?

Analysis from The Street here, and from Intelligent Enterprise here.

Saturday, September 01, 2007

Finished at last (well, almost)


It's finally complete - we're just back from a lovely break in Sant'Angelo at the new house. Not entirely a holiday - there was lots of cleaning to do, and dozens of lights to buy and wire in (more to come later, along with a significant amount of Ikea furniture to be trucked in and assembled - in November we hope). And I spent most mornings working (those pesky deadlines again) proving at least that the EDGE service (no UMTS in the mountains) and a 20 euro for 500Mb web deal from TIM were enough to keep me going.

The garden remains - a couple of 80 year old olives should have gone in this week, and work will start on the rest in the autumn.
Nothing to do now but work as hard as possible to pay the bills!

Thursday, July 19, 2007

No more mirror, mirror jokes... IBM snaffles DataMirror

IBM bought my former employers DataMirror earlier this week. Obviously their main interests will be the HA product line (a direct value add for them), and then the Transformation Server (TS) real time data integration components. Others have written here and here, so I won't labour that except to say that as part of the package they have won Constellar Hub (formerly dear old Information Junction - there's a phrase that doesn't escape my lips too often).

DataMirror never really "got" Constellar. Sure, I think they sweated the customers enough to make a return on their CDN$10million investment (what a steal that was!), but they never made a serious effort (afaik) to bring together the best of the Hub and TS. A shame, because I think it could have been a winner compared to the still rather tired 1990s EAI products like Informatica, DataStage, Ab Initio etc. In fact, TS+Hub could have turned out not unlike the Oracle Data Integrator (but with added "publish / subscribe", and strong IBM and Oracle partnerships).

Now the boot is on the other foot. Will IBM "get" DataMirror and its various product lines? All we can be reasonably sure of is that the Hub is now officially past its sell by date...

Am I glad I left DataMirror in 2001, and so didn't get bought again 6 years later? On balance, I think so. But I know some who have stayed all along - will they stay with IBM too?

Monday, July 09, 2007

Iona FUSEs Celtix and LogicBlaze

Iona says it is expanding customers' open SOA choices by bringing together under the FUSE banner products that were formerly part of Celtix ESB, or were acquired with the LogicBlaze purchase this April.

CBR points out that Iona seems to be favouring LogicBlaze supported components (Apache projects ActiveMQ, ServiceMix and Camel) over Celtix components (apart from CXF, which has moved over from ObjectWeb to Apache). The LogicBlaze purchase must be working out well, methinks...

Friday, July 06, 2007

Talend - another open source ETL tool

Bloor's Philip Howard writes at IT-Director about Talend Open Studio - unusual among ETL solutions in that it is a code generator (of Java, SQL, Perl) rather than an "engine" type of product.

As well as the usual drag-n-drop transformation GUI, this apparently supports business process modelling - which gives Talend a feature that many "real" ETL/EAI tools don't have. There's also support for using a server grid to parallelise processing, and there's an "on demand" SaaS offering.

Version 2.0 is now available, as well as a 2.1 release candidate which is said to add features including:

  • further optimizations for performance increase
  • support of new databases (including bulk load)
  • transaction management (connection sharing, commit and rollback)
  • Slowly Changing Dimensions support
  • MOM (Message Oriented Middleware) support for real-time integration jobs
  • fuzzy logic data matching (using the Levenshtein and metaphone algorithms)
  • normalization, denormalization and flow merge
  • support of SSH remote connections
  • support of PGP file decryption (through GPG binary)
  • reinforced support of the XML standard: DTD validation, XSD, XSLT transformation, significant improvements in hierarchical XML file generation, support of the XMLRPC Web Services protocol…
  • improvements to the tMap component, to support input filters and new joins types such (the cartesian product, first match, last match…)

Sounds like it might be worth a closer look...

Zoning in to Sonos

We've finally bitten the bullet and shelled out for a 2 zone Sonos music system. I see we're not alone; Rob Levy, BEA's CTO, says it is his favourite gadget - and Joel Spolsky has also endorsed it.

We started around Christmas buying 2 Squeezeboxes - because I'm cheap, and because the Sonos system needs at least one box on wired Ethernet - which we didn't have handy in the right rooms. Although the Squeezebox looks good, it didn't really deliver as I had hoped.

The worst things about the Squeezebox:

  • it's not very good at coping with poor wireless reception - which probably accounts for most of the other problems

  • although you can synchronise two boxes (say one in the kitchen, and one in the living room) they tend to drift apart; not a good sound!

  • The slimserver software is incredibly slow at (re)indexing

  • the whole solution relies on slimserver software installed (by an importer) on a linux NAS fileserver - that makes getting updates much harder.

  • Internet radio was listed, but I could never get it to work

  • Documentation and help was rather flaky

  • The boxes look great but the UI is crummy - I never managed to explain it to my wife (or even to myself...)




So we cut our losses and invested in two Sonos ZP80s - they're quite small boxes, which feed into your existing amplifiers (you can use ZP100 if you want a built in amp). Rather than recable our house, we got a couple of powernet adapters. Music sits on a (simple) NAS disc - no intelligence required there.

We had some difficulty getting the powernet working - in the end after three hours trying every feasible combination of sockets in our (1st floor) office and (ground floor) living room, the very helpful installer swapped the original Netgear products for Microlink and found a pair of sockets that worked. Fantastic! after that everything was very straightforward. There's a (pricey) i-Pod like controller - or you can use a desktop interface. Once you've sussed the Zones and Music buttons, it's a piece of cake to navigate.

It's using the same music library as Windows Media Player - so I can load music once, and use it either on my headphones from the laptop, or on the Sonos system. Having the same music in two zones is easy - and very reliable. Adding new zones is also very simple (I just know we'll need a third zone in the office...). And internet radio works a treat - though with a slightly robotic quality, from the selection of sites supported (plenty of UK ones - helpful for us as we get almost nothing on DAB or FM).

The Squeezeboxes won't be entirely wasted; they'll go out to Italy this summer, where they should benefit - eventually - from being wired in (assuming the builders have remembered the CAT-5...).

Sunday, July 01, 2007

Graphic example

Thanks to Oracle AppsLab for referencing this stunning presentation by Hans Rosling. Not just for the superbly informative animated graphics, but also for the moving and thoughtful content (and the sting in the tail - watch all 19 minutes for the climax).

Friday, June 29, 2007

Could open source BI close out incumbents?

Bloor's Philip Howard writes in The Register that Jaspersoft 2.0 is now a serious threat to mainstream BI (and ETL) vendors; a full enterprise license for $35k will knock spots off most of the competition.

Perhaps that helps to explain why the fizz has gone out of Cognos's recent results.

And it doesn't help that according to a survey sponsored by Sybase and reported by CBR, only 13% of UK BI projects work as expected. So naturally "31% of firms surveyed, particularly small and medium sized businesses, [are] keeping an especially close eye on keeping costs down to a manageable level."

Thursday, June 28, 2007

To stakeholder (verb, transitive)

We've all done it at some point in our scribblings; we use horrible new words just because we know that you know what we really mean.

One particularly fashionable word at the moment is stakeholder. It's a useful short-hand for a person or group that has an investment, share, or interest in something, as a business or industry - and which can include employees, customers, the local community - and (if necessary) Uncle Tom Cobbley and all.

Here's a new usage I found in a consultant's report recently:

    Customers should ideally not be adversely affected ... and if downstream impacts are identified the customer should be stakeholdered ahead of time and those impacts managed accordingly.


From the context, I think the author means that someone should explain what's happening to the customer, and manage his (understandable) wrath. But rather than being good for the customer, being stakeholdered sounds more like getting out the garlic and the silver stake. Hold your head in shame, anonymous author!

Monday, June 25, 2007

IBM loses 80% of its Informix customer base

Computer Business Review reports that IBM admits to losing 4 out of every 5 customers for the Informix database it bought back in 2001. That's 80,000 gone out of 100,000.

Where did they go - to DB2 (in which case IBM is happy, even if the Informix rump isn't), or to Oracle, MS or the open source competition?

There's a contrary opinion here from ComputerWorld. Informix may have been "IBM's dirty little secret" but CW believes that the recent "Cheetah" release, improved marketing and changes to senior management in IBM will add up to a better future. Oh, and those 20,000 customers turn into 20,000 members of the Informix User Group - so there may be more out there that CBR didn't count.

Monday, June 18, 2007

IBM acquires Telelogic; a Rational move?

Interesting. IBM just announced the acquisition of Telelogic for around $745M. Telelogic is a Swedish company specialising in software configuration management; obviously their products will come under the Rational brand within IBM.

I spent much of the the last year as a Telelogic CM/Synergy user - having previously been more familiar with IBM Rational's own ClearCase product. They each have their strengths and weaknesses; but both suffer most from the naivety of their users and administrators. I've now worked on several sites who have proudly handed over some seriously enterprisey sum of money to the vendor, and then used one or other of these products when CVS or Subversion would have worked just as well (or maybe better). Why buy an all-singing, all-dancing tool, and then tie its hands behind its back?

Anyway, read what the Register says here and here.

Friday, June 15, 2007

The new house is coming on nicely

Here's a view of our new venture Casa dei due Mori - the house of the two mulberries - under construction. We're certainly looking forward to spending lots of time there this summer when it should be finished. It's been a long trek to get this far - nearly four years from the thought to the finished article. I think it will be well worth the wait.

Ciao!

Posted by Picasa

Tuesday, June 12, 2007

Buffered LOB_APPEND

Jens Ulrik asked to see the code for a buffered LOB_APPEND mentioned in an earlier post LOB_APPEND eats resources....

Here it is as part of a package. Apologies for the rush job at the time; inelegant, but it was worth it...



gi_chunk_length integer := 0;
gv_chunk varchar2(32767);

-- add a character value to the CLOB being built up; catch null values
-- NOTE - final flush is done elsewhere
-- and we assume max(length(av_value)) < ki_maxchunksize

PROCEDURE lob_append (at_doc_clob IN OUT NOCOPY CLOB, av_value IN VARCHAR2)
IS
li_value_length integer;
li_new_chunk_length integer;
ki_maxchunksize constant integer := 32000;
BEGIN
-- prt_common_pkg.print_xml_string(av_value);
IF av_value IS NOT NULL THEN
li_value_length := length(av_value);
li_new_chunk_length := gi_chunk_length + li_value_length;

if li_new_chunk_length >= ki_maxchunksize
then
-- pad out the chunk as much as possible
gv_chunk := gv_chunk || substr(av_value,1,ki_maxchunksize - gi_chunk_length);
-- write out
DBMS_LOB.append (at_doc_clob, gv_chunk);
-- initialise remainder into next chunk
gi_chunk_length := li_new_chunk_length - ki_maxchunksize;
gv_chunk := substr(av_value,li_value_length +1 - gi_chunk_length);
else
gv_chunk := gv_chunk || av_value;
gi_chunk_length := li_new_chunk_length;
end if;
ELSE
prt_common_pkg.LOG ('lob_append null ignored',
prt_common_pkg.klog_ret
);
END IF;
END;

The final flush of the last chunk of a LOB is called like this:


IF gi_chunk_length > 0 THEN
-- flush out the last chunk of the CLOB
dbms_lob.append(at_doc_clob, gv_chunk);
gi_chunk_length := 0;
gv_chunk := null;
END IF;

ETI offers data integration projects as a built-to-order service

(Disclosure: I worked for Constellar, a competitor, from 1995 to 2001, and worked with - or perhaps I should say against - ETI*Extract at an Oracle customer site as long ago as 1994).

Bloor's Philip Howard reports that ETI is offering a 'built to order' service for data migration / integration, based on ETI Solution (formerly ETI*Extract). Just enter your requirements into their integration portal, or send them a Word document, and they claim to be able to deliver a generated solution at an average price of $25,000 - and they claim to guarantee a delivery time (usually just 2-4 weeks). For more info, see ETI Built To Order Integration.

In my opinion, ETI's products suffered poor sales in the past from being pricey and (compared to products like Informatica) difficult to use. The template-based generation (by no means unique to ETI - Constellar took a similar approach, as does Oracle Data Integrator) was too hard for most clients to customise. However, the fact that it is a pure generator (there's no runtime engine) helped ETI establish a niche supporting all kinds of legacy/mainframe data sources and targets, and now makes it possible for them to deliver generated code without leaving their IP behind.

Offering BTO seems like a good way to exploit the tool's underlying capabilities, and the skills ETI's own consultants have accumulated, without having to make the tool itself more (dare I say) user friendly.

One question - most integration projects don't know what their requirements really are until they start development and testing. How will this service deal with iterative development? What about late 'clarifications' - "oh, did I mention that the NOTES field got used to distinguish customer types during the early 80s; in the 90s it was used to hold foreign currency information for certain services..."?

So now let's see how this works in real life!

Saturday, June 09, 2007

Open Source BI and ETL - picking up pace?

A couple of recent stories to ponder on: SnapLogic raises $2.5M (admittedly from its founder's very own VC boutique) to build its LAMP based data services, while according to CBR, Ingres is preparing a Jaspersoft based BI/ETL appliance.

Meanwhile Pentaho has released its Data Integration 2.5 (formerly known as Kettle) and is showing at ODTUG what it calls "the world's most popular open source BI suite".

Thursday, June 07, 2007

How to build a calculator - not!

The wonderful Worse Than Failure (formerly wtf.com) has been running The Olympiad of Misguided Geeks at Worse Than Failure Programming Contest. It's an inspired idea: just how bad can you make a deliverable, while at the same time meeting the requirement.

Alex set contestants the task of building a calculator. Responses have included:

  • Driving it all through optical character recognition - my personal favourite so far

  • collecting inputs and simply sending them to CALC.EXE to do the work

  • one built on an extensible framework that allows you to add new digits. You want to add apples and oranges? well now you can!


All too good to miss - you can find them here, so enjoy!

Sunday, May 20, 2007

Open source BI vendors get busy

A useful article from Computer Business Review. Has anyone got any experience with Pentaho or Jaspersoft data integration tools being used in anger that they'd like to share?

Saturday, May 19, 2007

Celona Evolve - "progressive integration" competitor for Oracle Data Integrator?

A piece of work I've been doing has involved taking a cursory look at Celona Evolve, though sadly I haven't been in a position actually to give it a go.

Coincidentally, this week Bloor's Philip Howard has written about it at IT-Director.

For the Oracle-ite, it's positioned in a similar market segment to Oracle Data Integrator (formerly Sunopsis); providing traditional ETL but also talking up continuous (or "progressive") integration (EAI / SOA / EII - take your pick of acronyms).

So far, successful references for Celona are hard to find - always a problem for startups. And it's a UK company; my personal experience says that tends to be a disadvantage too. You may get a few good early adopters over here, but going to the States can be a killer. At present the product seems to rely heavily on Celona's accompanying services; they won't be able to grow properly until either the product is easier to use, or there's a large independent base of Celona skills in the market (as there is for other tools like Informatica, OWB, etc).

They're obviously gearing up for a more public stance; they "launched" on 8th May (hence the analyst briefing). Good luck, we'll see how things progress

Tuesday, May 08, 2007

Is big software now like big pharma?

An interesting summary at Barrons of a VC panel at the Software 2007 conference. The worry is that big companies like MSFT, ORCL and IBM have so much CIO mindshare that there's none left for the little guys; the richer get richer and the poor (or small) are shut out.

Even the SaaS players trailing behind Salesforce are tiny; as Ted Schlein says “They are all gnats. They are all tiny. Not one would be a small division in one of these larger software companies.” (Actually, even Salesforce's (Q407) revenue is just $144M; our Larry could buy it from petty cash...).

Moving swiftly on

I've reached the end of a nine month engagement, where I've been helping to maintain a 12 year old system; originally developed I suspect on Oracle7, and still running on Forms/Reports 6i. It's been interesting mainly as an example of how to start to rebuild a team's understanding of a system a team (mainly contractors) has been churned, original documentation is hopelessly outdated, and the institution's understanding of its own systems has been undermined.

We started to replace a LAN full of (flakey, unindexed, heavyweight) documents with a wiki-based repository of (current, hyperlinked, searchable, lightweight) 'bits of knowledge' that can be combined in various useful ways. One or two other groups at the client had already started to use PmWiki, so we built on that. When I get a chance I'll post specifically on how I found PmWiki as a tool, and how it compares with other wikis (TWiki and MoinMoin) that I've used in the past.

Now I've started with a new client, where I am advancing my career by working on a data migration study for a 20+ year old system, based on a pre-relational database, whose name of course I can't mention here. The good news is that the 'new technology stack' is firmly Oracle based; but the most interesting part will be dealing with the political and operational issues involved in migrating what are literally the core systems for this business.

On the way, there should be some interesting comparisons of EAI/ETL/EII tools; I hope we see Oracle Data Integrator put through its paces on the way. Certainly I should be able to post more on 'integration' generally for the next few months.

Wednesday, May 02, 2007

Sonic's Dave Chappell joins Oracle

Oracle must be well pleased to pick up Dave Chappell, formerly a VP and chief evangelist at Progress Software's Sonic division (he's still on the management team there, according to this link). When that gets taken down, you can find his O'Reilly author profile here.

Dave will be development VP for SOA/ESB. I wonder which products that includes - Oracle Data Integrator perhaps? Good luck Dave, enjoy the ride at Oracle - I hope we meet again at another JavaOne some day.

PS, don't confuse him with any other Dave Chappell (like this one who runs an eponymous IT consultancy and speaks (for example) on BPEL, or this one who (I think it's safe to say) has no IT connection at all.

Java Caching for Oracle Apps 11i

Steve Chan's article is about the use of caching interesting; now how long will it take for Oracle to put him together with the guys from Tangosol that they acquired in March?

Who knows, they could even finish off JSR 107: JCACHE - Java Temporary Caching API which Oracle submitted in 2001, and for which Tangosol's Cameron Purdy has been tech lead since sometime around 2002/3. Heck, most of the companies listed in the expert group have gone to the retirement home long since (names like Gemstone, Blusestone, iPlanet and SeeBeyond). Not bad for a project that was expected to take 12 weeks...

Toodle pip!

Thursday, April 26, 2007

Joins over histograms

Here's an awesome paper from Alberto Dell'Era and Wolfgang Breitling looking at "the formula used by the Cost Based Optimizer to estimate the cardinality of a [single-column] equijoin, when both the columns referenced in the join predicate have histograms collected."

Awesome because of the thought given to the presentation, and the care taken to test the results. Alberto has followed the lead taken by Jonathan Lewis's CBO book to carefully deconstruct and reverse engineer the algorithms used - with one or two surprising results.

Well worth a read...

Friday, April 13, 2007

Iona picks up LogicBlaze and its open source SOA platform

The team at LogicBlaze has been putting together open source systems including Apache ActiveMQ (a JMS implementation), Apache ServiceMix (an ESB) and LogicBlaze FUSE (their own SOA platform).

Iona and LogicBlaze have been working together for a while, so there shouldn't be too many integration problems.

I wish the team, especially James Strachan and Robert Davies, former founders of (and my colleagues at) SpiritSoft, good luck in their new home. It's interesting that Iona - which contributed several to the Spiritsoft team around the turn of the millenium - should be their next resting place.

Friday, April 06, 2007

WebMethods sells itself to Software AG

I said a couple of months ago that webMethods needed to shake up a bit. Well, now they've sold out to Software AG - originally best known in the 1980s for its Adabas database and Natural reporting, more recently a player in the XML / SOA world with Tamino and EntireX.

WebMethods was founded in 1996 and its IPO in 200 marked a high point of the EAI boom; an offer price of $35.00 turned into an opening of $195 and a first day close of $212. The price reached over $300 quite quickly, but since then has trended fairly steadily down to today's sub $10. Not a good investment overall!

Software AG wants this to be "a major step in [their] plans to more than double revenue to EUR 1 billion (USD $1.3 billion)." WebMethods' $200 odd million of annual revenue is a big - if barely profitable - step in that direction.

An article at Barrons notes that the share price is now trading above the offer - a sign that some in the market are expecting a higher bid (the analyst doesn't think they will be in luck).

The deal has pushed up Tibco's price too - on hopes that they will be the next target, perhaps.

Saturday, March 24, 2007

Cache in the bank - Oracle buys Tangosol

Oracle has picked up Tangosol - and its caching product Coherence - for the traditional undisclosed sum.

Well done Cameron Purdy. For a couple of years I worked at SpiritSoft, and we competed with Tangosol. Our engineers always said we had the better architecture - but Tangosol whipped our ass good anyhow. Cameron's team delivered a robust product and made it easy for people to buy - and the fact that they now claim 1500 implementations is testimony to lots of hard work all round.

We competed around the JCache JSR 107, which must be the slowest JSR to get to a review draft (and ironically, it was originally sponsored by Oracle). This 2002 TSS thread was already talking about excessive delays; since then very little seems to have happened - mind you the JCP (Java Community Process) site is having an off day, so perhaps it came out while I wasn't looking.

Soon after that TSS exchange Cameron joined the JSR 107 expert group, and IIRC he became the spec lead. But still no delivery. Perhaps now Oracle will push the standard some more? Anyway, all that competition is water long under the bridge, so let's recognise a winner, and as Cameron would say:

Peace

Thursday, March 22, 2007

John Backus - father of FORTRAN - RIP

As well as being the father of the first high level language suitable for numerical work - and of Algol 60 which can be taken as the root of most of today's widely used languages like C, C++ and Java - John Backus lives on as coauthor of Backus Naur Form (BNF) which we still use in the Oracle docs - for example:

relational_table ::= CREATE [GLOBAL TEMPORARY] TABLE [schema.]table
[(relational_properties)]
[ON COMMIT {DELETE | PRESERVE} ROWS]
physical_properties
table_properties;

Monday, March 12, 2007

Vitria is now private; Iona buys C24; Informatica Integration on demand

As I noted in October, Vitria is being taken private by its founders. Shareholders have now approved, and the transaction closed on 7th March.

This marks the end of a rollercoaster ride on NASDAQ (and by rollercoaster I mean there was a sharp climb at the beginning, but after a series of humps, bumps and loops you end up right back on the ground). Vitria's results over the last three years show everything gradually declining - revenue, assets, license sales. One positive - losses have also been reduced. Can Chang and Skeen turn the ship around, or is this just another step towards the sunset retirement home for distressed software companies?

Meanwhile Iona has picked up (London) City specialist integration boutique C24 for an undisclosed price. Given C24's small size - 12 employees - there should be no major digestion problem for Iona as long as C24's customers are kept sweet. Good luck to the C24 guys, a couple of whom I met while I was at SpiritSoft.

Finally, Informatica has launched the "first and only" on-demand data integration service. The "On Demand Data Replicator" - yours free for 30 days, and $1500/month from then on - is initially aimed at Salesforce.com customers; SaaS vendors like RightNow and NetSuite are next in line. The integration is (I guess) intended to be from your internal apps to your hosted apps, and vice versa.

Thursday, March 01, 2007

Oracle captures Hyperion - a well planned thrust at SAP?

Lots of buzz today about Orace's rumoured - then confirmed - acquisition of Hyperion (formerly known as, and still largely known for, the eponymous Essbase multi-dimensional database). Oracle bloggers like Mark Rittman have concentrated on the BI side of things, but it is worth remembering that Hyperion has also assembled a set of financial applications (mainly planning and modelling, as you’d expect) - not to mention a BPM product line. That could be very interesting as an add-on to Oracle Apps (not to mention PeopleSoft and Siebel).

Looking at their latest Q2 results, there’s no breakdown of revenue between product lines - but it’s interesting that the headcount is 1745 in Americas, 632 in EMEA and only 212 in APAC. Oracle’s wider/deeper international network could give a big boost to sales in Asia (as well as making admin savings at home). They'll keep the salesforce but dump the top-heavy administration.

The financial market seems reasonably positive about the news (see for example Barron's Eric Savitz. As well as providing a sell-up for Oracle Apps, Peoplesoft and Siebel, this can also get Oracle's foot further into the door at SAP sites.

    "Hyperion is the latest move in our strategy to expand Oracle’s offerings to SAP customers,” said Oracle President Charles Phillips. "... Now Oracle's Hyperion software will be the lens through which SAP's most important customers view and analyze their underlying SAP ERP data."

There is also a feeling that Oracle got a good price, catching Hyperion with its share price down. The impact on other BI players like Business Objects and Cognos is mixed: M&A activity might be expected to push up their price - but their share prices already factored in a bit of a punt on Larry Ellison coming round to tea; now he's spent his money on Hyperion, the others may well fall back.

Good luck to the Oracle BPM, BI and Apps product managers trying to make sense of it all!

PS: Another good take here from Curt Monash at DBMS2

Wednesday, February 28, 2007

WebMethods/Infravio X-Registry - free 45 day trial download available

Back in December, I asked would you pay $99000 for a starter pack.

Well, it looks like everyone else must have agreed, and not paid up. So webMethods has now announced a free trial download which "can be deployed for 45 days of testing and evaluation within a non-production environment".

After their recent results (see yesterday's post) I guess someone in marketing has decided it's about time to shake things up and create a few more leads for the salesforce to work on.

Tuesday, February 27, 2007

EAI results - a mixed bag

Back to looking at financials, a note on Motley Fool that touts Tibco as a "king of cash" reminded me to poke around some of the EAI vendors' financial results.

Tibco's Q4 (ending November 06) was pretty spectacular, with revenue up 20% to $160 million - license revenue growth being a very healthy 32% to $88 million. But earnings over the entire year were barely up; Tibco seems to have a pattern of a huge Q4 after flat Q1/2/3.

WebMethods earnings show license income down more than 10% in Q3 to $19.7 million (total revenues $53.1 million). Motley Fool's article New Product, Same Problem suggests that digesting recent acquisitions (eg Infravio) and restructuring the salesforce are affecting sales.

Vitria's results show a startling spike in license revenue up to $6.7 million up from $1.8 million in the same quarter to Dec 2005 - around half of that accounted for by two customers. Encouragingly, Vitria is in the black.

Finally, BEA takes a bit of a beating - shares down 10% even though revenues were up 15% on same quarter last year. Why? because forecasts are down for next quarter (the analysts wanted $385 million, but management expects only $350-364 million).

A Morningstar analyst rounds it all off in the same news item by suggesting that BEA will get tough competition in the SOA space from Oracle, IBM and Tibco.

Monday, February 26, 2007

Unique IDs for multi-master replication - Sequence or SYS_GUID?

Oracle-L is proving to be a good source of inspiration at the moment. Oracle ACE Syed Jaffar Hussain asked the question Is it possible to share a database sequence between multiple databases?

Using remote sequences
A couple of the replies took the question a bit too literally, and said yes, you can define a sequence on one database, and use it from another:

Select SequenceOwner.RemoteSeqName.NextVal@DBLinkName From dual


This works - and it can be made 'location transparent' by creating a local synonym for the remote sequence. But it is asymmetric (one database has to own the sequence) and it introduces a point of failure. If database 1 owns the sequence, database 2 can only insert rows if database 1 is actually available. If you could guarantee availability, why would you bother with replication?

Using a sequence local to each database
Several replies (including mine) suggested using carefully defined sequences which will deliver discrete values on two or more master databases. There are two basic patterns:

Partitioned: Suggested by several posters, the number space is divided up in blocks:

on db 1: create sequence myseq start with 1000000 max 1999999
on db 2: create sequence myseq start with 2000000 max 2999999


A variant on this is to use positive numbers for database 1, and negative numbers for database 2.

Interleaved: The more popular option is to use the old trick of assigning odd numbers to database 1, and even numbers to database 2:

on db 1: create sequence myseq start with 1 increment by 2
on db 2: create sequence myseq start with 2 increment by 2

This mechanism is much easier to manage (essentially, it doesn't need any further management). It is also easy to extend to 3, 4, 27 or 127 masters - just set the "start with" to the database number, and "increment by" to the maximum anticipated number of databases required.

A third option was also proposed by Mark D Powell: use SYS_GUID(). That has some disadvantages:
  • SYS_GUID() is bigger (16 bytes on 9iR2) than a NUMBER (eg a 13 digit integer needs around 8 bytes). Obviously the extra space follows through to indexes, foreign keys etc.

  • Another is simply that it is a RAW, which has some possibly undesirable implications; for example several common tools (including SQL*Developer 1.0) can't directly display RAW values; you have to explicitly select RAWTOHEX(id_column).



On the other hand, SYS_GUID can be defined as the default value for a column - unlike NEXTVAL which is normally set in a trigger or directly as part of an insert into/select from statement. Worse, it is very common to see a separate SELECT seq.NEXTVAL FROM DUAL for every ID generated. I've never investigated the relative performance of SYS_GUID() against getting a sequence number - anyone else like to share that? That may well be the most important consideration of all.

So I think I'll be coming back to this subject in future.

Update - later the same day:

I've quickly timed a million iterations of select sys_guid() from dual, and a million iterations of select sequence.nextval from dual (Oracle XE 10.2.0.1, HP dv1665 Centrino Duo running Windows XP)

select sys_guid() into variable from dual: 89 seconds
select seq.nextval into variable from dual: 40 seconds

For good measure, I've added a variant - one million calls of sys_guid() without a select:

variable := sys_guid() : 95 seconds

One of the potential advantages of sys_guid() I had anticipated is that it could be called directly from PL/SQL - but it looks like the implementation is less efficient than it might be; tracing shows that the PL/SQL function recursively selects SYS_GUID() from dual.

So I think I'll stick to traditional and more convenient sequence.nextval for now.

Chin chin!

Sunday, February 25, 2007

Avoiding application suicide by session pool

A recent post on Oracle-L mentioned:
    We had an issue affecting one of our production DBs. The middle tier application for some reason, went crazy spawning processes chewing up the DBs process parameter. DB started throwing errors indicating max process exceeded.


He went on to ask how to wake up PMON to clean up these processes - he'd had to bounce the database to get things cleaned up.

My response was to ask about the root cause, rather than the symptom. The poster may have been solving the wrong problem (or rather, after putting out the fire, he needed to find the cause and stop it happening again).

"Mid tier went crazy spawning processes" is often a symptom of session pool
madness. In such an application, X number of users share a smaller Y number of Oracle sessions. Everything tootles along happily; Users (midtier threads if you like) loop around:

  • get a session from the pool

  • issue one or two SQL

  • commit/rollback

  • give the session back


As long as the users spend less time in Oracle than they do in the rest of the
application (and waiting for user input etc), no problem.

Then something goes wrong; maybe a session sits on a lock that everyone needs; maybe a sequence cache isn't big enough (or is ordeed) and/or you forgot that We Don't Use RAC; maybe you had an SGA problem like ORA-4031.

What happens next:

  • all the Oracle sessions in the pool are busy

  • next midtier thread asks for an Oracle session

  • midtier pool manager says "no problem", launches a new Oracle session and adds it to the pool

  • that session becomes busy

  • and the next thread, and the next thread, and the next thread...


Soon instead of sharing say 100 Oracle sessions across 1000 processing threads,
your mid tier has responded to the blockage by adding 900 new sessions to the
load. That's probably made the problem worse, not better - kind of like slamming your foot on the accelerator when you see brakelights ahead in the fog.

I had exactly this problem performance last year, testing a J2EE app, using OC4J. We hit a 4031 problem (no bind variables in one part of the system) and then fairly immediately the application server did its lemming impersonation as described above.

Things to consider:
1) reduce the upper limit on the session pool size (definitely to below your Oracle processes level!)
2) if possible, slow down the rate of session starts (eg set a delay in the mid-tier session manager)
3) find out what caused the problem in the first case.

The good news is that if you dampen down this suicidal behaviour, you probably have a better chance of diagnosing the root cause next time.

Wednesday, February 21, 2007

Data warehouse appliance pros and cons

Andy Hayler writes here about the marketing hype surrounding data warehouse appliances like Netezza, and warns that buyers should not just consider the 20% of a project budget that is typically spent on hardware and software. The other 80% goes on - well, people like us.

However the economic case for the Netezza model can still be made - rather easily. I’m aware of a POC where the capital costs of a 50TB Netezza solution are considerably less than the business is being charged back each year just for the cpus and discs supporting the equivalent Oracle DW. The clincher is that Netezza doesn’t require the same level of DBA expertise - it just hasn’t got any tuning knobs for us to fiddle with. Oh, and critical queries run up to 150 times faster ...

If these appliances result in a hefty capital saving and a substantial reduction in DBA/performance tuning overheads, then they will certainly continue to gain market share.

Wednesday, February 14, 2007

Google Oracle - Good start, now what about Metalink?

The news is out - here from Eye on Oracle - that Oracle has finally opened the door to Google indexing of Oracle's documentation.

That's marvellous, but it's not exactly the fall of the Berlin wall. Many Oracle docs have been online and available for Google search for years - certainly for the database. Some of these have been (accidentally?) made available on customer websites (eg http://www.lc.leidenuniv.nl/awcourse/oracle/nav/docindex.htm - it's often universities) and others by Oracle themselves (eg http://download-east.oracle.com/docs/cd/B14117_01/nav/portal_3.htm). Just Google for key phrases like Browse the list of books, including PDF for printing for 9i or ADM link goes to the Administrator's Guide which gets you the 10g book list - and you'll see what I mean. I've included the 'repeat the search with omitted the results included' option.

What would be really useful is to open up Metalink itself (only customer SRs excepted). Too many Oracle developers need access, but can't get it because of short sighted officiousness. We developers (especially freelancers like me) often get the short straw - we're expected to be have all Oracle knowledge at our fingertips and yet we are effectively prevented from using one of the best available resources for it. Our employers just won't give us access to the support contract information necessary to get connected. BTW, I stress that the main fault is not Oracle's; it lies more with jobsworths (Brit expression I think - those people who say "it's more than my job's worth" as an excuse for anything).

Oracle Corp can and has made valid arguments for secrecy - mainly around its intellectual property rights and its competitors. But anyone who wants to indulge in industrial espionage can probably afford to buy a support license and get into Metalink anyway. The net effect is that the security defeats exactly the wrong group.

So go on Redwood - open up a bit more; you know you want to!

Monday, February 12, 2007

Schema models, FK constraints and Oracle Apps - a matter of entropy?

There's been an interesting series of posts on Oracle-L recently, which started with a request for a schema model of Oracle Apps. Jared Still posted this response, which included the observation:

Oracle Apps - Not quite as sure about it, but I believe its origins
predate the use of referential integrity in the database.



I worked on Oracle (UK) Accounting v3 during mid-late 80s, some ideas from which (code combinations for example) were 'borrowed' for what became Oracle Financials. AIRC that was around 1987-8. HR development started a little later, and being UK based was a lot more CASE (Oracle Designer) inclined (they shared the same building). Manufacturing originated in the US consulting organisation, and again they were somewhat more methodological than the original Apps team in Redwood. In both cases I think initial development still pre-dated the implementation of effective foreign keys in Oracle 7. For Financials, even the option of FKs in the dictionary was not yet on the menu.

The (defensible) strategy to 'disintegrate' the applications (eg having separate GL, AP, AR modules in Financials) made it easier to get early releases out of the door - but at the cost of hiding relationships. And of course the whole Application Foundation ethos of configurable code combinations and flexfields means that many relationships are impossible to implement as Oracle serverside constraints out of the box.

Since then, I guess the normal product development priorities have reigned: functionality/saleability first, customer bug fixes second, and engineering/non-functional improvements last. Performance fixing priority goes up and down the scale according to the level of pain being felt by critical customers (Cary Millsap will remember performance testing of release 9, for example).

Just try to explain to a VP of Applications that you want to spend (managers prefer 'invest') tens of man-years documenting and 'improving' internals. It's like painting the Golden Gate bridge - once you start, you never stop. That budget has to come from somewhere - and the other priorities always seem more attractive. So there is a tendency to maximise entropy (btw that's one of the main reasons why startups can beat gorillas).

All the large ERPs seem to suffer the same problems - it's not just Oracle. But as one poster said - the more gotchas there are in the Apps, the more work for us... in the short term at least.

Happy obfuscating!

Tuesday, February 06, 2007

The more the meta - 'Intentional Software' meets reality?

This quite long Technology Review article catches up with Charles Simonyi - Microsoft's former Chief Architect and the alleged inventor of Hungarian variable naming convention - who will be rocketing up to the International Space Station in April.

In his day job, he has been developing a 'new' concept now known as 'intentional software'; for every problem domain, developers will create generic tools which users can employ to "guide the software's future evolution".

Code generators, CASE tools and IDEs could all be seen as primitive examples of what he's on about. They also point to the possible fly in the ointment. As you approach the problem boundaries, any preconceived tool inevitably becomes increasingly inefficient; and the abstraction often makes it difficult or impossible for the user/developer to interfere productively. This is the law of leaky abstractions.

That's not to say that domain-specific generators are a bad thing per se; the template driven generator underlying Constellar Hub was a key component in its success, allowing us to quickly improve and extend the range of objects it could generate. When the tool runs out of steam, code developers can simply jump out of the tool and modify the generated code (sadly, that seems to be the permanent situation for most J2EE - more code and less declarative than any 1980s 4GL) or use a different abstraction.

But as far as Simonyi's work goes, you can count me in with the sceptics. Changing the representation of a problem (from textual code to a graphical parse tree, for example) does not in itself make the problem easier to solve. I saw my first 'visual' coding system in about 1981 - on a VT100 at that - and it was both brilliant and (without a mouse) completely **** useless at the same time. Still, it will be fun watching him throw all his money at it...

Ciao!

Sunday, January 14, 2007

Kalido in a box?

Bloor's Philip Howard writes in IT-Director.com that Kalido gets a competitor.

The problem Kalido addresses is dealing with change in a datawarehouse; for nearly 10 years Kalido has been the only company trying to The new kid on the block aiming to manage changing/evolving business models without having to completely rebuild your datawarehouse is BIReady, a small Dutch company (hmm, Kalido came out of Royal Dutch Shell ... do those Netherlanders know something we don't?) which claims to deliver 'fully model driven' data warehouse with support for history of change.

This is an area which has traditionally been dealt with by large numbers of staff and consultants. Kalido has managed to make some sales in large multi-national companies who value the benefits of managing not just evolving models, but multiple concurrently active models. BIReady looks too small to bite off these very large organisations (like Shell, Unilever etc) - but they may help to commoditise this part of the DW market, and perhaps make people think about model-driven warehousing further down the food chain.

One to watch.

Saturday, January 06, 2007

Listen to the music

I love listening to music. At work, I use my own laptop like a (rather large) iPod - its a great way to zone out of the open plan hubbub. Currently I'm just about to hit 6000 tracks, which I mostly have on permanent shuffle. We're looking into sound systems for our house in Italy and although I think the Sonos system (described here by Joel Spolsky) looks fab, I also tripped over the Slim Devices Squeezebox and wanted to have a go with that first, as it's a lot cheaper.

Essentially, we've moved our music collection onto a network attached file server (from Qnap, which conveniently comes with the Slimserver software pre-installed on Linux). You can continue to rip CDs using Windows Media Player (just pointing it at the new network directory).

Slimserver indexes the collection (very slowly - what's that all about?) and then you're ready to go.

Setting up the Squeezebox was easy - apart from revealing yet again what a poor wireless network we have. Eventually I convinced myself to replace the ADSL/wireless router, replacing a Belkin which has never been very reliable with a Netgear Rangemax, which was a doddle to set up and so far seems solid as a rock. The Squeezebox is in the living room, plugged into the home cinema's spare input (DVD, Sky+ and now this).

The Squeezebox looks fabulous - sleek and white and not too small (or large). The music reproduction is fine as far as my aging ears nnd cheap home cinema speakers can tell; and I easily can re-rip my CDs at a higher bit rate. However needing a separate remote is a bit of a pain (that now makes FIVE remotes to lose). That's where Sonos wins - they have a beautiful (massively expensive) wireless remote that can control all your Sonos Zone Players using an iPod like panel. The Squeezebox has a complete phonepad of function keys, and the interaction with the display is not entirely intuitive.

Anyway, it all works fine (as long as we balance the Squeezebox on a box of chocolates - it must be in the worst place in the house for wireless, as far as possible from the base station and practically behind the TV). So now I have to work out how to build playlists, and then decide if I want to fork out double or treble for Sonos at the house in Italy.

Choices, choices!

Wednesday, December 13, 2006

Would you pay $99,000 for a starter pack?

WebMethods has launched a starter pack for Infravio X-Registry, which combines a UDDI registry, a JAXR repository, and what they coyly describe as integrated governance [which] ensures that these policies can be effectively applied across design-time, run-time and change-time environments.

For the bargain basement price of just $99,000 you get to minimize the upfront cost of initial adoption. Or to put it another way, you get 2 cpus, 5 named users and a UDDI registry that will allow you to define just 25 web services.

Why on earth would anyone want to pay that much to adopt a vendor-ized version of open standard technologies? Now if they had said this was a free community edition, they might be in with a chance of luring unsuspecting punters in and hitting them with the upgrade later. But this way round? Seems daft to me...

Tuesday, December 12, 2006

Informatica buys Itemfield, gets a sprinkle of Gartner's magic dust

Informatica has agreed to buy Itemfield - an Isreali outfit specialising in integration of unstructured and semi-structured data. That adds a useful extra string to their bow.

Coincidentally Gartner's latest Magic Quadrant for Data Integration Tools has recently materialised. Informatica gets a positive write up - its stands alone with IBM in the leaders quadrant, head and shoulders above the pack (including Oracle).

Sadly my former Constellar colleagues at DataMirror don't even get on the map - DM just gets a rather dismissive one line mention for Transformation Server in the also-rans section. Is DataMirror under-marketing its product range - including Constellar Hub, or is Gartner just not too impressed with it?

Friday, December 08, 2006

Problems with Statspack

Everyone's talking about it - statspack isn't perfect. Driven from Jonathan Lewis to Doug Burns to Daniel Fink's OptimalDBA I thought it was worth pointing out that as a snapshot technique, your knowledge of the value of the increment is uncertain.

Suppose snapshot N of metric has value s(N)
lets call the increment N to N+1 x(N)

All we can say for sure is:
A) if f(N+1) <>= f(N+1)
B) if f(N+1) >= f(N) then x(N) >= f(N+1) - f(N)

In normal use (when the shared pool is stable), we only see the case B, and we replace the >= with =. But if your shared pool is being trashed (see earlier posts) then remember uncertainty, and consider building case (A) into your reports.

Sunday, October 15, 2006

Vitria goes private

Once one of the great hopes of the EAI boom of the late 90s, Vitria has struggled over the last five years. Now I just noticed that they recently announced that they are going private.

The purchasers are Vitria founders Dale Skeen (current CEO) and his wife Jomei Chang (founding president and CEO). When Reuters bought Teknekron (which soon morphed into Tibco) they used their $10m to start up Vitria. As this 2001 story explains, Vitria's 1999 float was one of the more ebullient - priced at $16, hitting $273 just three months later.

Vitria was perhaps the first EAI company really to 'get' the idea of process level (rather than data) integration. Perhaps their biggest problem has been that their proprietary approach had hardly started to gain acceptance by the time industry standards - web services, service oriented architecture, BPEL and the rest - pulled the rug from under their feet.

Adjusted for stock splits, a peak price of $400 compares to a sale price of $2.75 per share. It will be very interesting to see what Chang and Skeen manage to make of the rump of their creation.

Monday, September 25, 2006

Loading iostat output into Oracle

Alex Gorbachev posted an item on Pythian's blog last week about Basic I/O monitoring on Linux.

That's reminded me to post a couple of scripts here that can be used to load cpu and disk stats from iostat into Oracle tables.

I assume your iostat command line looks something like this:


iostat -x -t [interval] > iostat.log


You can then process iostat.log using this script to make it more SQL*Loader friendly:


cat iostat.log | processiostat > iostat.dat


And here's the processiostat script:

#!/bin/sh
#
# read and process an iostat.out file
# assumption is that it has been collected using -x -t
#
# Format looks like this (excluding leading comment sign
#=======================
#Linux 2.4.21-27.ELsmp (myserver.preferisco.com) 02/05/06
#
#Time: 13:01:58
#avg-cpu: %user %nice %sys %iowait %idle
# 0.96 0.06 0.49 1.93 96.55
#
#Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
#sda 80.85 0.00 22.62 29.90 388.10 522.72 194.05 261.36 17.34 0.00 0.08 0.02 0.11
#sde 44.04 0.00 12.19 1.78 203.58 14.28 101.79 7.14 15.59 0.01 0.61 0.33 0.46
#sdi 44.41 0.00 11.68 1.87 195.50 15.71 97.75 7.86 15.59 0.01 0.79 0.45 0.61
#
#Time: ...
#=======================
#
# Read the file from stdin
read os version longhost date
# extract hostname - works for rac2-4 at least
# discard empty line
read discard
while read label time
do
# discard cpu headings
read discard
# read cpu figures
read usercpu nicecpu syscpu iowait idle
echo CPU ${date}-$time $longhost $usercpu $nicecpu $syscpu $iowait $idle
# discard blank line
read discard
# discard headings
read discard
# now read in device stats
device="START"
until [ "$device" = "" ]
do
read device rrqms wrqms rps wps rsps wsps rkbs wkbs rqsz qusz await svctm util
if [ "$device" != "" ]
then
echo DISK ${date}-$time $longhost $device $rrqms $wrqms $rps $wps $rsps $wsps $rkbs $wkbs $rqsz $qusz $await $svctm $util
fi
done
done


That .dat output can then be loaded using this control file:

-- general ctl format for iostat tables
load data
infile *
into table IOSTAT_CPU REPLACE
WHEN (1:3)="CPU"
fields terminated by whitespace
( TSTAMP position (5) DATE "mm/dd/yy-hh24:mi:ss"
, HOST
, USERCPU
, NICECPU
, SYSCPU
, IOWAIT
, IDLE
)
into table IOSTAT_DISK REPLACE
WHEN (1:4)="DISK"
fields terminated by whitespace
( TSTAMP position(6) DATE "mm/dd/yy-hh24:mi:ss"
, HOST
, DEVICE
, RRQMS
, WRQMS
, RPS
, WPS
, RSPS
, WSPS
, RKBS
, WKBS
, RQSZ
, QUSZ
, AWAIT
, SVCTM
, UTIL
)


Then you can do comparisons, trend analysis, even draw pretty graphs using your favourite GUI (SQL*Plus in my case...)

Enjoy...

Tuesday, August 08, 2006

Back in the saddle

After five weeks off (two of them in Italy) I'm finally starting a new contract tomorrow.

Good news
- it's not on an industrial estate behind the sugar factory in Bury St Edmunds, with a visit once/day from a sandwich lady
- the office is on a modern business park with four Starbucks, a restaurant and (not coincidentally) a mobile phone shop

Less good news
- it's 100 miles away, so I'll be spending a deal of time in the car, and/or in B&Bs for the next few months.

At least it should be interesting work with an enthusiastic team (from those I've met so far) - and I'll be well placed for training into London to meet friends and colleagues. And in case it all gets too much, I've got another two weeks in Italy starting a week today with a couple of days in Venice, then back to the house in Le Marche for a rest. It's a hard life...

Thursday, July 06, 2006

Using global context variables

I recently stumbled on another 'new' feature - though in fact it was introduced way back in Oracle 9i. A questioner on the OTN database forum wanted to know how to invalidate cached package state derived from a table that might be updated from another session.

My first suggestion was to implement a time-to-live cache - treat the data as stale if it has been cached for longer than X seconds (using DBMS_UTILITY.get_time to record the load time). All sessions manage their own private caches; a cache can be inconsistent for up to X seconds (if the underlying data is updated immediately after the cache is loaded).

Vetaran OTN poster Jens Petersen came up with what could be a much neater solution, which is to use a global context. You can define your own namespace for use with the standard SYS_CONTEXT function, and arrange to get/set attribute name/value pairs - on a per-session or instance-global basis. Like a DIRECTORY, a CONTEXT isn't owned by a specific schema.

Creating the context

To be able to create a context the user must be granted CREATE ANY CONTEXT privilege.

create or replace context my_context
using testuser.my_context_pkg
accessed globally -- including this makes attribute values visible to all sessions
/

A package is linked to the context; this is the only interface that can be used to set attribute values in the context. This coupling is a little inelegant, but it does mean that only users with EXECUTE permission on the package will be able to set values.

create or replace package my_context_pkg
as
procedure set_context
(av_attribute in VARCHAR2
,av_value in VARCHAR2
);
end my_context_pkg;
/

create or replace package body my_context_pkg
as
procedure set_context
(av_attribute in VARCHAR2
,av_value in VARCHAR2
)
is
begin
DBMS_SESSION.SET_CONTEXT
( namespace=>'MY_CONTEXT'
, attribute=>av_attribute
, value=>av_value
);
end set_context;

end my_context_pkg;
/

Any API you like to define is supported; this example is the simplest, allowing a completely unconstrained setting of attributes pairs. Here's how you define and use an attribute:

my_context_pkg.set_context('Attribute Name', 'Attribute Value');

Now, what about that original problem - how to notify active sessions that data in their package state might be out of date?

Well, there are two approaches: replace the cache directly with (one or more) CONTEXTs, or construct a dirty cache notification scheme:

Replace cache with CONTEXT


Simply maintain the cache as attribute/values in a CONTEXT. This could be appropriate if the cache was relatively simple (eg a set of lookup codes and descriptions). The PL/SQL caching can be completely discarded; instead use SYS_CONTEXT to get the descriptions, and set up triggers on the lookup table that use MY_CONTEXT_PKG to update the context. The only minor snag is that the context values need to be set (at least once) on instance startup; this could be a scheduled job, or be incorporated into a logon trigger.

Dirty cache notification


Define a 'cache update sequence' context attribute; let's suppose it uses MY_CONTEXT with an attribute name of 'cacheseq'.

When the package initialises its cache, it should read and retain the latest value for 'cacheseq' (treating null as 0). Later, before using values from the cache, it should re-read the latest value for 'cacheseq'. If the numeric value is greater than the retained value, the cache must be dirty. Applications that update the underlying tables (and so invalidate any caches) must increment 'cacheseq' - normally by using a statement level trigger on insert/update/delete.

my_context_pkg.set_context
( 'cacheseq'
, nvl(to_number(SYS_CONTEXT('MY_CONTEXT','cacheseq')),0)+1
);

Updating a CONTEXT attribute is not transactional, and almost certainly not read consistent; if an update is rolled back, we may find our cache being refreshed unnecessarily. If two updates to the table are made concurrently, we might see 'cacheseq' increase by one only. But our dependent sessions should all see that there has been some update activity.

Performance


Before jumping in and using CONTEXTs, I wanted to be sure that the performance wouldn't be a limiting factor. I didn't want to find SYS_CONTEXT calls producing a hidden SELECT ... FROM DUAL as we all remember for SYSDATE, USER etc in the past. So here's a quick test:

declare
li_cached_integer integer := 0;
begin
for i in 1..100000 loop
li_cached_integer := nvl(to_number(sys_context('MY_CONTEXT','cacheseq')),0);
my_context_pkg.set_context('cacheseq',to_char(li_cached_integer+1));
end loop;
end;

Fetching the value 100,000 times (including char to number conversion) takes around 1 second; saving the modified value 100,000 times takes an additional 2 seconds (on an Intel Pentium 4 3.4Gb dual core running XP SP2 and Oracle XE).

Exporting Context Definitions


IMP/EXP won't import/export these objects, and I presume datapump won't either; so the easiest thing to do is create a SQL script using DBMS_METADATA package:

select DBMS_METADATA.GET_DDL(object_type, object_name)
from DBA_OBJECTS WHERE OBJECT_TYPE='CONTEXT';

Don't specify a schema in the call to GET_DDL, if you do, you get:

ORA-31604: invalid NAME parameter "SCHEMA" for object type CONTEXT in function SET_FILTER

Dictionary Views


There are some dictionary views that can be useful:




ALL_CONTEXTlists contexts that have been activated (ie a value has been set since startup)
DBA_OBJECTS(where object_type = 'CONTEXT')
V$CONTEXTlists attributes set in the current session (appears to ignore globally accessible attributes)

I'm planning to come back to this topic once I've had a longer play with multiple sessions setting and reading locally and globally accessible context variables.

Friday, June 30, 2006

Parting is such sweet sorrow

Shakespeare knew a thing or two about emotions, that's for sure.

Today my contract finally ends after two and a half years of repeated 3 month and 6 month extensions. There's so much I can quarrel with about how the project was architected and how it has been managed. But today I just feel glad to have been able to help, and sorry that I'll miss the next few steps.

All in all it has been a lot of fun. Thanks to everyone involved, here's hoping that the next job turns up soon (after a break at the farmouse in Italy of course) and that we can learn as much from each other as I have here.

Thursday, June 29, 2006

Anything 2.0

Doesn't it just make your head hurt when you see yet another news item, press release, blog item about XYZ 2.0?

Publishers O'Reilly claims authorship of the term Web 2.0, and for sure it has more than succeeded in its original purpose as a call to action. But (as the article mentions) the Web 2.0 (and anything 2.0) moniker has been hijacked by just about every tech PR company still up and running since the dot com shakeout (Wikipedia even defines Marketing 2.0). I started to foam at the mouth when I saw the April 2006 cover of UK venture capitalist 3i's iSIGHT magazine, which managed to cram in Trade 2.0, Business Web 2.0, Consumer 2.0 and Economy 2.0 (to round it off, they had yet another one of those hagiographic interviews with Tim Berners Lee in the back. Semantic, shemantic...).

When I see 2.0 I see a product at a point in time - and a pretty early point. Who remembers Oracle 2.0 (well, I do, but only just)? Windows 2.0? We all know that it's only from version 3 or above that a product really starts to earn its keep. To the extent that it is useful at all, "Web 2.0" is just a shorthand for certain aspects of recent developments on the web (interactivity? community? Flikr? AJAX? mash-ups? folksonomies? As the Queen says 'it means whatever I want it to mean').

Now (27 June 2006) CMP Media has actually managed to register Web 2.0 as a service mark in the US. Is there anything you can't sneak past the US Patent and Trade Office? Even though they issued a 'Final Refusal' in November 2004, over the following 18 months they evidently caved in to appeals from CMP.

I'd like to see a boycott of this lazy habit of shoving 2.0 after anything and everything (except your competitors - they're all 1.0 of course). Laughter is best; there's nothing succeeds in deflating the pretentious journo-marketeer as well as public ridicule - so please join me by posting any particularly ludicrous 2.0 claims here!