Friday, February 15, 2008

Nominative determinism

Nominative determinism: when your job or hobby matches your name. For example, the famous Major Major, from Catch 22. Or the British Antarctic Survey scientist I heard on Radio 4 yesterday explaining how he was going to drill down through 3km of ice - a Mr Core (or was it Bore - they both work, and I was in the car at the time).

The latest to catch my eye was in this quote at El Reg: "The vice president in charge of Red Hat's JBoss middleware Craig Muzilla, said Red Hat is going to undercut giants IBM, Oracle and Microsoft in the saturated and expensive middleware sector."

With a name like Muzilla, I guess working for an OSS company was pretty much inevitable.

DATAllegro, DW appliances and the Oracle lock-in effect

Andy Hayler has an interesting post on DATAllegro - talking about its sweet spot at the very high end of data volumes; apparently helped by a grid with up to 1TB/minute transfer speeds.

What caught my eye was Andy's observation that "Oracle customers can be a harder sell (than Teradata's, for example) not because of the technology, but because of the weight of stored procedures and triggers ... in Oracle's proprietary extension" (ie PL/SQL).

Just looking at the various technologies people talk about on OraNA blogs, I wonder whether the continuing growth in Java and .NET based applications over the last decade - where many of the business rules are executed outside the database - might hurt Oracle precisely by removing the PL/SQL "stickiness" Andy describes.

Admittedly Apex is a counter example - thoroughly PL/SQL, thoroughly Oracle; no chance of moving Apex clients to another RDBMS yet. And the tendency to provide management interfaces through PL/SQL (or very Oracle-specific SQL extensions) also helps to keep Oracle customers tied down. Which is good for us Oracle died-in-the-wool crowd. Whether the lock-in is good for the customers (or even Oracle) is another question; an existing customer lock-in can feel like a lock-out to new customers.

Any thoughts out there?

Regards Nigel

Tuesday, February 05, 2008

Schema Version Control

An outbreak of synchronicity has produced a volley of blogs and questions about database version control in the last few days.
I seen so many projects take code control seriously but have no real idea how to control a database definition (let alone how to control changes to multiple instances of development, test and live data, both in house and on client sites).

Code control is a walk in the park compared to "schema control" (where the delta , like an ALTER TABLE needs to be developed alongside the new version of the database object creation script).

And "data control" adds even more challenges:
  • can you upgrade the data in place? or do you need to create a replacement object?
  • Is there enough space?
  • Do you need to suspend user access? How long will it take?
  • what are the chances of reversing the upgrade if something goes wrong?
  • Will you even know if something goes wrong?
  • Who decides go forward, go back or muddle through?

A complex application version upgrade should be treated with as much respect as a data migration project. There should be a strategy (sure, it may be a standard strategy). Use standard tools / scripts. The more the process is repeatable - and the more it is repeated - the better. Make sure that all likely error conditions (missing log files, out of DB storage, non-writeable working directories, no listener) are tested for - test the tests!

By the way, the "process" includes all the operating procedures. Make sure that upgrades aren't nannied through by developers. Script the whole thing; have the production DBAs try it out the procedures as early as possible in test to make sure that they understand what's supposed to be happening (and how to know if it worked OK; the best thing is if the script just says:

IT ALL WORKED OK

or

SOMETHING WENT WRONG IN STEP 13 - SEE LOGFILE xxxyyy13

By the time the schema-and-data upgrade is executed for real, nothing - not even the unexpected - should be unexpected.

Monday, February 04, 2008

Patently ridiculous

SCO has had a pretty hostile press over the last few years as it morphed from a real software business into a patent mining outfit. Well, I've now come across another case of patent ambulance chasing over at Vincent McBurney's blog on ITToolbox. JuxtaComm and Teilhard versus IBM Microsoft Informatica SAP etc tells the story so far - basically that a small company is claiming it invented ETL, patented it in 1997, and can it now have loads-a-money if you please.

I followed Vince's link to Groklaw where InterSystems, one of the parties, is asking for help.

One of the parties is DataMirror (now part of IBM, and the purchaser of my former employer Constellar). As tne SQL Group, Constellar had an ETL system that seems to fit the key claims as early as 1994 (to my personal knowledge - it may have been on the market in 1993)

I'm stunned that (apparently) Oracle has already caved, handing over $2 million. One of the reasons I am so surprised is:
  • I was actually working for Oracle UK in 1994 - the Constellar Hub (then known as Information Junction) was evaluated for one of my clients (who ended up purchasing ETI instead).
  • Oracle UK was involved in quite a few pre-sales to Telecoms, Pharma and others over the period 94-2000.
  • Oracle US consultants helped support the product in at least a few US clients.
  • Constellar's US office was actually sublet from Oracle Corp, and I know many efforts were made to sell (the product and/or the company) to Oracle in around 1997 timescale.
  • Oracle's own WarehouseBuilder must date from around then if not earlier.

Of course, Constellar wasn't alone at the time. We competed over time against ETI, Prism, Vmark/Ardent/Ascential, Informatica, and all the others (as well as the ubiquitous roll-your-own, which is still probably the market leader).

Slipping the litigants $2 million may be cheaper than arguing the case for Oracle - but for the rest of us (especially any cash strapped open source projects) it's not an option. And I'm sure a sharp lawyer at IBM might like a bit of ammunition to see them off. Hey IBM, you're innocent!

So if any ETL pioneer (or their heirs and successors) wants my testimony to fight off JuxtaComm's malicious and meritricious suit - just let me know! I'm available for testing out the trans-atlantic business class beds any time you like...

Update 13 March 2008: more from Vincent here.

Monday, January 21, 2008

RAC Stretch Clusters - a survey

There's been an interesting digression on the Oracle-L list today on the subject of RAC stretch clusters.

A stretch cluster is one in which nodes are separated - possibly by several miles - mainly as a precaution against a complete data-centre outage. I know of a couple of examples where separations in the region of 20-30 miles (30-50 km) are either in use or planned, and other posters on Oracle-L have mentioned "several" or "a handful" of implementations worldwide.

Obviously, the main performance issue for a stretch cluster is the latency and bandwidth of the interconnect. The bandwidth isn't affected by distance, but the latency certainly is.

I'd be very interested to hear from anyone who has implemented a stretch cluster (in test or production) with as much detail as you are free to pass on, particularly:
  • what is the distance between sites
  • how many nodes at each site
  • is the workload evenly distributed (active/active homogeneous), partitioned (active/active, heterogeneous) or uneven (active/passive)
  • some indication of database size and transaction rates (in whatever units are meaningful to you
  • Any performance issues?

Either email me (nigel at preferisco dot com) or comment below. I would like to publish the results but let me know if any part(s) of your information is too sensitive to be broadcast, even anonymously.

Thanks in advance...

Friday, January 18, 2008

TiddlyWiki

I thought I'd seen all the permutations of Wikis out there: written in php, stored as files; written in PL/SQL and stored in Oracle; and with a range of different markup conventions. And recently I've successfully used wiki style markup to better decorate output from Oracle Designer (of which more later in a separate post).

Today, in one of those serendipitious chances that comes from following a couple of irrelevant but intriguing blog links, I tripped over TiddlyWiki. Rather than giving you a server-side wiki, the whole thing is packaged on the client side. A TiddlyWiki is a single HTML file that contains its own code; which self-edits as you add content. You save the file locally (and can then publish to a website if you like).

As a tool written by web designers it just looks great. You don't just click and get another page; the expanded content (called a Tiddler) zooms out at you. You can choose to expand or collapse whichever tiddlers you like - at the same time. That may be a big advantage over more boring wiki implementations.

For large communities, it's probably all wrong - personal changes in effect fork the entire wiki; but it may be a very effective personal tool - kind of mind mapping on the cheap, but much prettier than editors like treepad (and instantly portable, no software install required - keep it on your USB stick).

Basic usage instructions can be found here - written as a TiddlyWiki so you can try it out. I'm going to see if I can turn a rather unexciting Designer entity report into a thing of beauty...

Wednesday, December 19, 2007

Latest Bloor market research for Data Migration

You can find this survey either at Informatica http://www.informatica.com/info/bloorwpq407/ or at Bloor http://www.bloor-research.com/research/survey/876/data_migration_survey.html.

Generally sensible throughout but - unusually for Philip Howard's research output -it leaves some unanswered questions.

Even as a statistical forecasting effort, it does leave something to be desired. Rather a lot of tables have been included that don't contribute anythingto the sum of human knowledge - in particular a single CAGR (compound annual growth rate) of 10% appears to have been applied willy-nilly to every single region and sector (confirmed by retrofitting the figures -allowing for rounding to the nearest million). What a shame the research doesn't build on either Bloor or external figures for:-
  • likely frequency of application replacement (by industry: Banking hasa more rapid cycle than Trading Companies, and perhaps by context/application type)
  • industry/regional growth forecasts (eg from OECD)
  • No mention of government / public sector projects This is the biggest area for cockups, especially given the fragmented natureof PFI/PPI in the UK, and similar huckster schemes abroad.

However, the headline figures are worth absorbing, to give an order of magnitude for the market, and more importantly an idea of the extent of project failure.

  • 84% of data migration projects "fail" (not delivered on time and on budget)
  • Cost overruns average 30%; time overruns average 40%
  • Data migration is a $5 billion market this year (or maybe twice that if you throw in smaller projects that come in under the radar for this survery) growing to $12 billion by 2012 (sounds like a lot - but London is spending more than that on the 2012 Olympics...)

At a more detailed level, I'm not entirely comfortable with the attempt at separating "data migration" as a market in its own right from related markets (ETL, EAI, etc). I would prefer to see DM as a particular use case for ETL, EAI and DQ tools; many of the same tools will also have applications for data integration, application integration, business intelligence and other use cases - although mileage may vary.

Howard states that DM only aplies to "one off" migrations and is distinct from data integration; well, yes and no. DM may have to take place over months or years; during that period, "source" and "target" systems will have to coexist somehow. That may be achieved through "temporary" data integration/replication, or by partitioning the data and gradually handing over slices from source to target. Given the ever-changing nature of business, it is by no means inconceivable that the process never reaches its originally planned conclusion (the final decommissioning of the source system). Parts of the "source" may turn out to be worth retaining (ie the economic case for replacement is not viable).

Perhaps the biggest concern I have is that there is no mention of DM as a subset of business change. DM scoping is often imposed by (sometimes poorly considered) "business" considerations and decisions. There needs to be a feedback loop from DM processes into the overall project feasibility, scoping, planning, and costing. How many DM budgets are over budget simply because the budget was unrealistic to start with - because project costing was done using some rule of thumb [eg see John Morris's data migration blog] that happened to be inadequate in the circumstances?

Finally, I am amused (and not at all surprised) to find that hand coding is the market leader at 30%, beating ETL into second place with 28%. Given that ETL tools are often given away with applications, databases, and maybe even with leading brands of Cornflakes it is amazing that this situation hasn't much changed in the last 15-20 years since ETL tools first appeared. I always thought that DIY (roll-your-own for some american speakers) was our (Constellar's) biggest competitor back then - and here's yet more anecdotal evidence. Informatica and other tool developers - not to mention application/product architects - need to understand why that should be:

  • no tool is a magic bullet: even a market leading ETL tool may make some cases easier, but can make a few (important) cases much harder to manage (if all you have is a hammer, everything looks like a nail)
  • tools are expensive to own: they can be expensive to buy, but much more importantly skilled tool users are expensive to train or hire. The migration skill-base is fragmented (what works for PowerCenter is wrong for Oracle Data Integrator; Ascential does things differently from Ab Initio).
  • data migration projects are seen as "boring": so may not always attract the youngest, thrustingest, enterpriseyest architects. Like support, a critical area is treated like a leper in some organisations. No wonder the outcomes are sometimes suboptimal. Those cast into the outer darkness of a migration project eventually either transfer internally (and let their skills degrade), or hop off into a consultancy or frelance contract to rent out their newfound skills. Far too rarely does a competence centre emerge.
  • application architects (for in-house and packaged apps) often forget the requirement for data migration into their whizzy new systems. Nearly twenty years after it was first released, Oracle Apps still doesn't have a fully supported bulk migration solution for even the most basic data (eg Payables 11.5.10 finally added vendors, sites and contacts but still doesn't support their bank accounts...). What's unglamourous for the end user is equally unglamourous for the product developer, it seems.

Ho ho ho! Happy Christmas everyone

Wednesday, December 05, 2007

I cite, you borrow, he steals...

Anyone who appreciates the hard work that goes into a decent presentation or book should read Doug Burns' tale of woe. Isaac Newton is widely quoted acknowledging his debt to other scientists: "If I see further, it is because I am standing on the shoulder of giants". Don't let's tolerate those who seek to take the credit for another's efforts without due courtesy or recompense.

I find it particularly annoying as Doug (whose blog I read regularly, but whom I have never met in person) seems to be particularly (even pathologically?) careful to acknowledge where he gets his ideas from.

Take care to read the comments as well as the blog itself. This one will run and run...

Happy UKOUG by the way (from darkest Lancashire at the moment)!

Wednesday, November 07, 2007

Tombstone, AZ

Just back from a fortnight in Arizona, where we were visiting #1 daughter on exchange at UofA in Tucson. We took in Flagstaff, Williams, the Grand Canyon and Sedona as well as sights in and around Tucson itself.

A personal favourite was the Boothill Graveyard on the outskirts of Tombstone. It seems to have operated only for a very few years - and most of the occupants apparently died a sudden death - shot or hanged in the main (one is specifically shown as 'died of natural causes').

George Johnson was more unlucky than most. I wonder how they found out their mistake?
Posted by Picasa

Wednesday, October 17, 2007

Got an issue with that?

Here's a little summary of issue management systems I found in the Register for use with (agile) software products. It struck a chord with me - not least because I am right now working with a project team of around 30 trying to manage and organise a range of issues on an ever changing series of spreadsheets, using email as the repository.

I could add the example issue tracking system that comes with Oracle Apex - has anyone tried that?

Other wikis with issue management templates / recipes are available - including PmWiki PITS.

Monday, October 15, 2007

Latest Gartner Magic Quadrant for Data Integration

Informatica has helpfully syndicated the latest data integration magic quadrant. Out goes Ab Initio - apparently due to their perennial problems with secrecy (ie they won't tell anyone anything). Only IBM and Informatica make it to poll position (IBM slightly bolstered by its recent purchase of DataMirror). Hummingbird is the back marker, both for vision and ability to execute. Anyway, read it yourselves; no major surprises in there (and why would there be in such a mature market...).

Saturday, October 13, 2007

Is there WebLogic in Oracle's bid for BEA?

Finally, Larry has got out his wallet and made a bid for BEA( which of course they have rejected). This one has been waiting to happen for several years. Oracle's experience with app servers hasn't been entirely happy - but after a slow start they have managed to steamroller their way to a decent market share (often on the back of Oracle Apps); meanwhile most of the (often less well funded) pioneers have fallen by the wayside. Who remembers Persistence? Gemstone? Whatever happened to Sun's variously named products?

So who's left now? Oracle, IBM, BEA; MS IIS for the Netties, and a handful of low/no cost options led by JBoss (aka Red Hat).

Buying BEA presents an interesting marketing challenge in the J2EE app server area - though at least Oracle only has one app server at a time. Thing are much more exciting (ie confused) in the integration / add-ons market; Oracle Fusion is already in a state of upheaval, digesting bits and pieces from Siebel, Sunopsis, Hyperion and others. Throwing in BEA AquaLogic - which itself combines organic development and some BEA acquisitions, I seem to remember - will be "interesting" to say the least.

And don't forget other jewels at BEA. Tuxedo - a real piece of engineering - something Oracle could usefully combine with the best of its own server technology; and JRockit - a high performance JVM.

For those who hate acquisitions, remember: BEA bought in both JRockit (from a startup) and Tuxedo (originally developed at AT&T, bought in 1995)... and come to that, WebLogic itself (in 1998).

Thursday, October 11, 2007

Onwards and upwards

I've recently started a new contract which should prove very interesting. Starting off with a data migration study, I'm expecting there will be lots of performance planning and assurance to follow.

The setting is a rather large and unusual Oracle Apps implementation - client confidential of course. RAC, stretch cluster, massive batch processing, impossible response and resilience targets? Lots for me to (re)learn, and hopefully some useful previous experience to share.

The 250 mile weekly commute is a bit of a pain...

Bottoms up!

Tuesday, September 11, 2007

Cognos partners with Informatica

Cognos and Informatica have announced a strategic relationship; Cognos will sell and support Informatica Data Quality and Data Explorer products, and the two companies will "team to jointly provide customers with data integration capabilities" with a focus on performance management (rather than the wider data integration market?).

Interesting that Informatica's front page still highlights the May press release about being granted an injunction against Business Objects, while Cognos includes this release on its front page. A minor difference in marketing communications, or a subtle indication of who is the senior partner?

Analysis from The Street here, and from Intelligent Enterprise here.

Saturday, September 01, 2007

Finished at last (well, almost)


It's finally complete - we're just back from a lovely break in Sant'Angelo at the new house. Not entirely a holiday - there was lots of cleaning to do, and dozens of lights to buy and wire in (more to come later, along with a significant amount of Ikea furniture to be trucked in and assembled - in November we hope). And I spent most mornings working (those pesky deadlines again) proving at least that the EDGE service (no UMTS in the mountains) and a 20 euro for 500Mb web deal from TIM were enough to keep me going.

The garden remains - a couple of 80 year old olives should have gone in this week, and work will start on the rest in the autumn.
Nothing to do now but work as hard as possible to pay the bills!

Thursday, July 19, 2007

No more mirror, mirror jokes... IBM snaffles DataMirror

IBM bought my former employers DataMirror earlier this week. Obviously their main interests will be the HA product line (a direct value add for them), and then the Transformation Server (TS) real time data integration components. Others have written here and here, so I won't labour that except to say that as part of the package they have won Constellar Hub (formerly dear old Information Junction - there's a phrase that doesn't escape my lips too often).

DataMirror never really "got" Constellar. Sure, I think they sweated the customers enough to make a return on their CDN$10million investment (what a steal that was!), but they never made a serious effort (afaik) to bring together the best of the Hub and TS. A shame, because I think it could have been a winner compared to the still rather tired 1990s EAI products like Informatica, DataStage, Ab Initio etc. In fact, TS+Hub could have turned out not unlike the Oracle Data Integrator (but with added "publish / subscribe", and strong IBM and Oracle partnerships).

Now the boot is on the other foot. Will IBM "get" DataMirror and its various product lines? All we can be reasonably sure of is that the Hub is now officially past its sell by date...

Am I glad I left DataMirror in 2001, and so didn't get bought again 6 years later? On balance, I think so. But I know some who have stayed all along - will they stay with IBM too?

Monday, July 09, 2007

Iona FUSEs Celtix and LogicBlaze

Iona says it is expanding customers' open SOA choices by bringing together under the FUSE banner products that were formerly part of Celtix ESB, or were acquired with the LogicBlaze purchase this April.

CBR points out that Iona seems to be favouring LogicBlaze supported components (Apache projects ActiveMQ, ServiceMix and Camel) over Celtix components (apart from CXF, which has moved over from ObjectWeb to Apache). The LogicBlaze purchase must be working out well, methinks...

Friday, July 06, 2007

Talend - another open source ETL tool

Bloor's Philip Howard writes at IT-Director about Talend Open Studio - unusual among ETL solutions in that it is a code generator (of Java, SQL, Perl) rather than an "engine" type of product.

As well as the usual drag-n-drop transformation GUI, this apparently supports business process modelling - which gives Talend a feature that many "real" ETL/EAI tools don't have. There's also support for using a server grid to parallelise processing, and there's an "on demand" SaaS offering.

Version 2.0 is now available, as well as a 2.1 release candidate which is said to add features including:

  • further optimizations for performance increase
  • support of new databases (including bulk load)
  • transaction management (connection sharing, commit and rollback)
  • Slowly Changing Dimensions support
  • MOM (Message Oriented Middleware) support for real-time integration jobs
  • fuzzy logic data matching (using the Levenshtein and metaphone algorithms)
  • normalization, denormalization and flow merge
  • support of SSH remote connections
  • support of PGP file decryption (through GPG binary)
  • reinforced support of the XML standard: DTD validation, XSD, XSLT transformation, significant improvements in hierarchical XML file generation, support of the XMLRPC Web Services protocol…
  • improvements to the tMap component, to support input filters and new joins types such (the cartesian product, first match, last match…)

Sounds like it might be worth a closer look...

Zoning in to Sonos

We've finally bitten the bullet and shelled out for a 2 zone Sonos music system. I see we're not alone; Rob Levy, BEA's CTO, says it is his favourite gadget - and Joel Spolsky has also endorsed it.

We started around Christmas buying 2 Squeezeboxes - because I'm cheap, and because the Sonos system needs at least one box on wired Ethernet - which we didn't have handy in the right rooms. Although the Squeezebox looks good, it didn't really deliver as I had hoped.

The worst things about the Squeezebox:

  • it's not very good at coping with poor wireless reception - which probably accounts for most of the other problems

  • although you can synchronise two boxes (say one in the kitchen, and one in the living room) they tend to drift apart; not a good sound!

  • The slimserver software is incredibly slow at (re)indexing

  • the whole solution relies on slimserver software installed (by an importer) on a linux NAS fileserver - that makes getting updates much harder.

  • Internet radio was listed, but I could never get it to work

  • Documentation and help was rather flaky

  • The boxes look great but the UI is crummy - I never managed to explain it to my wife (or even to myself...)




So we cut our losses and invested in two Sonos ZP80s - they're quite small boxes, which feed into your existing amplifiers (you can use ZP100 if you want a built in amp). Rather than recable our house, we got a couple of powernet adapters. Music sits on a (simple) NAS disc - no intelligence required there.

We had some difficulty getting the powernet working - in the end after three hours trying every feasible combination of sockets in our (1st floor) office and (ground floor) living room, the very helpful installer swapped the original Netgear products for Microlink and found a pair of sockets that worked. Fantastic! after that everything was very straightforward. There's a (pricey) i-Pod like controller - or you can use a desktop interface. Once you've sussed the Zones and Music buttons, it's a piece of cake to navigate.

It's using the same music library as Windows Media Player - so I can load music once, and use it either on my headphones from the laptop, or on the Sonos system. Having the same music in two zones is easy - and very reliable. Adding new zones is also very simple (I just know we'll need a third zone in the office...). And internet radio works a treat - though with a slightly robotic quality, from the selection of sites supported (plenty of UK ones - helpful for us as we get almost nothing on DAB or FM).

The Squeezeboxes won't be entirely wasted; they'll go out to Italy this summer, where they should benefit - eventually - from being wired in (assuming the builders have remembered the CAT-5...).

Sunday, July 01, 2007

Graphic example

Thanks to Oracle AppsLab for referencing this stunning presentation by Hans Rosling. Not just for the superbly informative animated graphics, but also for the moving and thoughtful content (and the sting in the tail - watch all 19 minutes for the climax).