lab-notes

Revolutions End II and The Memory Wall

2012-01-08T17:08:00.000-08:00

The 2011 ITRS report for the first time uses the terms, "ultimate Silicon scaling" and "Beyond CMOS". The definitive industry report is highlighting for us that the end of the Silicon Revolution is in sight, but that won't be the end of the whole story. Engineers are very clever people and will find ways to keep the electronics revolution moving along, albeit at a much gentler pace.

In 2001, the ITRS report noted that CPU's would be hitting a Power Wall, they'd need to forgoe performance (frequency) to fit within a constrained power envelope. Within 2 years, Intel was shipping multi-core CPU's. Herb Sutter wrote about this in "The Free Lunch is Over".

In the coming 2011 ITRS report, they write explicitly about "Solving the Memory Wall".
Since 1987 and the Pentium IV, CPU clock frequency (also 'cycle time') has been increasing faster than DRAM cycle times: by roughly 40% per year. (7%/year for DRAM and ~50%/year for CPU chip freq.)

This is neatly solved, by trading latency for bandwidth, with caches.
The total memory bandwidth needs for multi-core CPU's doesn't just scale with the chip frequency (5%/year growth), but with the total number of cores accessing the cache (number of cores grow at approx 40%/year).
Cache optimisation, the maximisation of cache "hit ratio", requires the largest cache possible. Hence Intel now has 3 levels of cache, with the "last level cache" being shared globally (amongst all cores).

The upshot of this is simple: to maintain good cache hit-ratios, cache size has to scale with the total demand for memory access. i.e. N-cores * chip freq.
To avoid excessive processor 'stall', waiting for the cache to be filled from RAM, the hit-ratio has to increase as the speed differential increases. An increased chip frequency requires a faster average memory access time.
So the scaling of cache size is: ( N-cores ) * (chip freq)²

The upshot is:

cache memory has grown to dominate CPU chip layout and will only increase.

But it's a little worse than that...
The capacity growth of DRAM has slowed to a doubling every 3-4 years.
In 2005, the ITRS report for the first time dropped DRAM as its "reference technology node", replacing it with Flash memory and CPU's.

DRAM capacity growth is falling behind total CPU chip memory demands.
Amdahl posited another law for "Balanced Systems": that each MIP required 1MB of memory.

Another complicating factor is bandwidth limitations for "off-chip" transfers - including memory.
This is called "the Pin Bottleneck" (because external connections are notionally by 'pins' on the chip packaging). I haven't chased down the growth pattern of off-chip pins. The 2011 ITRS design report discusses it, along with the Memory Wall, as a limiting factor and a challenge to be solved.

As CPU memory demand, the modern version of "MIPS", increases, system memory sizes must similarly scale or the system becomes memory limited. Which isn't a show-stopper in itself, because we invented Virtual Memory (VM) quite some time back to "impedance match" application memory demands for with available physical memory.

The next performance roadblock is VM system performance, or VM paging rates.
VM systems have typically used Hard Disk (HDD) as their "backing store", but whilst the capacity has grown faster than any other system component (doubling every year since ~1990), latency, seek and transfer times have risen comparatively slowly. Falling, relatively, behind CPU cycle times and memory demands by 50%/year (??).

For systems using HDD as their VM backing store, throughput will be adversely affected, even constrained, by the increasing RAM deficit.

There is one bright point in all this, Flash Memory has been doubling in capacity as fast as CPU memory demand, and increasing in both speed (latency) and bandwidth.

So much so, that there are credible projects to create VM systems tailored to Flash.

So our commodity CPU's are evolving to look very similar to David Patterson's iRAM (Intelligent RAM) - a single chip with RAM and processing cores.

Just how the chip manufacturers respond is the "$64-billion question".

Perhaps we should be reconsidering Herb Sutters' thesis:

Programmers have to embrace parallel programming and learn to create large, reliable systems with it to exploit future hardware evolution.

Big Drives in 2020

2011-12-24T20:02:00.000-08:00

Previously I've written about Mark Kyrder's 7TB/platter (2.5 inch) prediction for 2020.
This is more speculation around that topic.

1. What if we don't hit 7TB/platter, maybe only 4TB?

There have been any number of "unanticipated problems" encountered with scaling Silicon and Computing technologies, will more be encountered with HDD before 202?

We already have 1TB platters in 3.5 inch announced in Dec-2011, with at least one new technique announced to increase recording density (Sodium Chloride doping), so it's not unreasonable to expect another 2 doublings in capacity, just in taking what's in the Labs and figuring out how to put it into production.

Which means we can expect 2-4TB/platter (2.5 inch) to be delivered in 2020.
At $40 per single-platter disk?
That depends on a) the two major vendors and the oligopoly pricing and b) the yields and costs of the new fabrication plants.

Seems to me that Price/GB will drop, but maybe not to levels expected.
Especially if the rapid decline in SSD's/Flash Memory Price/Gb plateaus and removes price competition.

2. Do we need to offer The Full Enchilada to everyone?

Do laptop and ultrabook users really need 4TB of HDD when they are constantly on-line?
1-2TB will store a huge amount of video, many virtual machine images and a lifetimes' worth of audio.
There might be a market for smaller capacity disks, either through smaller platters, smaller form-factors or underusing a full-width platter.

Each option has merits.
The final determinant will be perceived consumer Value Proposition, the Price/Performance in the end-user equipment.

3. What will the 1.8 inch market be doing?

If these very small form-factor drives in mobile equipment get to 0.5-2TB, that will seem effectively infinite.

There is no point in adopting old/different platter coatings and head-manufacturing techniques for these smaller form-factors unless other engineering or usability factors come into play: such as sensitivity to electronic noise, contamination, heat, vibration, ...

4. The fifth-power of diameter and cube-of-RPM: impact of size and speed?

2.5 inch drives are set to completely displace 3.5 inch in new Enterprise Storage systems within a year. This is primarily driven by Watts/GB and GB/cubic-space.

The aerodynamic drag of disk platters, hence the power consumed by a drive, varies with the fifth-power of platter diameter and the cube of the rotational velocity (RPM).

If you halve the platter size (5.25 inch to 2.5 inch), drive power reduces 32-fold.
If you then double the RPM of the drive (3600 to 7200), power increases 8-fold,
a nett reduction in power demand of 4 times.

Changing platter diameter by square-root of 2 (halving the recordable area), the drive power reduction is 5.5-fold. This is the same proportion for 5.25::3.5 inch, 3.5::2.5 inch and 2.5::1.8 inch.

Reducing a 2.5 inch platter to 1.92 inches allows a drive to be spun up from 5400 RPM to 7200 RPM whilst using the same drive power, with 60% of the original surface area.

Whilst not in the class of Enterprise Storage "performance optimised" drives (10K and 15K), it would be a noticeable improvement for Desktop PC's, given they will also be using large SSD's/Flash Memory as well in 2020 and this will be solely for "Seek and Stream" tasks.

There is very little reason to "de-stroke" drives and limit them to less than full-platter access if they are not "performance-optimised". It's a waste of resource for exactly the same input cost.

5. Will 3.5 inch "capacity-optimised" disks survive?
Will everything be 2.5 or 1.8 inch form-factor?

There are 3 markets that are interested in "capacity-optimised" disks:

Storage Appliances [SOHO, SME, Enterprise and Cloud]
Desktop PC
Consumer Electronics: PVR's etc.

When 1TB 2.5 inch drives are affordable, they will make new, smaller and lighter Desktop PC designs possible. Dell and HP might even offer modules that attach on the 100mm x 100mm "VIA" standard to the back of LCD screens. A smaller variant of the Apple Mac Mini is possible, especially if a single power-supply is available.

Consumer PVR's are interested in Price/GB, not Watts/GB. They will be driven by HDD price.
The manufacturers don't pay for power consumed, customers don't evaluate/compare TCO's and there is no legislative requirement for low-power devices. Government regulation could be the wild-card driving this market.

There's a saying something like this I though made by Dennis Ritchie:
"Memory is Cheap, until you need to buy enough for 10,000 PC's".
[A comment on MS-Windows lack of parsimony with real memory.]

Corporations will look to trimming costs of their PC (laptop and Desktop) fleets, and the PC vendors will respond to this demand.

Storage Appliances:
Already Enterprise and Cloud providers are moving to 2.5 inch form-factor to reduce power demand (Watts/GB) and floor-space footprint (GB/cubic-space).

Consumer and entry-level servers and storage appliances (NAS and iSCSI) are currently mostly 3.5 inch because that has always been the "capacity-optimised" sweet spot.

Besides power-use, the slam-dunk reasons for SOHO and SME users to move to 2.5 inch are:

lighter
smaller

smaller footprint and higher drive count per Rack Unit.
more aggregate bandwidth from higher actuator count
mirrored drives or better, are possible in a small portable and low-power case.

more robust, better able to cope with knocks and movement.

2.5 inch drives may be much better suited to "canister" or (sealed) "drive-pack" designs, such as used by Copan in their MAID systems. This is due to their lighter weight and lower power dissipation.
The 14-drive Copan 3.5 inch "Canister" of 4RU could be replaced by a 20-24 drive 2.5 inch Canister of 3RU, putting 3-4 times the number of drives in the same space.

6. What if there are some unforeseen "drop-deads", like low data-retention rates or hyper-sensitivity to heat, that limit useful capacities to the current 3-600GB/platter (2.5 inch)?

We can't know the future perfectly, so can't say just what surprises lie ahead.
If there is some technical reason why current drive densities are an engineering maximum, we cannot rely on technology advances to automatically reduce the Price/GB each year.

Even if technology is frozen, useful price reductions, albeit minor in comparison to "50% per year", will be achievable in the production process. It might take a decade for prices to drop 50% per GB.

I'm not sure how exactly designs might be made scale if drive sizes/densities are pegged to current levels.
What is apparent and universal, "Free Goods" with apparently Infinite Supply, will engender Infinite Demand.

If we do hit a "capacity wall", then the best Social Engineering response is to limit demand, which requires a "Cost" on capacity. This could be charging, as Google does with its Gmail service, or by other means, such as publicly ranking "capacity hogs".

IDC on Hard Disk Drive market: Transformational Times

2011-12-22T17:35:00.000-08:00

One of the problems, as an "industry outsider", of researching the field is lack of access to hard data/research. It's there, it's high-quality and timely. Just expensive and behind pay-walls.

A little information leaks via Press Releases and press articles promoting the research companies.

When one of these professional analyst firms makes a public statement alerting us to a radical restructuring of the industry, that's big news. [Though you'd expect "insiders" to have been aware of this for quite some time.]

What's not spelled out publicly, is How will this impact Enterprise Storage vendors/manufacturers?
There seems an implication that the two major HDD vendors will start to compete 'up' the value-chain with RAID and Enterprise Storage vendors, and across storage segments with Flash memory/SSD vendors.

IDC's Worldwide Hard Disk Drive 2011-2015 Forecast: Transformational Times
published in May, 2011. This report consists of Pages: 62 and the price starts from US $ 4500.

Headline: Transformation to just 3 major vendors. (really 2 major + 1 minor @ 10%)

"The hard disk drive industry has navigated many technology and product transitions over the past 50 years, but not a transformation. [emphasis added]

The HDD industry is poised to consolidate from five to three HDD vendors by 2012, and
HDD unit shipment growth over the next five years will slow.

HDD revenue will grow faster than unit shipments after 2012, in part because HDD vendors will offer higher-performance hybrid HDD solutions that will command a price premium.

But for the remaining three HDD vendors to achieve faster revenue growth,
it will be necessary by the middle of the decade for HDD vendors to transform into [bullets added]
storage device and
storage solution suppliers,
with a much broader range of products for a wider variety of markets
but at the same time a larger set of competitors."

Platters per Disk.

2011-12-22T16:55:00.000-08:00

Headline:

2.5 inch:

9.5mm = 1 or 2 platters
12.5mm = 2 or 3 platters
15mm = ? platters. Guess at least 3. 4 unlikely, compare to 3.5 inch density

3.5 inch

25.4mm = commonly 4. Max. 5 platters.

Why is this useful, interesting or important?
To compare capacity across form-factors and for future configuration/design possibilities.

Disk form-factors are related by an approximate halving of platter area between sizes:

8::5.25 inch, 5.25::3.5 inch, 3.5::2.5 inch, 2.5::1.8 inch, 1.8::1.3 inch, 1.3::1 inch...

What we (as outsiders) know, but only approximately, is the recording area per platter for the platter sizes. We know there are at least 3 regions of disk platter, but not their ratios/sizes, and these will vary per form-factor/platter-size:

motor/hub. The area of the inside 'torus' is small, not much is lost.
recorded area
outer 'ring' for landing and idling or "unloading" heads. Coated differently (plastic?) to not damage heads if they "skid" or come into contact with a surface (vs 'flying' on the aerodynamic air-cushion).

Research:

Chris Mellor, 12th September 2011 12:02 GMT, The Register, "Five Platters, 4TB".

Seagate has a 4TB GoFlex Desk external drive but this is a 5-platter
disk with 800GB platters.

IDC, 2009, report sponsored by Hewlett-Packard:

By 2010, the HDD industry is expected to increase the maximum number of platters per 2.5inch performance-optimized HDD from two to three,
enabling them to accelerate delivering a doubling of capacity per drive, and subsequently achieving 50% capacity increases per drive over a shorter time frame.

7th September 2011 06:00 GMT, The Register.

Oddly Hitachi GST is only shipping single-platter versions of these new drives, although it is saying they are the first ones in a new family, with their 569Gbit/in2 areal density. The announced but not yet shipping terabyte platter Barracuda had a 635Gbit/in2 areal density.

Sebastian Anthony on December 12, 2011 at 12:25 pm, Extreme Tech

Hitachi, seemingly in defiance of the weather gods, has launched the
world’s largest 3.5-inch hard drive:
The monstrous 4TB Deskstar 5K.
With a rotational speed of 5,900RPM,
a 6Gbps SATA 3 interface,
and the same 32MB of cache as
its 2 and 3TB siblings,
the 4TB model is basically the same beast
— just with four platters instead of two or three.
The list price is around $345

Silverton Consulting, 13-Sep-2011:

shipping over 1TB/disk platter using 3.5″ platters shipping with 569Gb/sqin technology

"Missed by _that_ much": Disk Form Factor vs Rack Units

2011-12-19T23:27:00.000-08:00

Apologies to 1965 TV series "Get Smart" and the catch-phrase "Missed by that much" (with a visual indication of a near-miss).

This is a lament, not a call to action or grumble. Standards are necessary and good.
We have two standards that we just have to live with now: too many devices depend on them for a change. Unlike the "imperial" to metric conversion, there would be few discernible benefits.

There's a fundamental mismatch between the Rack Unit (1.75 inches) or the vertical space allowed for equipment in 19 inch Racks (standard EIA-310) and the Disk form factors of 5.25, 3.5 and 2.5 inches defined by the Small Form Factor Committee.

There is no way to mount a standard disk drive (3.5 or 2.5 inch) exactly in a Rack. There are various amounts of wasted space.
Originally, "full-height" 5.25 inch drives could be mounted horizontally exactly in 2 Rack Units (3.5 inches), three abreast.

The "headline" size of the form-factor is the notional size of the platters or removable media.
The envelope allows for the enclosure.

So whilst "3.5 inch" looks like a perfect multiple of the 1.75 inch Rack Unit, a "3.5 inch" drive is around 0.5 inch larger.
Manufacturers of vertical-mount "hot-swap" drives allow around 1mm on the thinnest dimension, 9 mm on the "height" and 1.5 inches (42-43 mm) on the longest dimension (depth).

A guess at the dimensions of hot-swap housings:
1/32 in (0.8mm) or 1mm sheet metal could be used between drives (upright)
and 1.5-2mm sheet metal would be needed to support the load (with an upturned edge?).

In total, around 0.5 inch (12.5mm) might need be allowed vertically for supporting structures.
An ideal Rack Unit size for the "3.5 inch" drive form-factor would be 4.5 inches.

Or, "3.5 inch" drives could be 3.00 - 3.25 inches wide to fit exactly in 2 Rack Units.

Different manufacturers approach this problem differently:

Copan/SGI and Backblaze mount 3.5 inch drives vertically in 4 Rack Units (7 inches).
Both of these solutions aim for high-density packing, 28 and 11.25 drives per Rack Unit .

Copan, via US Patent # 7145770, uses 4U hot-swap "canisters" that store 14 drives in 2 rows, with 8 canisters per "shelf" (112 drives/shelf). In a 42 U rack, they can house 8 shelves, for 896 drives per Rack. Their RAID system is 3+1, with max 5 spares per shelf, yielding 79 data drives per shelf, and 632 drives per Rack.
These systems are designed specifically to hold archival data, with up to 25% or 50% of drives active at any one time, as "MAID": Massive Array of Idle Disks.
Backblaze are not a storage vendor, but have made their design public with a hardware vendor able to supply cases and pre-built (but not populated) systems.
Their solution, fixed-disks not hot-swap, is 3 rows of 15 disks mounted vertically, sitting on their connectors. The Backblaze systems include a CPU and network card and are targeted at providing affordable and reliable on-line Cloud Backup services [and are specifically "low performance"]. Individual "storage pods" do not supply "High Availability", there is little per-unit redundancy. Like Google, Backblaze rely on whole-system replication and software to achieve redundancy and resilience.

Most server and storage appliance vendors use "shelves" of 3 Rack Units (5.25 inches), but fit 13-16 drives across the rack (~17.75 inches or 450mm) depending on their hot-swap carriers.
"2.5 inch" drives fitted vertically (2.75 inch) need 2 Rack Units (3.5 inches). Most vendors fit 24 drives across a shelf. "Enterprise class" 2.5 inch drives are typically 12.5 or 15 mm thick.

Another possibility, not been widely pursued, is to build disk housings or shelves that don't exactly fit the EIA-310 standard Rack Units. Unfortunately, the available internal width of 450mm cannot be varied.

The form factors:

"5.25 inch": (5.75 in x 8 in x 1.63 in = 146.1 mm x 203 mm x 41.4 mm)
"3.5 inch" : (4 in x 5.75 in x 1 in = 101.6 mm x 146.05 mm x 25.4 mm)
"2.5" inch : (2.75 in x 3.945 in x 0.25-0.75 in = 69.85 mm x 100.2 mm x [7, 9.5, 12.5, 15, 19] mm)

Old disk height form factors, originating in 5.25 inch disks circa mid-1980's.

low-profile = 1 inch.
Half-height = 1.63 inch.
Full-height = 3.25 inch. [Fitting well into 2 Rack Units]

"Disk is the new Tape" - Not Quite Right. Disks are CD's

2011-12-14T02:39:00.000-08:00

Jim Gray, in recognising that Flash Memory was redefining the world of Storage, famously developed between 2002 and 2006 the view that:

Tape is Dead
Disk is Tape
Flash is Disk
RAM Locality is King

My view is that: Disk is the new CD.

Jim Gray was obviously intending that Disk had replaced Tape as the new backup storage media, with Flash Memory being used for "high performance" tasks. In this he was completely correct. Seeing this clearly and annunciating it a decade ago was remarkably insightful.

Disks do both the Sequential Access of Tapes and Random I/O.
In the New World Order of Storage, they can be considered functionally identical to Read-Write Optical disks or WORM (Write Once, Read-only Memory).

As the ratios between access time (seek or latency) and sequential transfer rate, or throughput, continues to change in favour of capacity and throughput, managing disks becomes more about running them in "Seek and Stream" mode than doing Random I/O.

With current 1TB disks, the sequential scan time (capacity ÷ sustained transfer rate) [1,000GB/ 1Gbps] is 2-3 hours. However, to read a disk with 4KB random I/O's at ~250/sec (4msec avg. seek), the type of workload a filesystem causes, gives an effective through put of around 1MB/sec, or 128 times slower than a sequential read.

It behoves system designers to treat disks as faster RW Optical Disk, not as primary Random IO media, and as Jim Gray observed, "Flash is the New Disk".

The 35TB drive (of 2020) and Using them.

2011-12-14T01:18:00.000-08:00

What's the maximum capacity possible in a disk drive?

Kryder, 2009, projects 7TB/platter for 2.5 inch platters will be commercially available in 2020.

Given that prices of drive components are driven by production volumes, in the next decade we're likely to see the end of 3.5 inch platters in commercial disks with 2.5 inch platters taking over.
The fith-power relationship between platter-size and drag/power-consumed also suggests "Less is More". A 3.5 inch platter needs 5+ times more power to twirl it around than a 2.5 inch platter - the reason that 10K and 15K drives run the small platters: they already use the same media/platters for 3.5 inch and 2.5 inch drives.

Sankar, Gurumurthi, and Stan in "Intra-Disk Parallelism: An Idea Whose Time Has Come" ISCA, 2008, discuss both the fifth-power relationship and that multiple actuators (2 or 4) make a significant difference in seek times.

How many platters are fitted in the 25.4 mm (1 inch) thickness of a 3.5 inch drive's form-factor?

This report on the Hitachi 4TB drive (Dec, 2011) says they use 4 * 1TB platters in a 3.5 inch drive, with 5 possible.

It seems we're on-track to at least the Kryder 2020 projection, with 6TB per 3.5 inch platter already demonstrated using 10nm grains enhanced with Sodium Chloride.

How might those maximum capacity drives be lashed together?

If you want big chunks of data, then even in a world of 2.5 inch componentry, it still makes sense to use the thickest form-factor around to squeeze in more platters. All the other power-saving tricks of variable-RPM and idling drives are still available.
The 101.6mm [4 inch] width of the 3.5 inch form-factor allows 4 to sit comfortably side-by-side in the usual 17.75 inch wide "19 inch rack", using just more than half the 1.75 inch height available.

It makes more sense to make a half-rack-width storage blade, with 4 * 3.5 inch disks (2 across, 2 deep) with a small/low-power CPU, a reasonable amount of RAM and "SCM" (Flash Memory or similar) as working-memory and cache and dual high-speed ethernet, infiniband or similar ports (10Gbps) as redundant uplinks.
SATA controllers with 4 drives per motherboard are already common.
Such "storage bricks", to borrow Jim Grays' term, would store a protected 3 * 35Tb, or 100TB per unit, or 200Tb per Rack Unit (RU). A standard 42RU rack, allowing for a controller (3RU), switch (2RU), patch-panel (1RU) and common power-supplies (4RU), would have a capacity of 6.5PB.

Kryder projected a unit cost of $40 per drive, with the article suggesting 2 platters/drive.
Scaled up, ~$125 per 35TB drive, or ~$1,000 for 100TB protected ($10/TB) [$65-100,000 per rack]

The "scan time" or time-to-populate a disk is the rate-limiting factor for many tasks, especially RAID parity rebuilds.
For a single actuator drive using 7TB platters and streaming at 1GB/sec, "scan time" is a daunting 2 hours per platter: At best 10 hours to just read a 35TB drive.

Putting 4 actuators in the drive, cuts scan time to 2-2.5 hours, with some small optimisations.

While not exceptional, its compares favourably with 3-5 hours minimum currently reported with 1TB drives.

But a single-parity drive won't work for such large RAID volumes!

Leventhal, 2009, in "Triple Parity and Beyond", suggested that the UER (Unrecoverable Error Rate) of large drives would force force parity-group RAID implementations to use a minimum of 3 parity drives to achieve a 99.2% probability of a successful (Nil Data Loss) RAID rebuild following a single-drive failure. Obviously, triple parity is not possible with only 4 drives.

The extra parity drives are NOT to cover additional drive failures (this scenario is not calculated), but to cover read errors, with the assumption that a single error invalidates all data on a drive.

Leventhal uses in his equations:

512 byte sectors,
1 in 10^16 probability of UER,
hence one unreadable sector per 200 billion (10TB) read, or
10 sectors per 2 trillion (100TB) read.

Already, drives are using 4Kb sectors (with mapping to the 'standard' 0.5Kb sectors) to achieve the higher UER's. The calculation should be done with the native disk sector size.

If platter storage densities are increased by 32-fold, it makes sense to similarly scale up the native sector size to decrease the UER. There is a strong case for 64-128Kb sectors on 7Tb platters.

Recasting Leventhal's equations with:

100TB to be read,
64KB native sectors,
or 1 in 1.5625 * 10^9 native sectors read for a UER of 1 in 10^16.

What UER would enable a better than 99.2% probability of reading 1.5 billion native sectors?
First approximation is 1 in 10^18 [confirm].
Zeta claims UER better than 1 in 10^58. Is possible to do much better.

Inserting Gibson's "horizontal" error detection/correction (extra redundancy on the one disk) is around the same overhead, or less. [do exact calculation].

Rotating parity or single-disk parity RAID?

The reasons to rotate parity around disk are simple - avoid "hot-spots", otherwise the full parallel IO bandwidth possible over all disks is reduced to just that of the parity disk. NetApp neatly solve this problem with their WAFL (Write Anywhere File Layout).

In order to force disks into mainly sequential access, "seek then stream", writes won't be simply cached, but shouldn't be written to HDD but kept to SMC/Flash until writes have quiesced.

The single parity-disk problem only occurs on writes. Reading, in normal or degraded mode, occurs at equal speed.

If writes across all disks are stored then written in large blocks, there is no IO performance difference between single-parity disk and rotating parity.

Revolutions End: Computing in 2020

2011-12-13T18:09:00.001-08:00

We haven't reached the end of the Silicon Revolution yet, but "we can see it from here".

Why should anyone care? Discussed at the end.

There are two expert commentaries that point the way:

David Patterson's 2004 HPEC Keynote, "Latency vs Bandwidth", and
Mark Kryder's 2009 paper in IEEE Magnetics, "After Hard Drives—What Comes Next?"
[no link]

Kryder projected the current expected limits of magnetic recording technology in 2020 (2.5": 7Tb/platter) and how another dozen technologies will compare, but there's no guarantee. Some unanticipated problem might, like CPU's, derail Kryders' Law before then: disk space doubles every year.
We will get an early "heads-up": by 2015 Kryder expects 7Tb/platter to be demonstrated.

This "failure to fulfil the roadmap" has happened before: In 2005 Herb Sutter pointed out that 2003 marked the end of Moore's Law for single-core CPU's in "The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software". Whilst Silicon fabrication kept improving, CPU's hit a "Heat Wall" limiting the clock-frequency, spawning a new generation of "multi-core" CPUs.

IBM with its 5.2GHz Z-series processors and gamers "over-clocking" standard x86 CPUs showed part of the problem was a "Cooling Wall". This is still to play out fully with servers and blades.
Back to water-cooling, anyone?
We can't "do a Cray" anymore and dunk the whole machine in a vat of Freon (a CFC refrigerant, now banned).

Patterson examines the evolution of four computing techologies over 25 years from ~1980 and the increasing disparity between "latency" (like disk access time) and "bandwidth" (throughput):

Disks
Memory (RAM)
LANs (local Networking)
CPUs

He neglects "backplanes", PCI etc, Graphic sub-systems/Video interfaces and non-LAN peripheral interconnection.

He argues there are 3 ways to cope with "Latency lagging Bandwidth":

Caching (substitute different types of capacity)
Replication (leverage capacity)
Prediction (leverage bandwidth)

Whilst Patterson doesn't attempt to forecast the limits of technologies like Kryder, he provides an extremely important and useful insight:

If everything improves at the same rate, then nothing really changes
– When rates vary, require real innovation

In this new milieu, Software and System designers will have to step-up to build systems that are effective and efficient, and any speed improvements will only come from better software.

There is an effect that will dominate bandwidth improvement, especially in networking and interconnections (backplanes, video, CPU/GPU and peripheral interconnects):

the bandwidth-distance product

This affects both copper and fibre-optic links. Using a single technology, a 10-times speed-up shortens the effective distance 10-times. Well know in transmission line theory.

For LANs to go from 10Mbps to 100Mbps to 1Gbps, higher-spec cable (Cat 4, Cat 5, Cat 5e/6) had to be used. Although 40Gbps and 100Gbps Ethernet have been agreed and ratified, I expect these speeds will only ever be Fibre Optic. Copper versions will either be very limited in length (1-3m) or use very bulk, heavy and expensive cables: worse in every dimension than fibre.

See the "International Technology Roadmap for Semiconductors" for the expert forecasts of the underlying Silicon Fabrication technologies, currently out to 2024. There is a lot of detail in there.

The one solid prediction I have is Kryder's 7Tb/platter.
A 32 times increase in bit-areal density, Or 5 doublings of capacity.
This implies the transfer rate of disks will increase 5-6 times, given there's no point in increasing rotational speed, to roughly 8Gbps. Faster than "SATA 3.0" (6Gbps) but within the current cable limits. Maintaining the current "headroom" would require a 24Gbps spec - needing a new generation of cable. The SATA Express standard/proposal of 16Gbps might work.

There are three ways disk connectors could evolve:

SATA/SAS (copepr) at 10-20Gbps
Fibre Optic
Thunderbolt (already 2 * 10Gbps)

Which type to dominate will be determined by the Industry, particularly the major Vendors.

The disk "scan time" (to fully populate a drive) at 1GB/sec, will be about 2hours/platter. Or 6 hours for a 20Tb laptop drive, or 9 hours for a 30Tb server class drive. [16 hours if 50TB drives are packaged in 3.5" (25.4mm thick) enclosures]. Versus the ~65 minutes for a 500Gb drive now.

There is one unequivocal outcome:
Populating a drive using random I/O, as we now do via filesystems, is not an option. Random I/O is 10-100 times slower than streaming/sequential I/O. It's not good enough to take a month or two to restore a single drive, when 1-24 hours are the real business requirements.

Also, for laptops and workstations with large drives (SSD or HDD), they will require 10Gbps networking as a minimum. This may be Ethernet or the much smaller and available Thunderbolt.

A caveat: This piece isn't "Evolutions' End", but "(Silicon) Revolutions' End". Hardware Engineers are really smart folk, they will keep innovating and providing Bigger, Faster, Better hardware. Just don't expect the rates of increase to be nearly as fast. Moores' Law didn't get repealed in 2003, the rate-of-doubling changed...

Why should anyone care? is really: Who should care?

If you're a consumer of technology or a mid-tier integrator, very little of this will matter. In the same way that now when buying a motor vehicle, you don't care about the particular technologies under the hood, just what it can do versus your needs and budget.

People designing software and systems, the businesses selling those technology/services and Vendors supplying parts/components hardware or software that others build upon, will be intimately concerned with the changes wrought by Revolutions End.

One example is provided above:
backing up and restoring disks can no longer be a usual filesystem copy. New techniques are required.

RAID: Something funny happened on the way to the Future...

2011-12-07T18:46:00.001-08:00

With apologies to Stephen Sondheim et al, "A Funny Thing Happened on the Way to the Forum", the book, 1962 musical and 1966 film.

Contents:

Summary:
Robin Harris of "StorageMojo" in "Google File System Eval", June 13th, 2006, neatly summaries my thoughts/feelings:

As regular readers know, I believe that the current model of enterprise storage is badly broken.

Not discussed in this document is The Elephant in the Room, or the new Disruptive Technology: Enterprise Flash Memory or SSD (Solid State Disk). It offers (near) "zero latency" access and random I/O performance 20-50 times cheaper than "Tier 1" Enterprise Storage arrays.

Excellent presentations by Jim Gray about the fundamental changes in Storage are available on-line:

2006 "Flash is good": "Flash is Disk, Disk is Tape, Tape is dead".
2002 "Storage Bricks". Don't ship tapes or even disks. Courier whole fileservers, it's cheaper, faster and more reliable.

[Top]
The current state of the Enterprise Storage market: Pricing and Budget Impact.

The promise inherent in the first RAID paper's title (Inexpensive Disks) doesn't seem to be met.
Are there other challenges, limits or oddities?

Budget Impact and per-GB pricing
A new entrant, Coraid, talking-up the benefits of its solution lays out some disturbing statistics:

With storage cost consuming 25% to 45% of IT budgets ... (ours) offer up to a 5-8x price-performance advantage over legacy Fibre Channel and iSCSI solutions.

Vendor Gross Margins
A commentary on one of the 6-7 dominant players (EMC, IBM, Network Appliance, Hewlett-Packard, Hitachi Data Systems, Dell, SUN/Oracle) who control 70-80% of the market by revenue: [compare to gross margins on Intel servers of 20-30%]:

Committing to massive disruption of the storage and networking businesses by moving to 40% gross margins from the current 60+% range through the use of volume and scale out technologies, open source software and targeted innovation leveraging the latest technology.

Can HP compete with 40% gross margins? Well, Apple has done pretty well. [Apple is known for 'premium pricing' and the best returns in the industry.]

A large, mature market
The market is large and growing, according to IDG:

... end-user spending on enterprise storage systems reached $30.8 billion in 2010, and the 18% growth over 2009 was the highest since IDC started tracking the market in 1993.
The enterprise storage systems market will grow at a comfortable 3.9% compound annual growth rate (CAGR) between 2010 and 2015 ...

Poor utilisation of raw disk capacity?
IBM, touting the benefits of it's XIV Storage , claims massive waste by others' systems:

Overall, we find that the reliability attributes of the system limit the net capacity of a system to 46%-84% depending, for the most part, on the RAID configuration. [46% for a RAID-1 (mirror) config. 84% for RAID-5 (parity/check disk)]

Overall, we see that, by virtue of its built-in efficiency, the XIV system uses 100% of its net capacity, compared with an estimated 28-61% net capacity used by comparable systems. [IBM list 3 types of "wasted space": Orphaned/Unreclaimable Space, Full Backups, Thick Provisioning. Clones and Backups are replaced by 'snapshots' in XIV. IBM neglect filesystem/database 'slack space' within allocated storage.]

The combined effect of the reliability and efficiency attributes is such that, on average, a traditional storage system using mirroring effectively uses less than 21% of its raw capacity (37% when using RAID-5). An XIV system uses approximately 46% of its raw capacity.

In the absence of good Operational Expenditure [OpEx] data, a guess
There is speculation that the Operational costs of 'Tier 1' (high-performance, high-availability, most expensive) storage is very high, but no good figures are available [Compare this recurrent cost to the retail price of 1TB SATA disks of ~$100, while fast, durable SAS drives, e.g. 146-160GB, are $200-300, 300-600GB are ~$400]:

With an estimated cost of Tier 1 storage at around $8,000 per TB per year,

Indirect Total Cost of Ownership [TCO] estimates
Hitachi Data Systems lays out the costs ownership and product conversion/data migration when pushing the benefits of their "virtual storage" architecture:

Forrester 2007 research has shown that in excess of 70 percent of enterprise IT budgets is devoted to maintaining existing infrastructure.
Migration project expenditures are on average 200 percent of the acquisition cost of enterprise storage.
With an average of four years useful life, the annual operating expenses associated to migration represent ~50 percent of acquisition cost.

Enterprise storage migration costs can exceed US$15,000 per terabyte migrated. [implying 2007(?) acquisition cost of $7,500/Tb.]
For example, an average FORTUNE 1000® company has an average of 800TB of network attached storage (NAS) and nearly 3PB of storage (InfoPro Wave 12–Q2, 2009) with, on average, 300TB per storage system
As the useful life of most storage systems is three to five years, ...

Current admin challenges
The size and complexity of current Enterprise Storage solutions, and the resulting administrator workload, has provoked comments along the lines quoted below. System complexity increases combinatorially as additional layers are added and heterogeneous systems and networks are interfaced. This increases admin workload, task difficultly and execution times, consequentially increasing preventable faults and errors.

Today, storage is the single most complex and expensive component in the virtualized data center.

How do Enterprises choose between Vendors and Products?
Many commentators assert that Storage is bought on a single metric: Price per GB.
Comparing other important performance metrics, {latency, IO/second, throughput MB/sec} for "random" and "sequential" I/O workloads is being addressed by the Storage Performance Council, with their SPC1 and SPC2 specifications. For those vendors who choose to participate and publish data on their systems, it provides an "Apples and Apples" comparison for potential customers: including system pricing and discount information.

But there's a problem: system price unrelated to cost of drives.
Overwhelmingly, the price of the raw disk drives is an almost insignificant fraction of the purchase price of "Tier 1" Storage arrays. Competitors claim that the dominant players price their disks up to 30 times the normal retail price, hiding the true cost of the infrastructure wrapped around the drives. Vendors often load special firmware in their drives to prevent substitution. [SPC1 and SPC2 detailed pricing confirms published retail prices of $1-2,000 per drive, 5-20 times retail prices.]

This pricing practice distorts customer system specifications by purchasing far fewer drives, creating additional administrative work in managing allocated disk space and achieving target performance levels.

What is missing is good data on: Price / GB-available-to-Applications.
[Ignoring all the other overheads and "slack space" for Logical Volume Managers and Operating Systems

With Enterprise Storage systems, this is at least 50-100 times the raw cost of drives.

The lesson: Optimising the utilisation of the cheapest and paradoxically least available resource in a system is poor practice.

Meanwhile, there are significant technical and performance issues looming in the world of Enterprise Storage:

In Triple Parity RAID and Beyond, Adam Leventhal, ACM Queue, Dec 2009, suggests that by 2020 three parity drives will be needed to achieve a 99.2% probability of a successful RAID rebuild recovering from a single drive failure. Multi-parity drives introduce new problems:

increases Price per GB (more drives for same capacity),
reduces write performance (1P = 4 IO, 2P = 6 IO, 3P = 8 IO, NP = 2*(N+1) IO)
increases compute intensity for parity calculations (1P uses trivial 'XOR')
increases system complexity in efforts to compensate for performance, such as delayed parity writes and caching.
system robustness and durability is adversely affected by increased component count and software complexity.

RAID rebuilds severely affect access times and throughput and now take from 3-24 hours, up from "minutes" in the first systems. Documented in "Comparison Test: Storage Vendor Drive Rebuild Times and Application Performance Implications", Feb 18, 2009, Dennis Martin. There are anecdotal reports of RAID rebuilds taking up to a week, degrading performance of all tasks and leaving organisations with protected data for the duration.

[Top]
Disk Drives: Characteristics and Evolution.

The architecture and organisation of Enterprise Storage Systems are driven by Usage demands and the underlying storage components.
What's gone before and what might becoming?

In "A Converstaion with Jim Gray", ACM Queue, 2003, the evolution and limits of Hard Disk Drive (HDD) technology is discussed, projected to occur around 2020:

But starting about 1989, disk densities began to double each year. Rather than going slower than Moore’s Law, they grew faster. Moore’s Law is something like 60 per-cent a year, and disk densities improved 100 percent per year.

Today disk-capacity growth continues at this blistering rate, maybe a little slower. But disk access, which is to say, “Move the disk arm to the right cylinder and rotate the disk to the right block,” has improved about tenfold. The rotation speed has gone up from 3,000 to 15,000 RPM, and the access times have gone from 50 milliseconds down to 5 milliseconds. That’s a factor of 10. Bandwidth has improved about 40-fold, from 1 megabyte per second to 40 megabytes per second. Access times are improving about 7 to 10 percent per year. Meanwhile, densities have been improving at 100 percent per year.

At the FAST [File and Storage Technologies] conference about a year-and-a-half ago, Mark Kryder of Seagate Research was very apologetic. He said the end is near; we only have a factor of 100 left in density—then the Seagate guys are out of ideas. So this 200-gig disk that you’re holding will soon be 20 terabytes, and then the disk guys are out of ideas. [now revised to 14Tb @ $40 for 2.5 inch drive]

The definitive paper on the limits and evolution of hard disk technology, including a comparison with a dozen other prospective technologies, by Mark Kryder :

"After Hard Drives—What Comes Next?" Kryder and Chang Soo Kim. IEEE Magnetics, Oct 2009.

Assuming HDDs continue to progress at the pace they have in the recent past, in 2020 a two-disk, 2.5-in disk drive [2 platters] will be capable of storing over 14 TB and will cost about $40.
Given the current 40% compound annual growth rate in areal density, this technology should be in volume production by 2020. [Expect a demonstration in 2015]

In 2005, Scientific American discussed his work and described "Kyrder's Law", in recent years HDD capacity has doubled every year, outstripping Moore's Law for CPU speed.

Sankar, Gurumurthi and Stan, ICSA 2008, describe the relationship of power consumption to RPM and platter size necessary to understand current drive design [reformatted]:

Since the power consumption of a disk drive is
proportional to the fifth-power of the platter size,
is cubic with the RPM, and
is linear with the number of platters... [citing a 1990 IEEE paper]

The external form-factor defines capacity of current HDD's in surprising ways:

The 2.5 inch form-factor allows thickness to vary between 7 mm and 19mm, though 19mm is now unusual.

Compare to the 25.4mm (1 inch) thickness of 3.5" drives.
Consumer drives, in laptops and PC's, are now normally 9.5mm, with some 7mm. [single platter?]
Enterprise drives are 15mm, allowing higher capacities by including more platters. [2-3?.
In 2020, Enterprise 2.5 inch drives will be 2 or 3 platters, hence 14-21TB.

The maximum platter size is not used in every drive.

For consumer drives and high-capacity/low-energy Enterprise drives, the largest platters possible are used.
For high-speed (10,000 [10K] and 15,000 RPM [15K]) drives, the same size platters are used in both 3.5 inch and 2.5 inch drives. This reduces power consumption and seek time through reduced head travel.
Vendors sell 3.5 inch 15K drives because they can fit more platters in the 25.4 mm vs 15 mm form factor. [4 platters are common, 5 platters are "possible"]
"More than an interface — SCSI vs. ATA", Anderson, Dykes, Riedel, Seagate, FAST 2003.

A fundamental characteristic of HDD's, "access density, or IOPS/GB" and its importance and evolution as discussed in 2004 by a disk drive manufacture. [Last row added to table].

1987 2004 times increase
CPU Performance 1 MIPS 2,000,000 MIPS 2,000,000 x
Memory Size 16 Kbytes 32 Gbytes 2,000,000 x
Memory Performance 100 usec 2 nsec 50,000 x
Disc Drive Capacity 20 Mbytes 300 Gbytes 15,000 x
Disc Drive Performance 60 msec 5.3 msec 11 x
Disc Drive Bandwidth 0.6 MB/sec 86 MB/sec 140 x Patterson 2004

Figure 2. Disc drive performance has not kept pace with other components of the
system.

	1987	2004	times increase
CPU Performance	1 MIPS	2,000,000 MIPS	2,000,000 x
Memory Size	16 Kbytes	32 Gbytes	2,000,000 x
Memory Performance	100 usec	2 nsec	50,000 x
Disc Drive Capacity	20 Mbytes	300 Gbytes	15,000 x
Disc Drive Performance	60 msec	5.3 msec	11 x
Disc Drive Bandwidth	0.6 MB/sec	86 MB/sec	140 x	Patterson 2004

Last row data from 2004 Patterson address on "Latency and Bandwidth", covering 20+ years evolution of all system components.

The vendor table omits "transfer rate" or "bandwidth", necessary to calculate the other fundamental characteristic of HDD's, described by Leventhal [ACM Queue 2009], disk "scan time":

By dividing capacity by throughput, we can compute the amount of time required to fully scan or populate a drive.

This storage analyst discussion of access density, cost and capacity using SPC benchmarks delves into more fine detail on this and related topics.

RAID array IO/sec performance suffers through the declining access density, which can addressed through higher RPM drives, "short-stroking"/"de-stroking" (using outer 20% of drives) or provisioning the number of drives ('spindles') not on capacity, but desired IO/sec. (Most designers and storage architects would consider this as "over-provisioning" and unnecessarily expensive.)

RAID rebuild times cannot proceed faster than the individual drive scan time, whilst in normal RAID-5 configurations, the total amount of data read for an (ND+1P) parity group, is N-times the drive size. The minimum time taken is (N*Disk-size ÷ common-channel speed), the parity-group scan time, analogous to single-drive scan time. The Networking concept of "over-subscription", the ratio of uplink to total downlink capacity (1:1 is ideal but expensive, higher numbers cause congestion in high-performance environments like server rooms), can be applied to the common-channel and supported drives.

[Top]
The Early View

Why are we now facing these problems?
Is this the future that the instigators of RAID foresaw?

In 1987/8 when Patterson, Gibson and Katz at UCB (University of California, Berkeley) wrote their industry-changing paper, "A Case of Redundant Arrays of Inexpensive Disks", their group was working on new processing architectures (MPP - Massively Parallel Processing) amongst other things.

Patterson et al describe an application scale-up problem as CPU's speed increases faster than disks and proffer this: "A Solution: Arrays of Inexpensive Disks".

1.2 The Pending I/O Crisis
What is the impact of improving the performance of some pieces of a problem while leaving others the same? Amdahl's answer is now known as Amdahl's Law...

Suppose that some current applications spend 10% of their time in I/O. Then when computers are 10X faster - according to Bill Joy in just over three years - then Amdahl's Law predicts effective speedup will be only 5X. When we have computers 100X faster - via evolution of uniprocessors or by multiprocessors - this application will be less than 10X faster, wasting 90% of the potential speedup.

and

In transaction-processing situations using no more than 50% of storage capacity, then the choice is mirrored disks (Level 1). However, if the situation calls for using more than 50% of storage capacity, or for supercomputer applications, or for combined supercomputer applications and transaction processing, then Level 5 looks best.

They also suggest that storage arrays will commonly be comprised of very large numbers of disks:

MTTR and thereby increase the MTTF of a large system. For example, a 1000 disk level 5 RAID with a group size of 10 and a few standby spares could have a calculated MTTF of 45 years.

But explicitly pose as unsolved questions, just how this might be accomplished:

Can information be automatically redistributed over 100 to 1000 disks to reduce contention?
Will disk controller design limit RAID performance?
How should 100 to 1000 disks be constructed and physically connected to the processor?
Where should a RAID be connected to a CPU so as not to limit performance? Memory bus? I/O bus? Cache?

A further insight into the thinking of these pioneers is found in the Introduction of "RAIDframe: A Rapid Prototyping Tool for RAID Systems" by Courtright, Gibson, et al in 1997. They give a very exact description of how they envision RAID arrays developing [many tiny drives] and why [access time and bandwidth].
Unfortunately, the 1.3 inch drive by HP (C3014 'Kittyhawk') cited was introduced in early 1992 and discontinued due to slow sales at the end of 1994.
The 1 inch drives they forecast did appear in 1999, the IBM Microdrive, packaged as Compact Flash (CF) cards, and while for a number of years were the largest capacity available in the CF form-factor. They were discontinued in 2006 when Flash Memory overtook them in capacity and Price per GB, never having made it into the Enterprise Storage market.
Wikipedia (Dec 2011) claims that by 2009 all drives smaller than 1.8 inch (used in portable devices) have been discontinued.

Another good source from 1995 is "ISCA’95 Reliable, Parallel Storage Tutorial", Garth Gibson, CMU. Gibson co-authored the original Berkeley RAID paper.

From the "Raidframe" paper:

Further impetus for this trend derived from the fact that smaller-form-factor drives have several inherent advantages over large disks:
smaller disk platters and smaller, lighter disk arms yield faster seek operations,
less mass on each disk platter allows faster rotation,
smaller platters can be made smoother, allowing the heads to fly lower, which improves storage density,
lower overall power consumption reduces noise problems.
These advantages, coupled with very aggressive development efforts necessitated by the highly competitive personal computer market, have caused the gradual demise of the larger drives.

In 1994, the best price/performance ratio was achieved using 3.5 inch disks, and the 14-inch form factor has all but disappeared.

The trend is toward even smaller form factors:
2.5 inch drives are common in laptop computers [ST9096], and
1.3-inch drives are available [HPC3013].
One-inch-diameter disks should appear on the market by 1995 and should be common by about 1998. [appeared 1999, only achieved commercial success in Digital Cameras]
At a (conservative) projected recording density in excess of 1-2 GB per square inch [Wood93], one such disk should hold well over 2 GB of data. [got there about 2002]
These tiny disks will enable very large-scale arrays.
For example, a one-inch disk might be fabricated for surface-mount, rather than using cables for interconnection as is currently the norm, and thus a single, printed circuit board could easily hold an 80-disk array. [did they mean 8x11in cards? see Gibson's 1995 tutorial, Slide 5/74, for a diagram]

Several such boards could be mounted in a single rack to produce an array containing on the order of 250 disks.
Such an array would store at least 500 GB, and even if disk performance does not improve at all between now and 1998, could service either 12,500 concurrent I/O operations or deliver 1.25-GB-per-second aggregate bandwidth.

The entire system (disks, controller hardware, power supplies, etc.) would fit in a volume the size of a filing cabinet.

To summarize, the inherent advantages of small disks, coupled with their ability to provide very high I/O performance through disk-array technology, leads to the conclusion that storage subsystems are, and will continue to be, constructed from a large number of small disks, rather than from a small number of powerful disks.

[Top]
The Gap: Then and Now.

Around 25 years on, have the original expectations been met?
Has the field and technology developed as they envisioned?
What are the unresolved questions?

Whilst EMC released one of the first commercial RAID system, one of the two commercial systems described in the UCB groups' followup 1994 paper, "RAID: High-Performance, Reliable Secondary Storage", was the Storage Technology Iceberg 9200, released that year. The Iceberg and the 1989 Berkeley RAID-I prototype used (32 and 28) 5.25 inch drives. The Berkeley RAID-II, written up in 1994, graduated to 72-144 * 320MB 3.5 inch drives supplied by IBM.

Whilst the main thrust of the 1987/8 paper was recommending the use of the smallest disk drives available, at the time 3.5 inch, this was not the practice, especially in commercial systems.

Even in the beginning, vendors and their customers, chose differently, using 5.25 inch drives.
The direct impact on system design was that the Iceberg needed dual-parity to achieve sufficient data-protection, especially for read-errors when rebuilding a RAID parity group after a single-disk failure. The UBER (Unrecoverable Bit Error Rate) of 10 in 10^14 and total parity-group size of 8-16 GB gave an unacceptably high chance of a rebuild failure.

The 5.25 inch drive was the smallest 'enterprise' drive available. The Fujitsu Super Eagle, at 10.5 inches, was discussed in the 1988 paper. At the same time 8 inch SCSI drives were on the market.

In 2005, Herb Sutter wrote a commentary on the end of Moore Law for single-core CPU's "The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software". Early in 2003, CPU's hit a "Heat Wall" limiting the clock-frequency. Whilst more transistors could be placed on a CPU die, the clock-frequency (speed) stalled. To improve CPU throughput, designers placed more cores in each CPU, the equivalent to a network connection using more parallel conductors to increase total bandwidth without increase the speed of an individual connection/conductor.

In contrast to this sudden change, Hard Disk access density has been steadily eroding whole array performance for the last two decades, steadily forcing design changes and increasing complexity.

It's unlikely that when fundamental limits are reached an "IO Performance Wall" will be created for hard disks. Though, like CPU's, some unexpected physical constraint may limit realisable capacities.

The projected maximum areal density of disk drives results in fixed sizes for 2.5 inch and 3.5 inch platters. The Industry Standard form-factors fix the size available, and potentially the number of platters possible in each form factor. With the advent of cheap Flash Memory in SSD's, there is little need for high RPM Hard Drives.

There has been one major form factor conversion, from 5.25 inch to 3.5 inch. Currently 'Tier 1' storage is being sold in both 3.5 inch and 2.5 inch form factors. To understand if 2.5 inch drives will become universal, we need to examine the history of the last conversion and the differences to now.

Slide 25/74, "Disk Diameter Trends", of Garth Gibson's ISCA’95 Reliable, Parallel Storage Tutorial covers these transitions:

Decreasing diameter dominated 1980s
• 5.25” created desktop market (16+ GB soon)
• 3.5” created laptop market (4+ GB 1/2 high; 500+ MB 19mm)
• 2.5” dominating laptop market (200+ MB; IBM 720 MB)
• 1.8” creating PCMCIA disk market (80+ MB)

Decreasing diameter trend slowed to a stop
• 1.8” market not 10X 2.5” market
• 1.3” (HP Kittyhawk) discontinued
• vendors continue work on smaller disks to lower access time

It isn't clear when 5.25 inch drives went out of production and Enterprise storage swapped completely to 3.5 inch drives. Seagate's 10Gb "Elite 9" ST410800N (the ST410800N manual, static URL) was released in 1994. The cut-over time will have been before the product's end-of-life in 1997/8.
Slide 20/74 of Gibson's 1995 tutorial plots the year of introduction of different drive capacities by drive size. It tracks 5.25 inch drives until 1994 with 10GB introduced, confirming the first estimate.

The reasons will have been complex and many, but at some point all the significant metrics would've favoured 3.5 inch drives and then result was foregone.

Metrics that matter to Enterprise Storage vendors:

Price per GB

Both form factors are in high-volume production, with a small premium for 2.5 inch drives.
Laptops and notebooks have outsold desktop PC's for a number of years, at least balancing demand for 2.5 inch drives.

GB per cubic-unit (normally per "Rack Unit" (RU) in a "19 inch" rack (1.75 in x 17.75 in x 24-36 in))

Standard vertical mounting sees 13-16 3.5 inch hot-swap drives in a 3RU space, and
24 2.5 inch drives in a 2RU space.
4.3-5.3 drives/RU vs 12 drives/RU.
The ratio of platter area, a close approximation to capacity per platter, is 2:1, making the ratio about 5:6.
2.5" drives a 20% higher storage density per platter for a given technology,
But there extra platters in 25.4 mm vs 15 mm. (50% more platters)
For high RPM drives with identical capacities, 2.5 inch wins,
for same-RPM drives, 3.5 inch drives store
Orienting 3.5 inch drives horizontally, 3-high by 4-wide (12) will fit in 2RU, the best packing is 6/RU.

Watts per GB and implied operational costs.

In higher RPM drives, Watts per GB currently is comparable (shown above).
For 'slow' 7200RPM drives, the fifth-power-law of platter diameter to power consumption means 2.5 inch drives will consume significantly (535%!) less power. (3-4W vs 15W)
Western Digital now sell a line of 'Green' 3.5 inch drives with variable RPM, reducing power demand considerably.
Operational costs, e.g. cooling capacity and cost of electricity, over the 5 year service life of equipment (~45,000 hours) may be the deciding factor.
1,000 * 3.5 inch drives will consume ~650,000 kilo-Watt-hours vs 125,000 kWhrs.
At $0.25/kWhr, a lifetime saving of ~$135 per drive.

Average access time (based on rotational latency and seek times).

Smaller drives can be spun faster and still consume less power.

[Top]

A little production theory: Why "popular" drives are much cheaper.

Accounting Theory has the concept of the "Learning Curve", also called "Experience Curve".
It originated with Wright in 1936 examining small-scale production and noting that doubling production, reduces costs 10-15%.

Does the Experience Curve scale-up?
If you produce 100,000 times more, (16+ doublings), can you realise these benefits all the way?

10%-per-doubling improvement = 18.5% (81.5% cost reduction)
15%-per-doubling improvement = 7.5% (92.5% cost reduction)

Large-scale Silicon chip production has very high "barriers to entry", the cost of building "fabrication plants" is now measured in Billions. As components are made smaller, with tighter tolerances, volume production plants for Hard Disk Drives must be following the same trend. To make "next generation" products with features and tolerances 70% the size of the current technology, every process is affected, needing to be more precise, cleaner and with less vibration. These new technologies don't come cheaply.

These plant costs and the product Research and Development costs have to be amortised across all units produced. For fixed and overhead costs to be a small fraction of the per-unit cost, the number of units now has to be very large: 10-100 million.

In 2011, there are forecast to be 400M+ PC's (desktops and laptops) produced. Laptops, using 2.5 inch drives, passed desktops as the most popular format around 5 years ago. These high volume demands underpin the production of both 3.5 inch and 2.5 inch disk drives.

Volume production counts for a lot.
If there isn't already a high demand for a product, the per-unit price will be considerably inflated to cover "sunk costs": everything involved in creating the product.
Once the plant and Research costs are paid for, per-unit costs can be lowered more.
After a time the Price/Capacity of old technology products, even when fully amortised, will be too much higher than new technology and demand will fall away rapidly.
When demand falls and production runs are too small, the plant becomes uneconomic because fixed-costs (operational/production costs) start to overwhelm the per-unit price. Manufacturers must close plants when they start to make losses: the situation can only get worse, rapidly.

As an example, consider the two machines that Seymour Cray designed for Control Data Corporation (CDC) in the 1960's/70's before leaving to create his own company:

CDC 6600 World's fastest computer: 1964-1969.
CDC 7600 World's fastest computer: 1969-1975. reputedly $5M (equivalent to $30M now).

Both CDC and Cray himself knew everything there was to know about both ECL and TTL technologies they used, yet by the mid-1990's, all the fastest computers were VLSI CMOS: the CPU chips in use today.

The difference between Intel et al and Cray/CDC was volume production.
When you hand-craft 100-1,000 machines, they cost millions.
When you build CPU's by the million, they cost $100-$1,000.
It redefines how you architect and organise super-computers.

This "substitution effect" affects all commodity products.

[Top]

Further Questions

Economics asserts "Price is the Mediator between Supply and Demand".
In a mature, free market, where purchasers have "perfect information" and products are fungible (perfect substitutes for each other), competition will drive prices down (and consumption of goods will increase) and inefficient producers will be driven from the market.

But the Enterprise Storage market has the very high gross margins associated with new markets or non-substitutable products.

Q: What's happening between vendors and purchasers to produce this skewed market?

Are Enterprise Storage products fungible or not?
Are secondary effects at work preventing product substitution? [warranty conditions, staff capability, integrated platform management software, decision inertia/product loyalty or technical conservatism.]
Is the purchasing criteria "Price/GB", I/O performance, (perceived) Reliability/Availability etc, functionality or something else?

Performance (latency and throughput) of HDD-based Enterprise Storage arrays is rated "good enough" in the market because for 10-15 years many new entrants have attempted to break into the market by offering low-cost or high-performance products.

Q: What "figure of merit" do purchases use to chose between vendors and products? Is it "just" price/GB, because it isn't "performance".

Without extensive customer research, this may be unknowable.

There aren't convincing reasons favouring smaller or larger form-factor drives unless high RPM drives or SSD (packaged in 2.5 inch or 1.8 inch) are included.

Q: Why has the take-up of SSD's in Enterprise environments been slow when the price/performance ratios are so far ahead of 'conventional' Storage systems?

Kryder's Law, the on-going compound increase in HDD size and price/GB has caused some changes in fundamental ratios:
Storage Arrays initially lashed together many small devices into "large enough" logical/virtual devices.
Somewhere in the last 25 years, drive capacity exceeded normal use cases by multiples.

The basic RAID I/O performance drivers for both latency and throughput, many spindles and actuators reduce latency and parallel transfers increase throughput, were invalidated, but the designs seemed not to change.

1988: IBM 3380 7.5GB. Databases, Files fitted within this limit.

RAID from 320MB-1GB, more spindles, more IO/sec, higher throughpu

2010: 600GB 10/15K, 2TB SATA drives.

These units are now (much!) larger than most common Databases and file stores.
Not just video, images, audio, scanned stuff. MS-Office documents as well.

Q: Why did Storage arrays not respond to this change in fundamental drivers?

Did the change happen so slowly that nobody noticed?
Storage vendors are generally very innovative and competitive, employing some of the "best and brightest" in computing.
This wasn't a failure of ability or capability. Perhaps of vision?
Are incumbent vendors locked into their own solutions, leaving innovation to new entrants?
Did consumers demand products they were familiar and comfortable with and prevent vendor changing designs?

Why didn't large numbers of really small HDD's get tried by major vendors, even as an experiment?
IBM, the leader in 1 inch drives, sold it's drive business to Hitachi Data Systems (HDS), a leading Storage system vendor with expertise in many related areas.

HDS had the capability to create custom packaging, custom electronics (ASIC's) and to redesign the 1 inch drive format (Compact Flash with IDE). For very small drives to be soldered onto boards, a simple, continuous serial interface was needed. SAS, Serial Attached SCSI, would fit the bill today.

In the 3 years before 1 inch HDD's lost their price advantage to Flash memory, HDS could have built a prototype and proven the concept, but (seemingly) didn't. Nor did any academic projects.

Q: Why did the Industry and Academic researchers not build a "many tiny drives" Array between 2000 and 2005?

To show that very low overhead Storage devices can be built outside Google datacentres, these are the full on-line instructions and Bill of Materials for one service providers solution:

"Petabytes on a budget: How to build cheap cloud storage", September 1, 2009
"Petabytes on a Budget v2.0: Revealing More Secrets", July 20, 2011

135TB for $7,384, around 50% more than the raw cost of the drives.

A chart in the first piece comparing the Price / GB of different solutions.

RAW:               $81,000 [660 @ 1.5TB ] 45 @ $120 == $5400
backblaze:        $117,000 [50% o'head]
Dell MD1000       $826,000
SUN/Ora X4550   $1,000,000
NetApp FAS-6000 $1,714,000
Amazon S3       $2,806,000
EMC NS-960      $2,860,000

Pix of Storage Pod & component costs.

Optical Disks as Dense near-line storage?

2011-11-28T13:31:00.001-08:00

A follow-up to an earlier post [search for 'Optical']:

Could Optical Disks be a viable near-line Datastore?
Use a robot arm to pick and load individual disks from 'tubes' into multiple drives.
Something the size of a single filing cabinet drawer would be both cheap and easily contain a few thousand disks. That's gotta be interesting!

Short answer, no...

A 3.5" drive has a form-factor of: 4 in x 5.75 in x 1in. Cubic capacity: 23 in³
A 'tube' of 100 optical disks: 5.5in x 5.5in x 6.5in Cubic capacity: 160-200 in³ [footprint or packed]

A 'tube' of 100, minus all the supporting infrastructure to select a disk and read it, is 7-9 times the volume of a 3.5in hard disk drive, or each Optical Disk must contain at least 7-9% of a HDD to be competitive.

To replace a 1Tb HDD, optical disks must be at least 7% of 1,000Gb, or 70Gb. Larger than even Blu-ray and 15-20 times larger than Single layer DVD's (4.7Gb).

Current per Gb price of 3.5" HDD's is around $0.05-$0.10/Gb, squeezing 4.7Gb DVD's on price as well.

2Tb drives are common, 3Tb are becoming available now (2011). Plus it gets worse.

There are estimate is of maximum possible 3.5" HDD size of 20-40Tb.
To be competitive, Optical disks would need to get up around 1Tb in size and cost under $1.

Around 2005, when 20-40Gb drives reigned, there was a time when 4.7Gb DVD's were both the densest and cheapest storage available. Kryders' Law, a doubling of HDD capacity every 1-2 years, has seen the end of that.

Journaled changes: One solution to RAID-1 on Flash Memory

2011-11-27T22:48:00.001-08:00

As I've posited before, simple-minded mirroring (RAID-1) of Flash Memory devices is not only a poor implementation, but worst-case.

My reasoning is: Flash wears-out and putting identical loads on identical devices will result in maximum wear-rate of all bits, which is bad but not catastrophic. It also creates a potential for simultaneous failures where a common weakness fails in two devices at the one time.

The solution is to not put the same write load on the two devices, but still have two exact copies.
This problem would be an especial concern for PCI-SSD devices internal to a system. The devices can't normally can't be hot-plugged, though there are hot-plug standards for PCI devices (e.g. Thunderbolt and ExpressCard), they are not usually options for servers and may be limited performance.

One solution, I'm not sure if it's optimal or not, but it is 'sufficient', is to write blocks as normal to the primary device and maintain the secondary device as snapshot + (compressed) journal entries. When the journal space hits a high-water mark the snapshot is made an exact copy (e.g. bring the snapshot up-to-date when a timer expires (hourly, 6-hourly, daily, ...) or when the momentum of changes will fill the journal to 95% before the snapshot could be updated).

If the journal fills, the mirror is invalidated and either changes must be halted or the devices go into unprotected operation. Both not desirable operational outcomes. A temporary, though unprotected, work-around is to write the on-going journal either to the primary device or into memory.

Implementation Outline:

With existing devices, a set of blocks on both devices need to be allocated for the journal. Whilst the journal area won't be 'written to' on the primary device, it needs to be there:

so identical data areas are available on both devices
if the primary and secondary devices are swapped, another device designated the new primary - either as an additional device or a replacement for the secondary.

I'd prefer additional chips be added to the Flash devices specifically for journaling. NOR chips, expensive but not as prone to wear, could even be used. [If similar speed.]

Better techniques of dealing with the journal as a set of version-changes with unique keys (e.g. sequence number) would allow a device to be removed from a mirror and rejoined with minimal updates, avoiding a slow and expensive full-copy. This edge-case would benefit from writing the journal to the primary. One of the most annoying behaviours of RAID systems is "popping" a drive for a second or two (as in "is this the right drive? Oops, no"), then having to wait hours for a full rebuild to complete. Even if nothing was changed on the volumes in that short time...

Scaling:

RAID-1 provides both protection against device failure and improves read I/O performance. Write performance is limited to the slowest device.

Mirroring can also be used as an operational technique to create full-copies of large/critical filesystems or databases on a live system with no downtime.
A 3rd or 4th volume is added to a mirror, synchronised, then after ensuring content-consistency "split-off" and used separately, typically as the base for a test/conversion environment or backups. Because the disk, or set of disks, can be loaded onto a truck/plane, very high effective bandwidths are possible for the price of a courier. It can be faster than volume 'snapshots' and over-the-wire replication, not to mention a fraction of the cost. Airlines are known to have moved data-centres between continents this way, whilst maintaining their 24x7 booking and flight systems.

This (block-exact + snapshot-and-journal) model can be scaled up to N-replicas by replicating either or both types of replica. For different operational requirements, different combinations would be preferred. All combinations have uses/advantages in specific instances and won't be enumerated.

Flash Memory: will filesystems become the CPU bottleneck?

2011-11-25T18:55:00.001-08:00

Flash memory with 50+k IO/sec may be too fast for Operating Systems (like Linux) with file-system operations consuming more CPU, even saturating it. They are on the way to becoming the system rate-limiting factor, otherwise known as a bottleneck.

What you can get away with at 20-100 IO/sec, i.e. consumes 1-2% of CPU, will be a CPU hog at 50k-500k IO/sec, a 5,000-50,000 times speed up.

The effect is the reverse of the way Amdahl speed-up is explained.

Amdahl throughput scaling is usually explained like this:

If your workload has 2 parts (A is single-threaded, B can be parallelised), when you decrease the time-taken for 'B' by adding parallel compute-units, the workload becomes dominated by the single-threaded part, 'A'. If you half the time it takes to run 'B', it doesn't halve the total run time. If 'A' and 'B' parts take equal time (4 units each, total 8), then a 2-times speed up of 'B' (4 units to 2) results in a 25% reduction in run-time (8 units to 6). Speeding 'B' up 4-times is a 37% reduction (8 to 5).
This creates a limit to the speed-up possible: If 'B' reduces to 0 units, it still takes the same time to run all the single-threaded parts, 'A'. (4 units here)

A corollary of this: the rate-of-improvement for each doubling of cost nears zero, if not well chosen.

The filesystem bottleneck is the reverse of this:

If your workload has an in-memory part (X) and wait-for-I/O part (W) both of which consume CPU, if you reduce the I/O wait to zero without reducing the CPU overhead of 'W', then the proportion of useful work done in 'X' decreases. In the limit, the system throughput is constrained by CPU expended on I/O overhead in 'W'.

The faster random I/O of Flash Memory will reduce application execution time, but at the expense of increasing % system CPU time. For a single process, the proportion and total CPU-effort of I/O overhead remains the same. For the whole system, more useful work is being done (it's noticeably "faster"), but because the CPU didn't get faster too, it needs to spend a lot more time on the FileSystem.

Jim Gary observed that:

CPU's are now mainly idle, i.e. waiting on RAM or I/O.
Level-1 cache is roughly the same speed as the CPU, everything else is much slower and must be waited for.
The time taken to scan a 20Tb disk using random I/O will be measured in days whilst a sequential scan ("streaming") will take hours.

Reading a "Linux Storage and Filesystem Workshop" (LSF) confrence report, I was struck by comments that:

linux file systems can consume large amount of CPU doing their work, not just fsck, but handling directories, file metadata, free block chains, inode block chains, block and file checksums, ...

There's a very simple demonstration of this: optical disk (CD-ROM or DVD) performance.

A block-by-block copy (dd) of a CD-ROM at "32x", or approx 3Mb/sec, will copy a full 650Mb in 3-4 minutes. Wikipedia states a 4.7Gb DVD takes 7 minutes (5.5Mb/sec) at "4x".
Mounting a CD or DVD then doing a file-by-file copy takes 5-10 times as long.
Installing or upgrading system software from the same CD/DVD is usually measured in hours.

The fastest way to upgrade software from CD/DVD is to copy an image with dd to hard-disk, then mount that image. The difference is the random I/O (seek) performance of the underlying media, not the FileSystem. [Haven't tested times or speedup with a fast Flash drive.]

This performance limit may have been something that the original Plan 9 writers knew and understood:

P9 didn't 'format' media for a filesystem: initialised a little and just started writing blocks.
didn't have fsck on client machines, only the fileserver.
the fileserver wrote to three levels of storage: RAM, disk, Optical disk.
RAM and disk were treated as cache, not permanent storage.
Files were pushed to Optical disk daily, creating a daily snapshot of the filesystem at the time. Like Apple's TimeMachine, files that hadn't changed were 'hard-linked' to the new directory tree.
The fileserver had operator activities like backup and restore. The design had no super-user with absolute access rights, so avoided many of the usual admin-related security issues.
Invented 'overlay mounts', managed at user not kernel level, to combine the disparate file-services available and allow users to define their own semantics.

Filesystems have never, until now, focussed on CPU performance, rather the opposite, they've traded CPU and RAM to reduce I/O latency, historically improving system throughput, sometimes by orders-of-magnitude.
Early examples were O/S buffers/caching (e.g. Unix) and the 'elevator algorithmn' to optimally reorder writes to match disk characteristics.

This 'burn the CPU' trade-off shows up with fsck as well. An older LSF piece suggested that fsck runs slowly because it doesn't do a single pass of the disk, effectively forced into near worst-case unoptimised random I/O.

On my little Mac Mini with a 300Gb disk, there's 225Gb used. Almost all of which, especially the system files, is unchanging. Most of the writing to disk is "append mode" - music, email, downloads - either blocks-to-a-file or file-to-directory. With transactional Databases, it's a different story.

The filesystem treats the whole disk as if every byte could be changed in the next second - and I pay a penalty for that in complexity and CPU cycles. Seeing my little Mac or an older Linux desktop do a filesystem check after a power fail is disheartening...

I suggest future O/S's will have to contend with:

Flash or SCM with close to RAM performance 'near' the CPU(s) (on the PCI bus, no SCSI controller)
near-infinite disk ("disk is tape", Jim Gray) that you'll only want to access as "seek and stream". It will also take "near infinite" time to scan with random I/O. [another Jim Gray observation]

And what are the new rules for filesystems in this environment?:

two sorts of filesystems that need to interwork:

read/write that needs fsck to properly recover after a failure and
append-only that doesn't need checking once "imaged", like ISO 9660 on optical disks.

''Flash" file-system organised to minimise CPU and RAM use. High performance/low CPU use will become as important as managing "wear" for very fast PCI Flash drives.
'hard disk' filesystem with on-the-fly append/change of media and 'clone disk' rather than 'repair f/sys'.
O/S must seamlessly/transparently:
1. present a single file-tree view of the two f/sys
2. like Virtual Memory, safely and silently migrate data/files from fast to slow storage.

I saw a quote from Ric Wheeler (EMC) from LSF-07 [my formatting]:

the basic contract that storage systems make with the user
is to guarantee that:
the complete set of data will be stored,
bytes are correct and
in order, and
raw capacity is utilized as completely as possible.

I disagree nowdays with his maximal space-utilisation clause for disk. When 2Tb costs $150 (7.5c/Gb) you can afford to waste a little here and there to optimise other factors.
With Flash Memory at $2-$5/Gb, you don't want to go wasting much of that space.

Jim Gray (again!) early on formulated "the 5-minute rule" which needs rethinking, especially with cheap Flash Memory redefining the underlying Engineering factors/ratios. These sorts of explicit engineering trade-off calculations have to be done for the current disruptive changes in technology.

Gray, J., Putzolu, G.R. 1987. The 5-minute rule for trading memory for disk accesses and the 10-byte rule for trading memory for CPU time. SIGMOD Record 16(3): 395-398.
Gray, J., Graefe, G. 1997. The five-minute rule ten years later, and other computer storage rules of thumb. SIGMOD Record 26(4): 63-68.

I think Wheeler's Storage Contract also needs to say something about 'preserving the data written', i.e. the durability and dependability of the storage system.
For how long? what what latency? How to express that? I don't know...

There is also a matter of "storage precision", already catered for with CD's and CD-ROM, Wikipedia states:

The difference between sector size and data content are the header information and the error-correcting codes, that are big for data (high precision required), small for VCD (standard for video) and none for audio. Note that all of these, including audio, still benefit from a lower layer of error correction at a sub-sector level.

Again, I don't know how to express this, implement it nor a good user-interface. What is very clear to me is:

Not all data needs to come back bit-perfect, though it is always nice when it does.
Some data we would rather not have, in whole or part, than come back corrupted.
There are many data-dependent ways to achieve Good Enough replay when that's acceptable.

First, the aspects of Durability and Precision need to be defined and refined, then a common File-system interface created and finally, like Virtual Memory, automated and executed without thought or human interaction.

This piece describes FileSystems, not Tabular Databases nor other types of Datastore.
The same disruptive technology problems need to be addressed within these realms.
Of course, it'd be nicer/easier if other Datastores were able to efficiently map to a common interface or representation shared with FileSystems and all the work/decisions happened in Just One Place.

Will that happen in my lifetime? Hmmmm....

Building a RAID disk array circa 1988

2011-11-20T21:49:00.001-08:00

In "A Case for Redundant Arrays of Inexpensive Disks (RAID)" [1988], Patterson et al of University of California Berkeley started a revolution in Disk Storage still going today. Within 3 years, IBM had released the last of their monolithic disk drives, the 3390 Model K, with the line being discontinued and replaced with IBM's own Disk Array.

The 1988 paper has a number of tables where it compares Cost/Capacity, Cost/Performance and Reliability/Performance of IBM Large Drives, large SCSI drives and 3½in SCSI drives.

The prices ($$/MB) cited for the IBM 3380 drives are hard to reconcile with published prices:
press releases in Computerworld and IBM Archives for 3380 disks (7.5Gb, 14" platter, 6.5kW) and their controllers suggest $63+/Mb for 'SLED' (Single Large Expensive Disk) rather than the
"$18-10" cited in the Patterson paper.

The prices for the 600MB Fujitsu M2316A ("super eagle") [$20-$17] and 100Mb Conner Peripherals CP-3100 [$10-$7] are in-line with historical prices found on the web.

The last table in the 1988 paper lists projected prices for different proposed RAID configurations:

$11-$8 for 100 * CP-3100 [10,000MB] and
$11-$8 for 10 * CP-3100 [1,000MB]

There are no design details given.

1994, Chen et al in "RAID: High-Performance,Reliable Secondary Storage" use two widely sold commercial system ascase studies:

NCR 6298 and
StorageTek's Iceberg 9200 Disk Array

The (low-end) NCR device was more what we'd call a 'hardware RAIDcontroller' now, ranging from 5 to 25 disks. Pricing $22-102,000.It provided a SCSI interface and didn't buffer. A system diagram was included in the paper.

The StorageTek's Iceberg was high-end device meant for connection to IBM mainframes. Advertised as starting at 100GB (32 drives) for $1.3M, up to 400Gb for $3.6M, It provided multiple (4-16) IBM ESCON 'channels'.

For the NCR, from InfoWorld 1 Oct 1990, p 19 in Google Books

min config: 5 * 3½in drives, 420MB each.
$22,000 for 1.05Gb storage
Add 20*420Mb to 8.4Gb list $102,000. March 1991.
$4,000/drive + $2,000 controller.
NCR-designed controller chip + SCSI chip
4 RAID implementations: RAID 0,1,3,5.

The StorTek Iceberg was released in late 1992 with projected shipments of 1,000 units in 1993. It was aimed at replacing IBM 'DASD' (Direct Access Storage Device): exactly the comparison made in the 1988 RAID paper.

The IBM-compatible DASD, which resulted from an investment of $145 million and is technically styled the 9200 disk array subsystem, is priced at $1.3 million for a minimum configuration with 64MB of cache and 100GB of storage capacity provided by 32 Hewlett-Packard 5.25-inch drives.

A maximum configuration, with 512MB of cache and 400GB of storage capacity from 128 disks, will run more than $3.6 million. Those capacity figures include data compression and compaction, which can as much as triple the storage level beyond the actual physical capacity of the subsystem.

Elsewhere in the article more 'flexible pricing' (20-25% discount) is suggested:

with most of the units falling into the 100- to 200GB capacity range, averaging slightly in excess of $1 million apiece.

Whilst no technical reference is easily accessible on-line, more technical details are mentioned in the press release on the 1994 upgrade, the 9220. Chen et al [1994] claim "100,000 lines of code" were written.

More clues come from an feature, "Make Room for DASD" by Kathleen Melymuka (p62) of CIO magazine, 1st June 1992 [accessed via Google Books, no direct link]:

5¼in Hewlett-Packard drives were used. [model number & size not stated]
The "100Gb" may include compaction and compression. [300% claimed later]
(32 drives) "arranged in dual redundancy array of 16 disks each (15+1 spare)
RAID-6 ?
"from the cache, 14 pathways transfer data to and from the disk arrays, and each path can sustain a 5Mbps transfer rate"

The Chen et al paper (pg 175 of CACM,, Vol 26, No 2) gives this information on the Iceberg/9200:

it "implements an extended RAID level-5 and level-6 disk array"
- 16 disks per 'array', 13 usable, 2 Parity (P+Q), 1 hot spare
- "data, parity and Reed-Solomon coding are striped across the 15 active drives of an array"
Maximum of 2 Penguin 'controllers' per unit.
Each controller is an 8-way processor, handling up to 4 'arrays' each, or 150Gb (raw).

Implying 2.3-2.5Gb per drive

The C3010, seemingly the largest HP disk in 1992, was 2.47Gb unformatted and 2Gb formatted (512by sectors), [notionally 595by unformatted sectors]
The C3010 specs included:

MTBF: 300,000 hrs
Unrecoverable Error Rate (UER): 1 in 10^14 bits transferred
11.5 msec avg seek, (5.5msec rotational latency, 5400RPM)
256Kb cache, 1:1 sector interleave, 1,7 RLL encoding, Reed-Solomon ECC.
max 43W 'fast-wide' option, 36W running.

runs up to 8 'channel programs' and independently transfer on 4 channels (to mainframe).
manages a 64-512Mb battery-backed cache (shared or per controller not stated)
implements on-the-fly compression, cites maximum doubling capacity.

and dynamic mapping necessary CKD (count, key, data) for variable-sized IBM blocks onto the fixed blocks internally.
a extra (local?) 8Mb of non-volatile memory is used to store these tables/maps.

Uses a "Log-Structured File System" so blocks are not written back to the same place on the disk.
Not stated if the SCSI buses are one-per-arry or 'orthogonal'. i.e. Redundancy groups are made up from one disk per 'array'.

Elsewhere, Katz, one of the authors, uses a diagram of a generic RAID system not subject to any "Single Point of Failure":

with dual-controllers and dual channel interfaces.

Controllers cross-connected to each interface.

dual-ported disks connected to both controllers.

This halves the number of unique drives in a system, or doubles the number of SCSI buses/HBA's, but copes with the loss of a controller.

Implying any battery-backed cache (not in diagram) would need to be shared between controllers.

From this, a reasonable guess at aspects of the design is:

HP C3010 drives were used, 2Gb formatted. [Unable to find list prices on-line]

These drives were SCSI-2 (up to 16 devices per bus)
available as single-ended (5MB/sec) or 'fast' differential (10MB/sec) or 'fast-wide' (16-bit, 20MB/sec). At least 'fast' differential, probably 'fast-wide'.

"14 pathways" could mean 14 SCSI buses, one per line of disks, but it doesn't match with the claimed 16 disks per array.

16 SCSI buses with 16 HBA's per controller matches the design.
Allows the claimed 4 arrays of 16 drives per controller (64) and 128 max.
SCSI-2 'fast-wide' allows 16 devices total on a bus, including host initiators. This implies that either more than 16 SCSI

5Mbps transfer rate probably means synchronous SCSI-1 rates of 5MB/sec or asynchronous SCSI-2 'fast-wide'.

It cannot mean the 33.5-42Mbps burst rate of the C3010.
The C3010 achieved transfer rates of 2.5MB/sec asynchronously in 'fast' mode, or 5MB/sec in 'fast-wide' mode.
Only the 'fast-wide' SCSI-2 option supported dual-porting.
The C3010 technical reference states that both powered-on and powered-off disks could be added/removed to/from a SCSI-2 bus without causing a 'glitch'. Hot swapping (failed) drives should've been possible.

RAID-5/6 groups of 15 with 2 parity/check disk overhead, 26Gb usable per array, max 208Gb.

RAID redundancy groups are implied to be per (16-disk) 'array' plus one hot-spare .
But 'orthogonal' wiring of redundancy groups was probably used, so how many SCSI buses were needed per controller, in both 1 and 2-Controller configurations?
No two drives in a redundancy group should be connected via the same SCSI HBA, SCSI bus, power-group or cooling-group.
This allows live hardware maintenance or single failures.
How were the SCSI buses organised?
With only 14 devices total per SCSI-2 bus, a max of 7 disks per shared controller was possible.
The only possibly configurations that allow in-place upgrades are: 4 or 6 drives per bus.
The 4-drives/bus resolves to "each drive in an array on a separate bus".
For manufacturing reasons, components need standard configurations.
It's reasonable to assume that all disk arrays would be wired identically, internally and with common mass terminations on either side, even to the extent of different connectors (male/female) per side.
This allows simple assembly and expansion, and trivially correct installation of SCSI terminators on a 1-Controller system.
Only separate-bus-per-drive-in-array (max 4-drives/bus), meets these constraints.
SCSI required a 'terminator' at each end of the bus. Typically one end was the host initiator. For dual-host buses, one at each host HBA works.
Max 4-drives per bus results in 16 SCSI buses per Controller (64-disks per side).
'fast-wide' SCSI-2 must have been used to support dual-porting.
The 16 SCSI buses, one per slot in the disk arrays, would've continued across all arrays in a fully populated system.
In a minimum system, 32 drives, would've been only 2 disks per SCSI bus.

1 or 2 controllers with a shared 64M-512M cache and 8Mb for dynamic mapping.

This would be a high-performance and highly reliable design with a believable $1-2M price for 64 drives (200Gb notional, 150Gb raw):

1 Controllers
128Mb RAM
8 ESCON channels
16 SCSI controllers
64 * 2Gb drives as 4*16 arrays, 60 drives active, 52 drive-equivalents after RAID-6 parity.
cabinets, packaging, fans and power-supplies

From the two price-points, can we tease out a little more of the costs [no allowance for ESCON channel cards]:

1 Controller + 32 disks + 64M cache = $1.3M
2 Controllers + 128 disks + 512M cache = $3.6M

As a first approximation, assume that 512M RAM costs half as much as 2 Controllers for a 'balanced' system. Giving us a solvable set of simultaneous equations:

1.0625 Controllers + 32 disks = $1.3M
2.5 Controllers + 128 disks = $3.6M

roughly:

$900,000 / Controller [probably $50,000 high]
$70,000 / 64M cache [probably $50,000 low]
$330,000 / 32 disks ($10k/drive, or $5/MB)

High-end multi-processor VAX system pricing at the time is in-line with this $900k estimate, but more likely an OEM'd RISC processor (MIPS or SPARC?) was used.
This was a specialist, low-volume device: expected 1st year sales volume was ~1,000.
In 1994, they'd reported sales of 775 units when the upgrade (9220) was released.

Another contemporary computer press article cites the StorageTek Array costing $10/Mb compared to $15/MB for IBM DASD. 100Gb @ $10/Mb is $1M, so congruent with the claims above.

How do the real-world products in 1992 compare to the 1988 RAID estimates of Patterson et al?

StorageTek Iceberg: $10/Mb vs $11-$8 projected.

This was achieved using 2Gb 5¼in drives not the 100Mb 3½in drives modelled
HP sold a 1Gb 3½in SCSI-2 drive (C2247) in 1992. This may have formed the basis of the upgrade 9220 ~two years later.

Using the actual, not notional, supplied capacity (243Gb) the Iceberg cost $15/Mb.
The $15/Mb for IBM DASD compares well to the $18-$10 cited in 1988.

But IBM, in those intervening 5 years, had halved the per-Mb price of their drives once or twice. The 1988 "list price" from the archives of ~$60/Mb are reasonable.

In late 1992, 122Mb Conner CP-30104 were advertised for $400, or $3.25/Mb.
These were IDE drives, though a 120Mb version of the SCSI CP-3100 was sold, price unknown.

The 8.4Gb 25-drive NCR 6298 gave $12.15/Mb, again close to the target zone.
From the Dahlin list, 'street prices' for 420Mb drives at the time, were $1600 for Seagate ST-1480A and $1300 for 425Mb Quantum or $3.05-$3.75/Mb.

The price can't be directly compared to either IBM DASD or StorageTek's Iceberg, because the NCR 6298 only provided a SCSI interface, not an IBM 'channel' interface.

The raw storage costs of the StorageTek Iceberg and NCR are roughly 2.5:1.
Not unexpected due to the extra complexity, size and functionality of the Iceberg.

High Density Racking for 2.5" disks

2011-11-11T17:16:00.001-08:00

Datacentre space is expensive: one 2010 artice puts construction at $1200/sq ft and rental at $600/pa/sq ft for an approx design heat load of 10-50W/sq ft. Google is reputed to be spending $3,000/sq ft building datacentre with many times this heat load.

There are 3 different measures of area:

"gross footprint". The room plus all ancillary equipment and spaces.
"room". The total area of the room. Each rack, with aisles & work-space uses 16-20 sq ft.
"equipment footprint". The floor area directly under computing equipment. 24"x40", ~7sq ft.

Presumably the $600/pa rental cost is for "equipment footprint".

The 2½ inch drive form-factor is (100.5 mm x 69.85 mm x 7-15 mm).
Enterprise drives are typically 12.5 or 15mm thick. Vertically stacked, 20-22 removable 2½ inch drives can be fitted across a rack, taking a 2RU space, with around 15mm of 89mm unused.

2½ inch drive don't fit well in standard 19" server racks (17" wide, by 900-1000mm deep, 1RU = 1.75" tall), especially if you want equal access (eg. from the front) to all drives without disturbing any drives. Communications racks are typically 600mm deep, but not used for equipment in datacentres.

With cabling, electronics and power distribution, a depth of 150mm (6") should be sufficient to house 2½ inch drives. Power supply units take additional space.

Usable space inside a server rack, 17" wide and 1000mm deep, would leave 850mm wasted.
Mounting front and back, would still leave 700mm wasted, but create significant heat removal problems, especially in a "hot aisle" facility.

The problems reduces to physical arrangements that maximise exposed area (long and thin rectangles vs the close-to-square 19" Rack, if dual sided, with a chimney) or maximise surface area and minimise floor space - a circle or cylinder.

The "long-thin rectangle" arrangement was popular in mainframe days, often as an "X" or a central-spine with many "wings". It assumes that no other equipment will be sited within the working clearances needed to open doors and remove equipment.

A cylinder, or pipe, must contain a central chimney to remove waste heat. There is also a requirement to plan cabling for power and data. Power supplies can be in the plinth as the central cooling void can't be blocked mid-height and extraction fan(s) need to be mounted at the top.

For a 17" diameter pipe, 70-72 disks can be mounted around the circumference allowing 75 mm height per row, 20-24 rows high allowing for a plinth and normal heights. This leaves a central void of around 7" to handle the ~8kW of power of ~1550 drives.

A 19" pipe would allow 75-80 disks per row and a 9" central void to handle ~9.5kW of ~2000 drives.
Fully populated unit weight would be 350-450kg.

Perhaps one in 8 disks could be removed allowing a cable tray, a not unreasonable loss of space.

These "pipes" could be sited in normal racks either at the end of row requiring one free rack-space beside them, or in a row, taking 3 rack-spaces.

As a series of freestanding units, they could be mounted in a hexagonal pattern (the closest-packing arrangement for circles) with minimum OH&S clearances around them, which may be 600-750mm.

This provides 4-5 times the density of drives over the current usual 22 shelves of 24 drives (480) per rack, with better heat extraction. At $4-5,000/pa rental per rack-space (or ~$10/drive), it's a useful saving.

With current drive sizes of 600-1000Gb/drive, most organisations would get by with one unit of 1-2Pb.

Update: A semi-circular variant 40inx40in for installation in 3 Rack-widths might work as well. Requires a door to create the chimney space/central void - and it could vent directly into a "hot aisle".

Papers:
2008: "Cost Model: Dollars per kW plus Dollars per Square Foot of Computer Floor"

2007: "Data center TCO; a comparison of high-density and low-density spaces"

2006: "Total Cost of Ownership Analysis for Data Center Projects"

2006: "Dollars per kW plus Dollars per Square Foot Are a Better Data Center Cost Model than Dollars per Square Foot Alone"

Questions about SSD / Flash Memory

2011-11-10T20:13:00.001-08:00

Seagate, in 2010, quote their SSD UER specs as:
Nonrecoverable read errors, max: 1 LBA per 10^16 bits read
where a Logical Block Address (LBA) is 512 bytes. Usually called a 'sector'.

But we know that Flash memory is organised as 64Kb blocks (min read/write unit).
Are Seagate saying that errors will be random localised cells, not "whole block at a time"?
Of course, the memory controller does Error Correction to pick up the odd dropped bits.
Current RAID schemes are antagonistic to Flash Memory:
The essential problem with NAND Flash (EEPROM) Memory is that it suffers "wear" - after a number of writes and erasures, individual cells (with MLC, cell =/= bit) are no longer programmable. A secondary problem is "Data Retention". With the device powered down, Seagate quote "Data Retention" of 1 year.

Because Flash Memory wears with writes, batches of components will likely have very similar wear characteristics and if multiple SSD's are mirrored/RAIDed in a system they will most likely be from the same batch, evenly spread RAID writes (RAID-5 writes two physical blocks per logical block) will cause a set of SSD's to suffer correlated wear failures. This is not unlike the management of piston engines in multi-engined aircraft: avoid needing to replace more than one at a time. Faults, Failures and Repair/Install Errors often show up in the first trip. Replacing all engines together maximises the risk of total engine failure.

Not only is this "not ideal", it is exactly worst case for current RAID.
A new Data Protection Scheme is required for SSD's.

Update 23-Dec-2011. Jim Handy in "The SSD Guy" blog [Nov-17, 2011] discusses SSD's and RAID volumes:
So far this all sounds good, but in a RAID configuration this can cause trouble. Here’s why.

RAID is based upon the notion that HDDs fail randomly. When an HDD fails, a technician replaces the failed drive and issues a rebuild command. It is enormously unlikely that another disk will fail during a rebuild. If SSDs replace the HDDs in this system, and if the SSDs all come from the same vendor and from the same manufacturing lot, and if they are all exposed to similar workloads, then they can all be expected to fail at around the same time.

This implies that a RAID that has suffered an SSD failure is very likely to see another failure during a rebuild – a scenario that causes the entire RAID to collapse.

What are the drivers for the on-going reduction in prices of Flash Memory?
Volume? Design? Fabrication method (line width, "high-K", ...)? Chip Density?

The price of SSD's has been roughly halving every 12-18 months for near on a decade, but why?
Understanding the "why" is necessary to be forewarned of any change to the pattern.
How differently are DRAM and EEPROM fabricated?
Why is there about a 5-fold price difference between them?
```
Prices (Kingston from same store, http://msy.com.au, November 2011):
DDR3 6Gb 1333Mhz $41 $7/Gb
SSD 64Gb  $103    $1.50/Gb
SSD 128Gb  $187 $1.25/Gb
```
It would be nice to know if there was a structural difference or not for designing "balanced systems", or integrating Flash Memory directly into the memory hierarchy, not as a pretend block device.
Main CPU's and O/S's can outperform any embedded hardware controller.
Why do the PCI SSD's not just present a "big blocks of memory" interface, but insist on running their own controllers?
Hybrid SSD-HDD RAID.
For Linux particularly, is it possible to create multiple partitions per HDD in a set, then use one HDD-partition to mirror a complete SSD. The remaining partitions can be setup as RAID volumes in the normal way.
The read/write characteristics of SSD and HDD are complementary: SSD is blindly fast for random IO/sec, while HDD's currently stream reads/writes at higher sustained writes.
Specifically, given 1*128Gb SSD and 3*1Tb HDD, create a 128Gb partition on all HDD's. Then mirror (RAID-1), the SSD and one or more of the HDD's (RAID-1 isn't restricted to two copies, but can have as many replicas as desired to increase IO performance or resilience/Data Integrity). Remaining 128Gb HDD partitions can be stripped, mirrored or RAIDed amongst themselves or to other drives. The remaining HDD space can be partitioned and RAIDed to suit demand.

Does it make sense, both performance- and reliability-wise, to mirror SSD and HDD?
Does the combination yield the best, or worse, of both worlds?
Is the cost/complexity and extra maintenance worth it?

Enterprise Drives: Moving to 2.5" form factor

2011-11-04T00:47:00.000-07:00

Update [23-Dec-2011]: IDC, in 2009, discussed in a HP report, the migration to 2.5 inch drives in Enterprise Storage. Started in 2004, projected to be complete in 2011.

Elsewhere, a summary of IDC's 2009 Worldwide HDD shipments and revenue:

The transition from 3.5in. to 2.5in. performance-optimized form factor HDDs will be complete by 2012.
Growing interest in new storage delivery models such as storage as a service, or storage in the cloud is likely to put greater storage capacity growth demands on Internet datacenters.
The price per gigabyte of performance-optimized HDD storage will continue to decline at a rate of approximately 25% to 30% per year.

In their seminal paper, "A Case for Redundant Arrays of Inexpensive Disks (RAID)", [1988] Patterson, Gibson and Katz discuss a number of disk performance metrics.

In their first figure they compare drive types across a number of metrics, looking for a top-line "Price/Performance" comparison, given that "reliability" can be maintained. Factors are:

performance (bandwidth, latency, IO/sec)
unit size or volume
power/volume
Price/MB
MB/volume

The authors say:

... so we concentrate on the price-performance and reliability. Our reasoning is that (if) there are no advantages in price-performance or terrible disadvantages in reliability, then there is no need to explore further.

and

(RAID) offers an attractive alternative to SLED (Single Large Expensive Disk), promising improvements of an order of magnitude in performance, reliability, power consumption, and scalability.

Interestingly, weight has been omitted as a metric. Even the 8 in Fujitsu Eagle (6RU high and 600+mm deep) in 1981 stored 474.2MB, and at 65kg was heavy enough that planning for unpacking, installation and floor loading was necessary. Even now, the weight of "fully loaded" Disk Arrays with 3½ in HDD's has to be taken into account.

Since the paper was written, Disk Arrays have moved from using disk form-factors of 8 in, 5¼ in, 3½ in and are now flirting with 2½ in drives. In each of these transitions, there was no sudden change, rather a gradual phase-out as vendors introduced new models at premium prices, until the last old-format model was discontinued and customer-support gradually phased out. The whole process taking up to 10 years, but as with any adoption of new technology, the "majority" users (not early- or late-adopters) change in a 3-5 year window, depending on their "hardware refresh" cycles.

The driving forces for consumers have been Price and Size and the related Storage Density (Gb/volume) and Specific Power (W/Gb) which are Financial and Operational factors, not "technical" nor "performance".

For vendors, the technical driver is the on-going improvement in disk-drive capacity/recording densities, interface changes (SCSI to SAS) and improvements in CPU/RAM price/performance/capacity. Plus the inevitable commercial driver of "planned obsolescence", but in light of historical improvements in Silicon technologies (Moore's Law etc), this effect hasn't been significant historically.

Others have written serious technical comparisons of 3½ in vs 2½ in drives for Enterprise use, which are fully worth reading.

Here I've been attempting to describe a bigger arc:
the commercial and technical forces driving these on-going changes.

Understanding the forces may help inform us of:

"limits to growth" - Will we end up with 1" drives or stop at 2½ in drives?
"limits of technology" - The impact of hitting a recording density limit.

There is a particular problem with 2½ in drives, wasted rack-space, or Drive Packing Density [DPD]:

The 8" Fujitsu Eagle put one drive in 6RU (10.5" platter). [DPD = 0.16/RU]
5¼ in drives fitted 3 abreast in 2RU (5.25" platter). [DPD = 1.5/RU]
Full Height Drive Bay: 3¼″ high by 5¾″ wide, and up to 8″ deep (83×146×200 mm)
3½ drives fit 14 drives per "shelf" of 3RU (3.5" platter) [DPD = 5/RU]
or 24 drives in 4RU [DPD = 6/RU]
2½ in drives allow around 2-3 times the density, 24 drives in 2RU [DPD = 12/RU]

Currently "disk shelves" for 3½ drives are around the same 600mm taken by 8" and 5¼" drives, although the raw drives are only 150mm deep, leaving significant empty space in the back of the shelf..

2½ in drives are only 100mm deep, meaning they could be stacked 4-6 deep in a 2RU unit.
At 5W/drive, could those 100+ drives be kept sufficiently cool with standard front-to-back airflow (max 55°C operating)?
At 220-250gm per drive, plus fittings and support electronics/cabling, total weight would approach 50kg/shelf.

In producing "Design Rules" for the latest generation of disk drives, 2½ in, how can Drive Packing Density be maximised without compromising access and cooling? Tucking drives away inside a 19" rack works very well until one needs changing. Moving operating drives is not good practice, so slide-out trays aren't a good choice, especially as they will also block access ways and create a potential tipping hazard. E.g. early IBM 2311 drives were known for being unstable with drawers extended .

Packaging could be the Packaging Density limit for the 2½ in generation of drives.
An academic treatment of the topic would be inadequate if based solely on the raw drive volumes, 377 cm³ vs 105 cm³ [1 cubic-centimetre is also 1 millilitre]. Physical factors dominate choice of mounting geometry, meaning side-to-side cooling or an internal vertical chimney would be needed with simple front-and-back lineal packing.

Moving to the next generation of drives, 1.8 in (54 mm × 8 mm × 71 mm vs 70 mm × 7–15 mm × 100 mm for "2½ in"), will only exacerbate these wasted space/cooling problems. As yet, there is no commodity production/use of 1.8 in drives, it's unlikely that Enterprise HDD's will progress there anytime soon.

The prime candidates for 1.8 in form-factor are SDD's for installation in servers or laptops/netbooks.

Types of 'Flash Memory'

2011-11-03T20:03:00.000-07:00

"Flash Memory", or EEPROM (Electrically Erasable Programmable Read Only Memory), is at the heart of much of the "silicon revolution" of the last 10 years.

How is it packaged and made available to consumers or system builders/designers?

Mobile appliances are redefining Communications and The Internet, precisely because of the low-power, high-capacity and longevity - and affordable price - of modern Flash Memory.

There are many different Flash Memory configurations: NAND, NOR, SLC, MLC, ...
This piece isn't about those details/differences but the in how they are packaged and organised.
What technology does what/has what characteristics is out there on the InterWebs.

Most Flash Memory chips are assembled into different packaging for different uses at different price points. Prices are "Retail Price (tax paid)" from a random survey of Internet sites:

Many appliances use direct-soldered Flash Memory are their primary or sole Persistent Datastore.
The genesis of this was upgradeable BIOS firmware. Late 1990's?
Per-Gb pricing not published: approx. derivable from model price differences.
Commodity 'cards' used in cameras, phones and more: SD-card and friends.
Mini-CD and Micro-SD cards are special cases and attract a price premium.
Some 'high-performance' variants for cameras.
A$2-3/Gb
USB 'flash' or 'thumb drives':
A$2-3/Gb.
High-end camera memory cards: Compact Flash (CF). The oldest mass-use format?
IDE/ATA compatible interface. Disk drive replacement for embedded systems.
Fastest cards are 100MB/sec (0.8Gbps). Max is UDMA ATA, 133MB/sec.
Unpublished Bit Error Rate, Write Endurance, MTBF, Power Cycles, IO/sec.
A$5-$30/Gb
SATA 2.5" SSD (Solid State Drives). Mainly 3Gbps and some 6Gbps interfaces.
MTBF: 1-2M hours,
Service Life: 3-5 years at 25% capacity written/day.
IO/sec: 5,000 - 50,000 IO/sec [max seen: 85k IO/sec]
BER: "1 sector per 10^15-16 bits read"
sustained read/write speed: 70-400MB/sec . (read often slowest)

Power: 150-2000W active, 75-500mW idle
32Gb - 256GB @ A$1.50-$2.50/Gb.
SATA 1.8" SSD. Internal configuration of some 2.5" SSD's.
Not yet widely available.
SAS (Serial Attached SCSI) 2.5" drives.
not researched. high-performance, premium pricing.
PCI "SSD". PCI card presenting as a Disk Device.
Multiple vendors, usual prices ~A$3-4/Gb. Sizes 256Gb - 1Tb.
"Fusion-io" specs quoted by Dell.Est A$20-25/Gb. [vs ~$5/Gb direct]
640GB (Duo)
NAND Type: MLC (Multi Level Cell)
Read Bandwidth (64kB): 1.0 GB/s
Write Bandwidth (64kB): 1.5 GB/s
Read IOPS (512 Byte): 196,000
Write IOPS (512 Byte): 285,000
Mixed IOPS (75/25 r/w): 138,000
Read Latency (512 Byte): 29 μs
Write Latency (512 Byte): 2 μs
Bus Interface: PCI-Express x4 / x8 or PCI Express 2.0 x4
Mini-PCIe cards, Intel: 40 and 80Gb. A$3/Gb
Intel SSD 310 Series 80GB mini PCIe

* Capacity: 80 GB,
* Components: Intel NAND Flash Memory Multi-Level Cell (MLC) Technology
* Form Factor, mini PCIe, mSATA, Interface,
* Sequential Read - 200 MB/s, Sequential Write - 70 MB/s,
* Latency - Read - 35 µs, Latency - Write - 65 µs,
* Lithography - 34 nm
* Mean Time Between Failures (MTBF) - 1,200,000 Hours

Roughly, prices increase with size and performance.
The highest density chips, or leading edge technology, cost a premium.
As do high-performance or "specialist" formats:

micro-SD cards
CF cards
SAS and PCI SSD's.

When is an SATA drive not a drive, when it's compact flash

2011-11-02T20:08:00.000-07:00

The CompactFlash Association has released the new generation of the CF specification, CFast™ [from a piece on 100MB/sec CF cards]

10-15 years ago, CF cards were used almost exclusively in all Digital cameras, now they are used only in "high-end" Digital SLR's (DSLR), presumably because of cost/availability compared to alternatives like SDHC cards.

The UDMA based CF standard allows up to 133MB/sec transfer rates.
The new SATA based standard, CFast, allows 3Gbps (~300MB/sec) transfer rates.

In another context, you'd call this an SSD (Solid State Disk), even a "SATA drive".
There are two problems:

common cameras don't currently support CFast™, the SATA based standard, and
'fast' CF cards are slower than most SSD's and attract a price premium of 5-10 times.

I'm not sure what decision camera manufacturers will make for their Next Generation high-end storage interface, they have 3 obvious directions:

CF-card format (43×36×5 mm), SATA interface, CFast™
1.8inch or 2.5inch SSD, SATA interface
34mm ExpressCard. PCIe interface.

and the less obvious: somehow adapt commodity SDHC cards to high-performance.
Perhaps in a 'pack' operating like RAID.

The "Next Generation Interface" is a $64 billion question for high-end camera manufactures, the choice will stay with the industry for a very long time, negatively affecting sales if made poorly.

Manufacturers are much better off selecting the same standard (common parts, lower prices for everyone), but need to balance the convenience of "special form factors" with cost. Whilst professional photographers will pay "whatever's needed" for specialist products, their budgets aren't infinite and excessive prices restrict sales to high-end amateurs.

Perhaps the best we'll see is a transition period of dual- or triple-card cameras (SDHC, CF-card and CFast™), with the possibility of an e-SATA connector for "direct drive connection".

Update 04-Nov-2011:
Here's a good overview from the Sandisk site of form-factor candidates to replace the CF card form-factor of (43×36×5 mm):

SanDisk® Solid State Drives for the Client
"A variety of form factors, supporting multiple OEM design needs."

SanDisk® Solid State Drives for the Client
Product Name	Interface	Form Factor	Measurements
SanDisk U100 SSD	SATA	2.5"	100.5 mm x 69.85 mm x 7 mm std allows 9.5mm, 12.5mm, 15mm
SanDisk U100 SSD	SATA	Half-Slim SATA	54.00 mm x 39.00 mm x 3.08 mm (8-64GB), x 2.88 mm (128-256GB), Connector 4.00 mm
SanDisk U100 SSD	SATA	mSATA	30.00 mm x 50.95 mm x 3.4 mm (8-64GB), x 3.2 mm (128-256GB)
SanDisk U100 SSD	SATA	mSATA mini	26.80 mm x 30 mm x 2.2 mm (8GB), x 3.2mm (16-128GB)
SanDisk iSSD(TM)	SATA	SATA uSSD	16 mm x 20 mm x 1.20 mm (8 GB-32 GB) x 1.40 mm (64 GB) x 1.85 mm (128 GB)
Standard 1.8 in disk	SATA	1.8"	54 mm x 71 mm x 8 mm
Express Card	PCIe	Express Card	34mm x 75 mm x 5mm 54mm x 75 mm x 5mm
Compact Flash	ATA	CF-II	36mm x 43 mm x 5mm

Flash Memory vs 15,000 RPM drives

2011-11-01T01:08:00.000-07:00

Some I.T. technologies have a limited life or "use window". For example:

In 2001, the largest Compact Flash (CF) card for Digital cameras wasn't flash memory, but a hard disk, the 1Gb IBM "microdrive" ($500 claimed price). After Hitachi acquired the business, they produced 4Gb and 6Gb drives, apparently for sale as late as 2006, with the 4Gb variant used in the now discontinued Apple iPod mini.

Around 2006, the 1" microdrive hard disks were out-competed by flash memory and became yet another defunct technology.

Flash memory storage, either SSD or PCI 'drives', for servers now cost A$2-3/Gb for SATA SSD and $3-4/Gb for PCIe cards [A$3/Gb for Intel mini-PCIe cards].

Currently, Seagate 15k 'Cheetah' drives sell for around $2/Gb, but their 2msec (0.5k IO/sec) performance is no match for the 5KIO/sec of low-end SSD's or the 100-250KIO/sec of PCI flash.

10,000 RPM 'Enterprise' drives cost less, around $1.50/Gb, whilst 1Tb 7200 RPM (Enterprise) drives come in at $0.25/Gb.

The only usual criteria 15,000 RPM drives beat other media on is single-drive transfer rate^*.
Which in an 'Enterprise' RAID environment is not an advantage unless a) you're not paying full price or b) you have very special requirements or constraints, such as limited space.

I'm wondering if 2010 was the year that 15,000 RPM Enterprise drives joined microdrives in the backlot of obsolete technologies - replaced in part by the same thing, Flash Memory.

Part of the problem stems from the triple whammy for any specialist technology/device:

Overheads are maximised: By definition 'extreme' technologies are beyond "cutting-edge", more "bleeding-edge", meaning research costs are very high.
Per Unit fixed costs are maximised: Sales volumes of specialist or extreme devices are necessarily low, requiring those high research costs to be amortised over just a few sales.
If the technology ever becomes mainstream, it is no longer 'specialist' and research costs are amortised over very large numbers of parts.
Highest Margin per Unit: If you want your vendor to stay in business, they have to make suitable profits, both to encourage 'capital' to remain invested in the business and have enough surplus available to fund the next, more expensive, round of research. Profitable businesses can be Low-Volume/High Margin or High-Volume/Low Margin (or Mid/Mid).

Specialised or 'extreme performance' products aren't proportionately more expensive, they are necessarily radically more expensive, compounding the problem. When simple alternatives are available to use commodity/mainstream devices (defined as 'least cost per-perf. unit' or highest volume used), then they are adopted, all other things being equal.

^*There are many characteristics of magnetic media that are desirable or necessary for some situations, such as "infinite write cycles". These may be discussed in detail elsewhere.

RAID, Backups and Recovery

2011-10-31T15:36:00.000-07:00

There's an ironclad Law of SysAdmin:

"Nobody ever asks for backups to be done, only ever 'restores'"

Discussing RAID and Reliable Persistent Storage cannot be done without reference to the larger context: Whole System Data Protection and Operations.

"Backups" need more thought and effort than a 1980-style dump-disk-to-tape. Conversely, as various politicians & business criminals have found to their cost with "stored emails", permanent storage of everything is not ideal either, even if you have the systems, space and budget for media and operations.

"Backups" are more than just 'a second copy of your precious data' (more fully, 'of some indeterminate age'), which RAID alone partially provide.

A rough taxonomy of 'backups' are:

Operating System (O/S) software, modulo installed packages.
O/S configuration.
O/S logs.
Middleware (Database, web server, ..) software.
Middleware configuration
Middleware data.
Middleware logs.
Application {Software, Configuration, Data, Logs}

There are, minimally, two distinct type of data:

Transactional, eg. Database, comprised of a transaction log and the database.
Files.

"Snapshots" provide perfect point-in-time recovery for transactional data (the "roll forward log" on one snapshot, older database snapshot on another).
Versioning systems provide perfect event-in-time recovery for textual and other file data.
Whilst filesystems can be journalled, the potential to capture-for-replay every filesystem transaction isn't normally available due to volumes and complexity.

Since the destruction of the World Trade Centre buildings in 2001, every organisation with critical I.T. systems and a need for strong Data Protection/Reliable Persistent Storage understand that a minimum requirement for 'second copy data' is a Remote Duplicate Copy. This implies at least a secure second facility.

Two critical design and operational parameters are:

"Delta-T", the difference in timestreams between the live system and Duplicate Copies.
"Sufficient Distance" for the second facility to not be affected by designed-for "events".

"Disaster Recovery" and its Planning is all about managing these and related factors, along with time-to-restoration and time-to-recovery.

Can "Zero Delta-T" and "Zero TTR" systems be built at all? For all datastore sizes? Economically?
Is there a relationship between "Delta-T" and system cost?
Are there better organisations/designs?

Banks and airlines have attempted this for decades. Insisting on completed transactions being committed to Remote Storage adds significant latency to the process. In a single-threaded application, this limits the processing rate. To achieve high-rate applications, (real-time) parallelism must be embraced with all its complexity and problems.

Revisting RAID in 2011: Reappraising the 'Manufactured Block Device' Interface.

2011-10-30T18:47:00.000-07:00

Reliable Persistent Storage is fundamental to I.T./Computing, especially in the post-PC age of "Invisible Computers". RAID, as large Disk Arrays backed up by tape libraries, has been a classic solution to this problem for around 25 years.

The techniques to construct "high-performance", "perfect" devices with "infinite" data-life from real-world devices with many failure modes, performance limitations and a limited service life vary with cost constraints, available technologies, processor organisation, demand/load and expectations of "perfect" and "infinite".

RAID Disk Arrays are facing at least 4 significant technology challenges or changes in use as I.T./Computing continues to evolve:

From SOHO to Enterprise level, removable USB drives are becoming the media of choice for data-exchange, off-site backups and archives as the price/Gb drives multiples below tape and optical media.
M.A.I.D. (Massive Array of Idle Disks) is becoming more widely used for archive services.
Flash memory, at A$2-$4/Gb, packaged as SSD's or direct PCI devices, is replacing hard disks (HDDs) for low-latency random-IO applications. E-bay, for example, has announced a new 'Pod' design based around flash memory, legitimising the approach for Enterprises and providing good "case studies" for vendors to use in marketing and sales.
As Peta- and Exabyte systems are designed and built, its obvious that the current One-Big-Box model for Enterprise Disk Arrays is insufficient. IBM's General Parallel File System (GPFS) home page notes that large processor arrays are needed to create/consume the I/O load [e.g. 30,000 file creates/sec] the filesystem is designed to provide. The aggregate bandwidth provided is orders of magnitude greater than can be handled by any SAN (Storage Area Network, sometimes called a 'Storage Fabric').

Internet-scale datacentres, consuming 20-30MW, house ~100,000 servers and potentially 200-500,000 disks, notably not as Disk Arrays. Over a decade ago, Google solved its unique scale-up/scale-out problems with whole-system replication and GFS (Google File System): a 21st century variant of the 1980's research work at Berkeley by Patterson et al called "N.O.W." (Network of Workstations). A portion of which was the concept written up in 1988 as the seminal RAID paper: "Redundant Arrays of Inexpensive Disks".

The 1988 RAID paper by Patterson et al and a follow-on paper in 1993/4 "RAID: High-Performance, Reliable Secondary Storage" plus 'NOW' talks/papers contain a very specific vision:

RAID arrays of low-end/commodity drives, not even mid-range drives.
Including a specific forecast for "1995-200" of 20Mb 1.3" drives being used to create 10Tb arrays (50,000 drives. Inferring a design from a diagram, taking a modest 5-6 racks).

RAID as a concept and product was hugely successful, as noted in the 1994 Berkeley paper, with a whole industry comprising many products and vendors springing up within 5 years. IBM announced the demise of the SLED (Single Large Expensive Disk) with the 3990 Model 9 in 1993 as a direct consequence of RAID arriving.

These were all potent victories, but the commercial reality was, and still is, very different to that initial vision of zillions of PC drives somehow lashed together:

Enterprise Disk Arrays are populated with the most expensive, 'high-spec' drives available. This has been the case from the beginning for the most successful and enduring products.
The drives are the cheapest element of Enterprise Disk Arrays:
the supporting infrastructure and software dominate costs.
Specialised "Storage Fabrics" (SANs) are needed to connect the One-Big-Box to the many servers within a datacentre. These roughly double costs again.
Vendor "maintenance" and upgrade costs, specialist consultants and dedicated admins create the OpEx (Operational Expenditure) costs dwarfing the cost of replacement drives.
Large Disk Array organisations are intended to be space-efficient (ie. low overhead (5-10%) for Redundancy), but because of the low-level "block device" interface presented, used and usable filesystem space is often 25% of notional space. Committed but unused space, allocated but unused "logical devices" and backup/recovery buffers and staging areas are typical causes of waste. This has led directly to the need for "de-duplication", in-place compression and more.
With these many layers and complexity, Disk Array failure, especially when combined with large database auto-management, has lead to spectacular public service-failures, such as the 4-week disruption/suspension of Microsoft's "Sidekick" sold by T-Mobile in 2009. There is nothing publicly documented about the many private failures, disruptions and mass data losses that have occurred. It is against the commercial interests of the only businesses that could collect these statistics, the RAID and database vendors, to do so. Without reliable data, nothing can be analysed or changed.
The resultant complexity of RAID systems has created a new industry: Forensic Data Recovery. Retrieving usable data from drives in a failed/destroyed Disk Array is not certain and requires specialist tools and training.
To achieve "space efficiency", admins routinely define very large RAID sets (group size 50) versus the maximum group of 20 disks proposed in 1994. These overly large RAID groups, combined with link congestion, have lead to two increasing problems:
- RAID rebuild times for a failed drive have gone from minutes to 5-30 hours. During rebuilds, Array performance is 'degraded' as the common link to drives is congested: all data on all drives in the RAID set needs to be read, competing with "real" load.
- Single parity RAID is no longer sufficient for a 99+% probability of a rebuild succeeding. RAID 6, with dual parity (each parity calculated differently), has been recommended for over 5 years. These extra parity calculations and writes create further overhead and reduce normal performance more.
Whilst intended to provide very high 'reliability', in practice large Disk Arrays and their supporting 'fabrics' are a major source of failures and downtime. The management of complex and brittle systems has proven to exceed the abilities of many paid-professional admins. As noted above, the extent and depth of these problems are unreported. Direct observation and anecdotal evidence suggests that the problems are wide-spread, on the point of being universal.
At the low-end, many admins still prefer internal hardware RAID controllers in servers. As a cost-saving measure, they often also select fixed, not hot-swap, drives. This leads to drive failures not being noticed or corrected for months on end, completely obviating the data-protection offered by RAID. These controllers are relatively unsophisticated, which creates severe problems in the case of power loss: "failed" drive state can be forgotten, as the controller relies on read errors to prompt data reconstruction via parity.
Instant, irreversible corruption of complete hosted filesystems results: the worst result possible.
Low-cost "appliances" have started to appear offering RAID of 2-8 drives. They use standard ethernet and iSCSI or equivalent instead of a dedicated "storage fabric". SOHO and SME's are increasingly using these appliances, both as on-site working storage and for off-site backups and archives. These are usually managed as "Set and Forget", avoiding many of the One-Big-Box failure modes, faults and administration errors.

Questions to consider and investigate:

Reliable Persistent Storage solutions are now needed for a very large range of scale: from 1 disk in a household to multiple copies of the entire Public Internet in both Internet Search companies and Archival services, like "The Wayback Machine". There are massive collections of raw data from experiments, exploration, simulations and even films to be stored/retrieved. Some of this data is relatively insensitive to data-corruption, other needs to be bit-perfect.
A "one-size-fits-all" approach is clearly not sufficient, workable or economic.
Like Security and encryption, every site, from household, to service provider, to government agencies, requires multiple levels of "Data Protection" (DP). Not every bit needs "perfect protection", nor "infinite datalife", but identifying protection levels may not be easy or obvious.
Undetected errors are now certain with the simple RAID "block device" interface.
Filesystem and object-level error detection techniques, with error-correction hooks back to the Storage system, are required as the stored data-time product and total IO-volume increases (exponentially) beyond the ability of simple CRC/checksum systems to detect all errors.
There may be improvements in basic RAID design and functionality that apply across all scales and DP levels, from single disk to Exabyte. Current RAID designs work in only a single dimension, duplication across drives, whilst at least one other dimension is available, and already used in optical media (CD-ROMs): data duplication along tracks. "Lateral" and "Longitudinal" data duplication seems an adequate descriptions of the two techniques.
Specific problems and weaknesses/vulnerabilities of RAID systems need analysis to understand if improvements can be made within existing frameworks or new techniques are required.
Scale and Timing are everything. The 1988 solutions and tradeoffs were perfect and correct at the time. Solutions and approaches created for existing and near-term problems and technologies cannot be expected to be Universal or Optimal for different technologies, or even current technologies for all time.

Why did the 1988 and 1994 Patterson et al papers get some things so wrong?

Partly because of human perceptual difficulties dealing with super-linear changes (exponential or higher growth exceeds our perceptual systems abilities)
and also because "The Devil is in the Detail".
Like software, a "paper design" can be simple and elegant but radically incomplete. To probe the designs' limits, assumptions, constraints and blind-spots, it requires implementation and real-world exposure. The more real-world events and situations any piece of software is exposed to, the deeper the level of understanding of the problem-field that is possible.

My Ancient Computer Project

2011-10-25T20:09:00.000-07:00

Preamble: If you're really looking for advice on floppy drives: 8", 5¼" and 3½", try these links:
http://www.retrotechnology.com/herbs_stuff/s_drives_howto.html
http://www.classiccmp.org/dunfield/
http://www.deviceside.com/

The good folk at "Device Side Data" sell a USB adaptor for 5¼ inch floppy drives and a range of relevant cables, power-supplies and enclosures.
For about US$100, you can have a working 5¼in USB setup, but it's bring-your-own-drive.
Sourcing a 5¼ inch floppy drive may be tricky: they went out of production 10-15 years ago.
Sourcing 5¼ inch floppy disks, new or 'slightly used' is probably a greater challenge.

Some weeks ago a friend (AF) asked if I could copy a 5¼ inch floppy disk for him, leading to this little adventure...

AF had a 3½ inch floppy that he might have copied everything onto and wanted to check.
Without specialist equipment, I knew I wouldn't be able to recover all potentially readable data, but I offered to do what I could with a standard drive.

In the end, AF decided to try something else, so I didn't get to do his copy.
But it did make me get my old 386-SX properly setup, networked so I could move files to/from it and a way to easily copy 5¼ in and 3½ floppy disks.
Before this, I still booted it occassionally, but the real-time clock battery had died and it had no network card or CD-ROM drive.

This decision led to weeks of farnarkling and some interesting lessons.

While it's unclear if my 386-SX will survive another 2 decades, the software can live on through tools like QEMU, WINE/Crossover and even DOSBOX. So there is some value in recovering the data both on the hard disk and my collection of floppies.
The impetus to recover data from unreadable media, 5¼in floppies, is obvious.
For the readable 3½in floppies, taking a copy now is a good investment of time if I ever want to access the data again: magnetic media does degrade over time. In another 20 years, the coating on those floppies may be flaking off in big lumps.

386-SX Initial Config:

purchased late 1991, ~$3250
386-SX CPU, 20Mhz (selectable to 8Mhz). No FPU. 8Mhz ISA bus, 8-slots.
Dual floppy drives, 5¼ in [boot] and 3½ in.
IDE disk. 80Mb, WD AC280.
single parallel port, dual serial ports, one used for mouse, other for modem.
5Mb RAM [max at time]
Super VGA1024x768 16-col, 640x480, 256 col. 512Kb [K-i-l-o not Mb]
33cm display. [fixed scanrate, can be destroyed @ wrong Hz]
mini-tower. pre-ATX power-supply (no 'soft' power switch or 'halt')
No sound-card.
DOS 5.0 and Windows 3.0

Current state-of-play is:

386-SX has DOS 6.20, Win 3.11, Networking, CD-ROM and Zip drive all working.
Hard-drive cloned/backed up and (5) ZIP disks read and copied. [ZIP drive back in its box]
Single 3½in floppy as A: drive. BIOS only allows booting from first drive.
5¼ in drive working as 2nd drive in a Linux machine (2001, Celeron 667) with built-in networking.
Working through copying all 5¼ in floppies I can find.
Tally so far [40]: 1 unreadable, 1 with errors.
Update 04-Nov-2011: 210 floppies read, 40-50 5 in: 8 "no data". 150+ 3½in, 5 "no data"
Now have a USB 3½ in floppy drive, can read those at leisure on newer machines.

The most important lesson for me came about two weeks in:

I was fixated on doing everything on my 1991 vintage 386-SX.
At one point I was running through the options of replacing the motherboard and the various costs which weren't attractive given I was 'just playing'.
Since USB became ubiquitous, finding machines/motherboards with floppy drive controllers is increasingly difficult, which means even embedded boards with FDC's are rare and expensive.
Then I realised I already had everything I needed in the 2001 vintage Celeron system I had tucked away.
It's loaded with Fedora Core 3 (support ended in 2004) with a linux 2.4 kernel. Old, but usable.
It was perfect for what I wanted to do.
It had a floppy drive controller and I could transplant the cable (with 5¼in 'slot' connectors) from the 386-SX to the Celeron 667.

I also learnt a little about floppy drive connectors:

5¼in drives have a 'slot' (card-edge) connector on the drive and a header not unlike a 40-pin IDE connector (34-pin is used).
'Classic' 3½in drives use a socket connector similar to IDE connectors (but 34-pin)
The $30 USB 3½in drive I bought uses an incompatible tiny connector (two versions, a plug and a socket with a conversion cable). Previously, I didn't know this variant existed. Noted to save other people from popping open put-together-permanently cases. I didn't care about the warranty, but the case needs to be firmly shut or drive operation is affected.

Before starting any work, I had to backup the original 3860-SX disk. My memory was that I'd bought a 30Mb 'RLL' drive (a 20Mb ST-506 drive with a modified controller).
Turns out I really had an 85Mb IDE (now called ATA or PATA), a "WD Caviar® AC280", not only larger, but it would allow me to connect an IDE CD-ROM drive in their as well. My unused hardware pile has any number of CD-ROM drives.

Most importantly, an IDE/ATA drive gave me the option of backing up the drive via an IDE/USB interface... Of which I have a number of versions.

I also had an old system backup of around 30 * 720kb 3½ floppies. DOS 5.0 and Win 3.0.
Using QEMU on a Linux system, I was able to restore this backup to a virtual disk drive.
Shuffling all those disks was painful and slow.

Connecting an early IDE drive to a modern(ish) IDE/USB interface didn't work.
Probably because additional commands were introduced to identify the drive, possibly because this old drive responded to "CHS" (cylinder-head-sector), not LBA (Logical Block Addressing). From the AC280 spec. sheet, the drive electronics did support any reasonable CHS settings, not only the physical layout.

My next preferred method was to connect a second IDE drive as a 'slave' (D: in DOS), fdisk and format it and copy the original drive contents, then connect this drive to a modern system with IDE/USB interface and back it up.

The first 3.5" HDD I tried from my unused pile (1Gb Fujitsu) had errors.
Next drive tried was a 3.5" 4.3Gb. Worked reliably.
In the final config, I replaced that drive with a slower, quieter 2Gb 2.5in drive, cloning it via an IDE/USB interface.

The Phoenix BIOS in the 386-SX is very old. Not only doesn't it support LBA drives, it seemed to limit drives to 1023 cylinders - and 15 heads/63 sectors. Around 470Mb.
A very large fraction of my time was spent fiddling with disk CHS specifications and attempting to get fdisk to ignore the BIOS settings.
[No, I can't update the firmware, the BIOS is pre "Flash-the-BIOS"].

I could setup multiple (large) partitions on the drive with Linux and the IDE/USB interface, but then would run into troubles under DOS and the 386-SX.
I tried 'fdisk' from DOS 5.0, 6.2, 6.22, Win-95 and FreeDos on the 386-SX, but all would only see the 470Mb allowed by the BIOS. Extra partitions would be displayed, but couldn't be changed.

I don't have enough unused 1.44Mb 3½in floppies to back up the entire 85Mb drive, so was very glad my 2nd method worked.

Getting a working ISA-bus (not PCI) network card was simple: I had two in my "unused bits pile".
The one I chose I'd bought new, a Netgear NE2000 clone, but I didn't realise that or find the box (with install floppy) for a while. Relied on Windows NE2000 driver and Internet downloads at first.
The other card didn't have a clear name/identifier, nor did I get a good match from the chip numbers.

This was another 'surprise': how to get any info on installed ISA cards.
I also spent 2 or more days fiddling with the IRQ/DMA settings on the NE2000 card. I'd forgotten the problems that PCI made go away. I ended up with IRQ 5 (COM2) and found an IO address range through trial-and-error. I don't know for sure if its a clash or not.
I couldn't find a tool that would list for me all the cards + settings in the system.
Norton's "SI" provides everything but DMA ports.
MSD (Microsoft Diagnostics) didn't help either.

Getting the CD-ROM working was a good idea, if a little problematic.
Using standard Linux tools, I was able to create an ISO image of the original DOS/Windows system and also add some additional tools.
The problematic part was creating a disk image with Uppercase filenames. Whilst DOS 6.2 (really MSCDEX) reads the root directory correctly, no files or directories can be read/listed.
Perhaps it is the 'Joliet' option I use that causes this... Had the same difficulty with the FreeDos CD.

I managed to get non-DOS booting via 3½in floppies:

FreeDos [tried last]. Booted, read CD-ROM, not installed on HDD.
BG-TLB: http://www.giannone.ch/bgtlb/ [Tiny Linux Boot]
floppyfw - floppy firewall. http://www.zelow.no/floppyfw/

Until I tried it, I wasn't aware that FreeDos uses linux as a base. Uses 'SYSLINUX' as the boot loader and seems to have a kernel.sys.

It took a good deal of searching to find any Linux that would support:

vanilla 386
no FPU
no RAM disk for under 12Mb,.

"Floppyfw" recognised the (single) NE2000-compatible network card, but didn't include a shell.
BG-TLB does include BusyBox, but only support 'plip' networking over the parallel port (not tried).
So while I have seen a linux shell prompt and been able to mount the DOS/FAT filesystem, I haven't been able to dual-boot the 386-SX or run it as a Linux only system.

Whilst FreeDos could read its install CD-ROM, DOS 6.2 was unable to read files/directories contents (the lowercase name problem above).

Another Linux not-tried was tomsrtbt: "Tom's floppy which has a root filesystem and is also bootable." http://www.toms.net/rb/tomsrtbt.FAQ. It advertises itself as "The most GNU/Linux on 1 floppy disk."
Which might have worked, but it formats 3½in floppies at the non-standard 1.7Mb, not 1.44Mb, with the caveat "Doing this may destroy your floppy drive".

One of the 'surprises' I got was being unable to replace the original 3½in floppy drive with a newer drive. The interface and connectors were all the same, but the newer drive wouldn't work in the old system.
Was it me connecting it incorrectly, a faulty drive or something more?
Unable to tell and unwilling to devote a bunch of time testing it.

One of the worst 'surprises' I got was after installing the IOmega (parallel-port) ZIP drive software on the system after a fresh install of Windows 3.11. [Windows 3.11 had decent Networking support and Microsoft still have downloadable a good TCP/IP stack.]
I had it all setup, tested and working and foolishly, in Windows, selected "Optimise settings" and the system hung.
Whereafter, the system couldn't see any Comms ports, serial or parallel. Which was very problematic because the 386-SX didn't come with a mouse port (DIN or PS/2). I used a serial port for the mouse.
Windows hung when it booted, leading me to try to revert the ZIP drive install and later to re-install Windows.
There were countless reboots and 6-8 hours later I gave up and went to bed.
I had realised/diagnosed that when the machine booted, the BIOS reported "0 serial ports" and "0 parallel ports". The BIOS setup screen only allowed me to selected HDD and floppy drive settings.

First thing in the morning, I powered on the machine and it worked perfectly. Including the original copy of Windows that would hang.
All I can think of was a power-cycle (off/on) cleared the fault, whereas a 'cold' or 'warm' reset (reset switch or ctrl-alt-del) didn't. In the many reboots, I hadn't thought to power-cycle the machine. [A note for myself and others experiencing weirdness on old hardware.]

I also have an old Dell Inspiron 7000, dating from 1999. I got it with a removable ZIP drive, figuring I could do backups and bulk-data transfers using it and the parallel-port ZIP drive. While tinkering around, I disassembled the other ZIP drive. It's an IDE/ATAPI drive, but the connectors are non-standard. I was hoping I'd be able to kludge it to work on another system - but not to be.

One of the BIOS limits, noted above, was it will only boot from the first floppy drive. When I'd configured the machine, I'd made the 5¼in drive "A:". Part of my reconfig was to move that drive and make the 3½in drive "A:", so it could boot from it. And most "floppy disk images" on the net are 1.44M for 3½in drives.
Then I executed a perfect "rookie mistake": I forgot to make a bootable 3½in system disk before moving the 5¼in drive. And all my system disks were, of course, 5¼in.

The 386-SX is in a "mini-tower" case with very limited space between the back of devices in the drive bays and the motherboard etc. This makes running cables and changing drives quite time consuming. Especially with older connectors that are loose and can be jiggled off. This was part of the reason to move to a 2½in HDD - much more space. I did need to find a spare 2½in-3½in mounting kit first.
I didn't believe the weight of the whole system, let alone just the removable cover. Meaty!

Early on I replaced the on-board/real-time clock battery so the system would remember the time over reboots. More modern systems use 3V lithium batteries (CR2032) for this. This old motherboard used a 4.5V alkaline battery (mounted off-board with Velcro). About a week in, I used 3 AAA alkaline batteries in a modified carrier (and a cannibalised connector) to craft a replacement. A less pretty way is to load 3 batteries in a cut-to-length tube with wire soldered directly to exposed battery ends. Soldering wires directly onto batteries requires some technique. You may need help if you try it. There could be an explosion risk with alkaline batteries becoming overheated (they are marked "do not dispose of in fire"). Research this properly or get help if you choose to do this.

The system, including DOS, quite happily accepts dates of 2011. No problems there...

One of the problems I haven't addressed yet is: "How do I clean the drive heads?"
Back in the day when I was a sometime 'operator' on mainframes, cleaning tape drive heads was part of the ritual. We used Isopropyl alcohol + cotton swabs - because it didn't leave a residue. The swabs would always come away stained with oxide coating from the tape.
For these old drives, I've two reasons to want to be able to clean the heads:
a) these are old drives and may well have an internal dust build-up, and
b) older disks are likely to shed more of their coating than when new.
Some media formulations from the late 1980's are known to suffer problems. I've heard first-hand accounts of the work needed to recover period audio-tape recordings due to this problem.

5¼ inch floppy drives load the heads directly in-line with feed-slot.
It is possible to get a swab in there, though v. difficult to see what's happening.
3½in floppy drives drop the disk to both lock the disk in-place and load the heads, hence the heads, being offset, aren't accessible for swabbing through the feed slot.

Another thing to investigate...

What is an electronic document and why that Matters.

2011-01-27T12:50:00.001-08:00

#comment-2157 agimo.govspace.gov.au back-to-the-future-another-chance-to-influence-coe-development

#comment-2158 agimo.govspace.gov.au back-to-the-future-another-chance-to-influence-coe-development

First one says:
 There isn't yet a clear Industry Standard, so do Nothing for a while
longer.

Second says:
 The basis of your problem is you're conflating many problems with
conflicting requirements under the banner of "document exchange".
 And there's *never* a good reason to give non-authors an editable copy.

With a little postscript: FOSS definitively solved the "multiple
concurrent update versioning problem" - 20 years ago?
I didn't say, the 'secret sauce' is Use Simple Text Files!

That's at the heart of all these problems: binary formats.
Should've been banned 30 years ago... But weren't and now we're paying
for it.

Feedback model of Software Production

2010-10-06T06:38:00.000-07:00

The bottom part of the diagram is a representation of a single step in the software production cycle.
It consists of two Amplifiers (A1, A2) with gains G1 and G2.
[The top part is an outline of the possible succession of steps in the process. This too has interesting characteristics. Not addressed here.]

Part of the output of A1 leaks back, via A2, into the input of A1.
A classic schematic for a wide-band amplifier or an oscillator.

How much is fed back, and in what time-relationship to the input?
The value "R". (Really it should be "Z" for impedance, and have lead/lag/in-phase components)

The crucial aspect of the diagram is that all inputs are "positive" - if the output directly follows the input, not the inverse (as found in a classic "negative feedback", or correcting, system.)

The model says there are only 3 internal metrics to be measured an one external inflow and two outflows.
The 3 internal metrics:

G1 - Raw Lines of Code per Day
R - errors per hundred Lines of Code
G2 - Time to fix one error. In a normal Project, this can't be less than 1/2 day because of overheads, analysis and diagnosis, testing and change commit.

The external inflow is Errors from steps further along the chain.
I've forgotten to draw its inverse, an outflow to previous steps.

The most obvious outflow is "Delivered Lines of Code".
You can see from the nature of "R", there may be significant 'lag' (time delay) so at notional Project Delivery, Errors may not be fully realised.

There are exactly 3 dynamic states of this system, given that G1, G2 and R are constant.
The "gain around the loop", G = G1 * ( G2 * R ).
[if r = 1/R; then G = G1 * (G2 / r ). Possibly more intuitive.]

The Loop Gain, G, can be greater than, equal to, or less than 1. [mistake in diagram, I wrote '0' (zero)]
The system dynamic are:

G < 1. System "converges". It heads towards the X-axis asymptotically (?). The smaller G, the faster the convergence.
G = 1. System is in a "holding pattern". It never moves towards completion, but doesn't move away.
G > 1. The System "diverges". Every day of effort produces more than one day's extra effort. The classic "Death March".

Having 3 variables, they can be manipulated to push the system in a desired direction. If you were paying for the project and wanted a result, you'd want the smallest values for R and G2.

Reduce G1. Slow the coding process. Counter-intuitive and probably would cause the Project Managers' head to explode. But its there in the figures. Spend more time fixing errors and less in creating more new code and their concomitant errors.
Reduce G2. Better people, better systems, better tools, better training to find faults faster and produce better solutions. So do subject-area experts. Lesson: Don't move people away from what they know.
Reduce R. Better Design, Design and Code Reviews and better people/tools/processes for coding. Create less rework, adopt practices that reduce errors (like pair-programming). And most crucially, don't over-work people. Even a small increase in Error rate can push a project into the Red Zone...

Programming Projects are defined by Rework and its management.

Projects do not "fail/get later one day at a time". They fail before they start.
Let go, they often gallop away from a solution.
No amount of effort can retrieve them - the more effort expended the more that's needed to complete them. This was the genesis of Microsoft's 2005 "Longhorn Reset". 25,000 person-years were discarded.

Wireless Broadband Plans (Australia)

2009-12-20T01:02:00.000-08:00

Talking to a friend about "going broadband" led me to think I should write something down...

If you are "on the road", MacDonalds offer free WiFi hotspots. If that works for you, saves all the hoopla of getting 3G broadband.
'Lifehacker' suggests there is a 50Mb download limit.
maccasfreewifi.com provides a hotspot finder for 706 locations.

Additional Comments [19-Jan-2010].
This writeup of George Bray's is clear and concise.
I think it shows a definitive, cheap, simple solution to Mobile Broadband, plus flags a bunch of caveats.
Stop now and go read his piece if you want or need to actually do something soon.

Here's a writeup and video of my "Broadband from the Beach" equipment.

http://geobray.com/2010/01/19/broadband-3g-from-the-beach/

George

Caveat:

Not having had/run a mobile wireless broadband service myself, there'll be stuff I don't know about...
Please contact me by email or "comment" (below) to correct errors/misinformation, provide links or additional war-stories/advice/information/tips/comments.

Mobile Wireless Broadband:

There is some confusion about "Mobile" and "Wireless" Internet/Broadband.
There are mobile-phone 3G networks, long-distance data-over-wireless and 802.11 Wireless Ethernet, or "WiFi". This piece concentrates on 3G-based services. Australia is yet to see broadscale WiMax or 802.11 networks.

Mobile or Wireless Internet/Broadband and Laptops are an extremely powerful combination.
The only caveat is Mobile Internet generally comes from a USB device (or via USB from your phone).
The lack of a hardware/stand-alone firewall means it's difficult to share a connection and you give up some security.

Hand-crafted arrangements can be built:

... in Armidale and the only place I could get reliable 3G link was from my bedroom window.
Nokia N95 in the window with bluetooth link to EeePC 701SD on the desk and from there an Ad-hoc WiFi link providing access to me anywhere in the house.

Whirlpool Forums (a.k.a. "Broadband Choice") are a very good resource: http://forums.whirlpool.net.au/forum/114
Whirlpool have a list of Networks and Resellers, as of Dec-2009.

There are three "3G" networks in Australia offering data (HSDPA: High-Speed Downlink Packet Access) and competing reasonably aggressively:

Telstra - best coverage, generally best potential speed. Max 42Mbps,
Optus
Vodafone, 3, Hutchinson

"3G" (3rd Generation) mobile phone services are based on CDMA (Code Division Multiple Access) protocols. This means if you have signal, you can get service (not so with GSM, which is Time Division Multiple Access). It also means you can improve your download speeds and effective coverage by attaching a larger aerial.

Telstra have very few or no resellers of its NextG broadband network.

Optus and Vodafone/Hutchinson have many resellers: folks like Virgin Mobile and Exetel offer great packages using one of these networks.

But there is a nasty 'gotcha': "Roaming Charges".
On some plans, especially low-end, you can be charged (from memory) around $100/Gb for roaming.
This may have changed, but is worth checking for specifically.

"Roaming" is where your provider doesn't have a service, but another provider does and will carry your bits - for a price...

On older 2G ("GSM" or 'digital mobile') networks, the data is called GPRS - General Packet Radio Service and used to be quite expensive and comparatively slow. Some carriers use this as a fall-back when you are 'out of area'.

Some friends did the "Gray Nomad" thing through Western Australia and back to Sydney this year.
They got a Telstra 3G USB modem and a new laptop.
It generally worked very well. They had about a week of 'glitches' at the beginning, but eventually found a helpdesk operator who'd seen the problem and got them on-line.
They also had a problem with the laptop power-supply 'failing'. It ended up being a seed lodged in the connector.

Travelling, they had 3 types of service:

None.
Slow, out in farming communities.
Fast, near mining towns and cities.

They were able to stay in-touch without on-site support: that's pretty impressive.
George notes on the CLUG list he regularly achieves 4Mbps with a tripod-mounted directional aerial.

Google threw up a few sites offering service comparisons:

broadbandguide.com.au
comparebroadband.com.au
broadbandexpert.com.au (Seems to be limited, perhaps sponsored)

They have different coverage. For example, only 'compare broadband' threw up Exetel plans for $17.50/month for 1Gb and they only say "speeds vary".

Selecting your plan:
[See also below "Specific advice for selecting a service and setting up your connection"]

Choose Prepaid or Contract. Prepaid means you can experiment.
Contracts: You are locked in for 12-24 months with "Early Termination" penalties. Beware of "automatic renewal" at the end of the contract - they may require you to specifically "opt out".
Prepaid: You need to look for $ per Gb and 'Expiry' - how long you've got to use a payment. (Hint: Don't apply payments until close to when you need them of you might lose them.)
Prepaid 'gotcha's: Recharges must be planned.

First problem is to change plan options (phone or internet mobile plans) before applying the 'recharge'. Payments/recharges cannot be altered once applied. [From my limited experience]
It can take 48 hours for a 'recharge' to be applied to your account for some carriers. While you might technically be in credit, your service could stop or be charged at a (much) higher rate until the Telco systems process your payment.

Modems: Are they 'locked' to a network/ISP, or can you use it anywhere? You're also going to be interested if the modem is supported by your Operating System.
Important user advice: Select your modem/gateway first.
Speed and Download limits: Mostly how plans are differentiated.
Beware of the costs of downloading e-mail.
Monitoring Usage: Whether on a pre-paid or contract plan, you will most likely have to monitor the usage seen by your provider. George in his CLUG post, points out that that there is no standard way and potentially is an application tied to a specific operating system. There is also the issue of stale and incorrect usage data being reported by the tool.
This is a nasty 'gotcha': If you are paying by the Mb and can't reliably track it, don't fly too close to the limit. At best you'll be suddenly cut-off or 'traffic shaped', at worst get a very nasty bill.

Virgin Mobile (and presumably others) also provide a Home Phone package.

For $80/month (Dec-2009) Virgin Mobile give you a box that connects via the Optus 3G network and gives you unlimited VoIP calls within Australia (with a normal dial-in number), 4Gb/mth download and they're a little coy about the speed. But it falls back to 128kbps.

People like Virgin (and Optus?) have a "Fair Use" policy. If they deem your usage 'excessive', your account may be cancelled or you'll incur fines.
'gotcha': On the CLUG list there was a report of Exetel dumping a customer as they "were unprofitable". You can't avoid this: the service providers supply you their service subject to their Terms and Conditions. You must agree to the "T&C" or there is no contract.

The only 'remedy' is to avoid the problem. If you intend to heavily use a service, sign-up for an appropriate plan.

Great Tips:
Tridge happened to mention at a CLUG meeting that at home he runs both an ADSL link and a back-up Mobile Wireless link. If ADSL goes out, he can still access his home computers from outside.

The normal Unix "route" software for IP doesn't allow for asymmetric links: there can only be one default route.
Packets may find their way to your host via a second route, but must return by the default route.

The details of "asymmetric routing" using "iproute2" are spelled out in this thread on the CLUG list.
Links to iproute2 Project and Linux Advanced Routing and Traffic HOWTO.

Speed and Latency:
"Oils ain't Oils" applies.
4Mbps on mobile wireless will not provide the same apparent level of service as 4Mbps over fixed-wireless, 802.11 or fixed-line (ADSL or Cable).
Whilst you may have a theoretical download speed of 4Mbps, TCP/IP needs to correct errors and to wait to 'ACK' packets to come back before sending more. The round-trip-time can limit link performance.

Mobile wireless suffers from transmission errors (noise + lost packets), variable data rates (there are other users competing for air-time) and long latency (affects round-trip-time) due to many layers of packetisation, buffering and forwarding.

Long latency only marginally affects big downloads (the TCP/IP 'ACK' problem), but can be a deal-breaker for real-time services (VoIP, Skype, IM, ...) and when downloading many small files - as in normal websurfing and collecting email.

Radio reception is fickle for many reasons - tune in to AM radio at night for an example.
Mobile networks necessarily need to track 'handsets' and SIM cards to route traffic: what towers are you currently near?

Which means mobile networks must have a centralised handset location database and a single or limited number of interconnect points. This means your phone or USB dongle is many hops from 'central office'. The ISP connects from there.

Telstra trumpet they support 21Mbps (going to 42Mbps?) - but not on all towers and your "actual speed" will vary with aerial, equipment, signal strength, other users and backhaul capacity...
As my Gray-Nomad friends found, Telstra won't invest in unwarranted (unprofitable?) capacity.

Even when you hit a "white spot" (high-speed tower), you have to compete with everyone else in the area for a slice of bandwidth. May be great in the bush, may be less great in highly used areas - like the city or a rural/regional tower offering the only effective broadband in the area.

Additional Comments:
[30-Dec-2009] Thanks to CLUGers, in no particular order, for their help and comments:

Alex Satrapa
Scott Ferguson
Ian Munsie
Keith Goggin
Miles Goodhew
George Bray

A great piece on using mobile wireless by George Bray sent on the CLUG list.

A note on the CLUG list on setting up Telstra 3G (the MF626) on Linux (Ubuntu 9.04) by Carlo Hamalainen. [19-Feb-2010]

Contract 'gotchas':

"Early Termination Fees" that most providers are charging these days:
Sign up for a two year contract costing $480 all up, they'll sting you for $540 if you terminate the contract before the two years is up (not to mention, they automatically sign you up for a further two years if you don't opt out during the three month grace period at the end of the contract).

The moral of the story is make sure you check the conditions on those contracts, folks. Early termination, "roaming" (in the case of at least one provider, they will charge you the 'normal rate' for data if you fall off the 3G network onto their GPRS network), "acceptable usage" and other caveats.

General comments on Mobile vs Fixed-line service:

Summary: "it sucks, don't do it." :P

Seriously, Virgin Mobile offers free tethering with their $54/month iPhone plan - that's $54/month including payments on the phone. I don't have the details on me, and I can't find the plan on the Virgin Mobile website, (the kiosk in the Canberra Centre near Mac1 will be able to help you).
I think it's a 300MB/month data allowance.

On the flip side, I've never had a good experience with 3G broadband except for the situation where you can mount the antenna in just the right spot and leave it there.
In other words, you get the best 3G coverage in the same places you get the best ADSL 2+ coverage.

Optus specific comments:

provide an internet addressable IP to their mobile broadband customers
does not charge extra for roaming onto their slower GPRS link
provides a "usage meter" to help you track your usage (some reports of delays and misreporting)
Virgin use the Optus network with mostly the same Terms and Conditions, with the additional proviso that peer-to-peer traffic may be capped.

Non-mobile wireless, Canberra only:

Netspeed 'Longreach' works with Linux (SuSE 11.2) with a 2km range, but "Slows when Foggy". The Netspeed whitepaper says it's a fixed wireless network using a Motorola product with a 10km range. Expect 7Mbps, with a 14Mbps maximum. Their 'availability' checker lists 3 deployed base stations (Canberra South Side) and 2 towers under deployment (north side). Good rates if it works for you.

Tethered data on the Hutchison/Three network:

I use my Nokia 6110N paired via Bluetooth to my netbook for tethered Internet access. I pay $8/mo for the privilege of the "Internet" connection (Otherwise you're in a walled garden). They have a $5/mo plan with much higher data cost and lower included data volume. Their roaming data charges used to be astronomical, but I think they changed this recently ...
Generally it's slow, the latency's massive, but it works and it's very convenient for email/web. Also, if you commit to a 12 or 24-month plan they double your included data volume.

Specific advice for selecting a service and setting up your connection:

When shopping for a 3G plan I would recommend selecting a modem first, then finding an ISP that will sell you it - or will sell you a SIM only.

One feature you should look for is an external antennae connection - very useful in some areas. Check to make sure the model fits without interfering with access to other ports, and that it doesn't raise to laptop off the desk.
I'd recommend a 5m USB repeater cable for better reception, and to allow placing the antennae away from the user.

Make sure your machine is capable of supplying more than 500ma to the USB ports. Consider buying a "split-tail" USB lead - one input for data and power, another for extra power from a second USB port if your machine is USB power taxed or lacking. For about $3 from JayCar you can buy a portable USB power supply - it takes 4 AA batteries and reduces the output to 5V, hack on another single AA carrier and it will supply 5V from Ni-MH batteries.

Most of the UMTS (3G) modems I've looked at have the option of selecting HPSDA only (no roaming surcharges). I'd recomment setting your modem to that unless you are going into a bad signal area. Some plans do not include SMS.

Check the modem model numbers - when I was looking a couple of months ago I noticed that Dick Smith in Civic had 4 prepaid Dodo Huawei modems.
I noticed that one of them had an extra alpha in the model number - one sales rep assured me the "G" model was the same as the non-G, another said it was better. A quick search on the internet showed the "G" as a lower powered model - supposedly no longer distributed. The Huawei has an external antennae connection.

You will hear reports of slow connection speeds under Linux with some modems - generally because the user has an old driver module with a very small maxSize (the module takes a vendor and product parameter only).

Gateway, connection sharing and stand-alone firewall:

... have a bigpond 7.2 Home Network Gateway (3G9WB)
4 port Ethernet router
configurable security
runs off 12V DC with internal voltage regulator more than adequate for car use
can use high gain antenna for fringe area
EXPENSIVE $90 per month 5GB shaped
VOIP works OK mostly @ ~40kms near line of sight to tower (or better

So I bought a new Intel Mac Mini...

2009-12-19T21:00:00.001-08:00

In early 2006 I bought a Power PC (PPC, G4, DDR2-SODIM) Mac Mini to use for a Uni course.
Could've gone Intel, but it was new technology, so I avoided it...

Coming up to 4 years on, and I've filled the disk, the CPU fan is a little noisy, new things like 'Google Chrome' are 'intel-only' and OS/X 10.6 ('Snow Leopard') is Intel only... Plus VMware and Parallels can run on Intel platforms.

This time I paid the ~20% extra for a top of the line 2.53Ghz core duo, 4Gb DDR3 RAM, 320Gb 2.5" SATA - which is roughly 4 times the capacity in all dimensions as the PPC version.

First surprise, doesn't come with a DVI-VGA adaptor in the box. BUT, now there are 2 video connectors, 'mini-DVI' and 'miniPort'. The shop only had a 'miniPort' VGA adaptor on the day and I went home with that...

Attach to KVM, boot and - no monitor found. VGA adaptor works fine directly connected.

Second surprise, 800Mhz Firewire is a 9-pin connector not 6-pin (400Mhz on PPC Mac).

Booted new Mac and it says "got an old Mac?" and gives instructions to reboot the old one "holding down the 'T' key" to start a transfer. Which meant closing out all my running Apps and discovering just what they meant.

Which was hold down the 'T' key whilst the machine was shutting down, before the reboot.
It comes up in 'Transfer mode' - displaying a "firewire transfer" symbol. Apparently you can use an 'ethernet' transfer mode... Didn't try that.

After 2-2.5 hours, the transfer completed - 75Gb moved over. Accounts, Preferences, Applications and User Data copied and working.

I've had to reload Adobe Acrobat Reader, apparently it wasn't a 'Universal Binary'.

Even the licensed & registered 3rd-party software I had (MS-Office, NovaMind, OmniGraffle) 'Just worked' on reboot. The transfer was clever enough to not move system binaries. As a bonus, I got a work Perl system back... In the past I'd installed a second, non-Apple, version and managed to zap CPAN.pm on both versions...

Third surprise was initialising "Time Machine". It took 2.5hrs to transfer 75Gb, which is about right - I've timed the old HDD @ ~20Mb/sec sustained raw read rate.

It took a day and a half (!!) for "Time Machine" to initialise... 33-36 hours, Ouch!
The USB drive, a 1Tb Western Digital 'Book for Mac', runs at about 10Mb/sec for raw reads (36Gb/hour).

I can only guess why it ran so slowly... The "Time Machine" is a HFS filesystem.
"Activity Monitor" showed some very large numbers for Gb read from the system Disk, but the amount written to the USB drive matched the progress numbers reported by 'backupd'.

In the middle of the initialise, I rebooted the machine because I could see no evidence of progress.
Rebooted, restarted the initialise and off it went again. Seemed to pick up from where it left off.

There were one or two other times that it took hours, literally, to backup a few Gb.
Now it does 2-3Gb in 5-10 minutes, as you might expect.

What I love about "Time Machine" is this is the first time I've had a good backup system for my home desktop. Seems odd as this is what I do professionally, but there it is.

Another cute thing with "Time Machine" is it can use a Network connected device - like a wireless connected Apple 'Time Capsule' or instructions are out on the web for using a "NAS" - like an old Linux box.

The "Time Machine" interface is well thought out.
You 'enter Time Machine' and the Finder comes 'front and centre', the background changes to 'beginning of the Universe' picture, all the other Apps slide away and the rest of the decorations appear.
Neato!

When you're done, your desktop just slides back into place. Very cute.

There's a major caveat with "Time Machine" - it's granularity is "whole file".
I use 'mailbox' format (a 'folder' contains all messages in a single file) versus 'maildir' format used by Apple Mail (every message is one or more files in a directory). Everytime a new message gets added to one of my 'folders', the entire file is written. I've looked and there are several which as 1+Gb :-(

It's pretty much 'rsync' with a nice GUI and a few tricks in the HFS filesystem. One of which I read (to confirm) is that it allows directories to be linked. This was removed from POSIX because you can get complex loops in filesytems...

Things I don't like about "Time Machine", but aren't reasons to bail:

I haven't been able to find how to turn on decent logging.
The default syslog entries written in /var/log/system are high-level (#files, total size).
Where/How do I get a list of the files it dumped?
I could exclude Thunderbird's mail folders and save on backup time and space.
What I'd really like if I could tell it that these files grow by concatenation, and to link to just the most recent version... Though data might be lost if file is truncated, but the semantics of that should be simple. Eg. if the current backup version is perfect subset of the new file, just concatenate the new lines. Otherwise, create a copy.
Haven't found, after a quick look, the API. Be nice to be able to integrate with 'rsync'.

'locate' works, but there is some internal magic that seems to exclude 'Desktop' files...
Almost everything is scripts, but I can't find the list of excludes. Perhaps it is just silently failing because of the number of files I have in a directory there...

[30-Dec-2009]
The Western Digital "MyBook" only transfers 10Mb/sec (36Gb/hr). Seems slow.
If "Time Machine" reads or writes every block on the filesystem, that would explain the initial load time and observed volumes of data read.

Mini-display port versus Mini-DVI.
If two VGA adaptors are used, Display-port is the 'primary' (system boot and menus).
If VGA and DVI on Mini-DVI: the Mini-DVI is the 'primary'.
[Don't have enough dongles to test all combos]

Working with a USB/DVI KVM.
The Intel Mac Mini has a DVI-D connector, the PPC Mac Mini has a DVI-I (integrated, allows for DVI-D or DVI-A).
The connectors on the KVM I bought are DVI-I, which won't plug into a DVI-D socket. (A DVI-D cable will plug into either a DVI-I or DVI-D socket).
I won't spring $50 for a separate DVI-D cable until I talk to the vendor.

Meanwhile, I saw a $25 VGA to DVI-A cable.
Connects to the KVM, but nothing displays. Presumably because the DVI-D connection on the monitor only does digitial. Who'd have thought :-)

The PPC Mac Mini came with a VGA/DVI-I adaptor. It doesn't work in reverse. :-)

The KVM exhibits the usual "doesn't play well with sleeping computers" problems:

While the KVM does connect speakers/microphone through, there is sporadic hum.
with 2 or 4 ports connected, put a system to sleep via menu and it is 'woken' after a second or two. Something to do with the USB interface of the KVM. Execute a delayed command, flip to another input - All Fine.
Single device connected - system will sleep from menu, but at some random time, it's alive again.
Waking a system is impossible without unplugging/replugging its USB connection to the KVM
Not 'sleep' exactly, The keyboard and mouse sometimes become unresponsive. Either reconnect them to the KVM or reset the KVM (remove all powered connections, including video).

This DVI/USB KVM has the habit of complaining about selecting an inactive port. It will lock up (needs 'reset') if moved to an inactive port. Not sure if that's sometimes or if there on the port for more than a few seconds.

I tried turning off the 'beep' from the keyboard - locked up & needed a full 'reset'. Could've been I swapped it to an inactive port.

DDR3: Memories ain't Memories

2009-11-30T15:04:00.000-08:00

Purpose: Summary/Intro to DDR3 memory with Intel Nehalem (iCore 7, Xeon 3400, 3500, 5500, 7500). NOT a reference document. Dell biased. [Nov 2009]

Audience:
Those needing to configure memory sub-systems on Intel Nehalem/DDR3 systems:

Selecting between pre-configured systems.
Populating DDR3 memory on a new motherboard
Upgrading an existing system
Configuring a new server

Background: I got into trouble specifying some new Dell servers. DDR3 is different enough that you will most likely not get the result you intend. This is about Intel, not AMD, processors though some concepts will transfer.

Selecting a DDR3 memory sub-system config requires knowing what you want to Optimise/Maximise:

Speed (Latency or Throughput) vs RAM Size vs Price vs Power vs H/A features

This note is skewed to Dell equipment, read their "DDR3 Technical Whitepaper" for Poweredge 11g servers" (PDF).
The Intel Xeon 3400 series architecture is described in this Intel PDF.

Dell's "Quick Reference Guide" for Installing and Upgrading DDR3 Memory in 11g servers contains an excellent summary of memory options for 5500 series processors. [There is an error in the last diagram. Text says 24Gb, diagram shows 36Gb]

Dell's "Transition Guide" for 11th Generation servers, (PDF), page 11, outlines their DDR3 memory restrictions by type, speed and ranks per MCH.

Reading the Intel doco, these acronyms were unclear to me:
UP = UniProcessor, DP = Dual Processor, MP = Multi Processor

Wikipedia has a good page on "DDR3 SDRAM", lots of technical info.
Other of their articles describe "ranks" and the relationship to x4 and x8 (4-bit or 8-bit DIMM's): DDR_SDRAM, and DIMM#Ranking.
Dual-rank (DR) DDR3 is made with x8, Single-rank (SR) with x4 DIMM's.
Quad-rank (QR) modules exist, not sure what organisation.

Basic DDR3 data:
1Gb to 16Gb DIMM's possible. 1,2,4,8 Gb sold by Dell.
speed (Mhz) in 266Mhz steps: 800, 1066, 1333, ?1600? (not available from Dell in 2009)
64-bit transfers. 72-bits transferred for ECC.
800 = 10 ns, 6.4 GiB/s per channel (1,000,000,000 per Gi) B = bytes.
1066 = 7.5ns, 8.533 GiB/s
1333 = 6.0ns, 10.667 GiB/s
1600 = 5.0ns, 12.8 GiB/s

Intel describe Nehalem as a 'tock' (new architecture) in 45nm. A 'tick' (more speed) coming next.

32nm variants coming later.

Wikipedia on Intel Nehalem microarchitecture:
on-chip Memory Controller Hubs (MCH).
DDR3 memory or DDR2 "Fully Buffered" some chips.
No Northbridge chip on 'advanced' chips, with Southbridge collapsed into PCH (Peripheral Controller Hub).
QPI (Quick Path Interconnect) for NUMA: access non-local memory in multi-chip systems.
Replaces FSB on high-end.

Wikipedia summary of the Xeon 5500 series (DP not UP). Note the 'basic', 'standard', 'advanced' ranges. DDR3 and QPI speeds change as well as CPU clock speed through the range.

There is an obvious throughput limiter: the common 8Mb L3-cache and the MCH interface.
Presumably it's "non-blocking". i.e for 3*MCH @ 1333Mhz, around 32Gb/sec (Intel X5570).

DDR3 memory sub-system Variables:

Prices and Sizes:
faster DIMM's are more expensive.
larger DIMM's are more expensive per Gb.
8Gb is largest from Dell and premium priced.
Registered DIMM's are more expensive as well as use more power.
Different sized DIMM's can be mixed - but same size & rank in every corresponding slot.

Speeds:
DIMM speed - not the whole story. Fastest bus speed possible.
Bus Speed. All buses clock at same rate, including with Dual Processor.
Processor Memory Controller Hub speed (MCH) - may be limited.
5500 series (DP) has 3 MCH + 2 QPI links, but varies from 800Mhz to 1333
3400 series (UP) has 2 MCH at 1333Mhz. May only support 2 slots/MCH.

NUMA:
Memory is tied to a chip and non-local memory is accessed across the QPI (direct MCH to MCH).
Without a second processor, only one set of DDR3 slots is accessible, for a DP board, halves the available slots.
Processors on a DP board have to be identical. Can't mix'n'match different models.

Slots:
Max DIMM Slots/channel: 3.
Some processors only support
Total DIMMs: Processors * MCH * slots/MCH.
Some boards only supply 2 slots/MCH.
Lower-end motherboards may only provide 2 of 3 available MCH
slot 0 - furtherest from MCH (?). Must be populated first. Implies geographic addressing.
Dell derates maximum bus speed:
- slot 0: 1333
- slot 1: 1066
- slot 2: 800Mhz
I.e. if anywhere on the bus there is a chip plugged into any 'slot 2', the bus is limited to 800Mhz.

Ranks:
ranks - roughly 'chip select' or 'bank select'. A type of Interleaving.
up to 8 ranks per Channel.
DIMM's come in 1-, 2-, 4-rank: SR, DR, QR
single QR allowed (Dell), slot 0 only.
Diff ranked DIMM's can be mixed - but same size & rank in every corresponding slot.
Dell don't supply SR or QR DIMM's as a server option. (2009, appears that way)

DRAM chips:
x4, x8 - DRAM chip organisation. 4- and 8-bit chips.
x4 is SR.
x8 for DR. and QR is ?

DIMM Types:
Unbuffered vs Registered (UDIMM, RDIMM)
Parity vs ECC
address parity only on RDIMM's.
RDIMM's use 1Watt more per DIMM.
Can't mix DIMM types on a bus?

Other:
Other memory configs available: Mirror, Lock-step and SDDC (Advanced ECC)

"Mirror" mode - 2 DIMM's per MCH, but in RAID 1 config.
For high reliability. Even with ECC, caters for dead chips/connection.
reports half installed memory to O/S.

Advanced ECC mode - ganged MCH's. 128-bit fetches.
Implements SDCC - Single Device Data Correction for x8 DIMM's

Dell and Intel recommend "Balanced Configurations" and "Memory Optimised" modes.

Standard Memory organisations from Dell are:

High Performance (Slot 0 only used)
Balanced Performance
High Capacity (all slots populated)
Mirror Mode or Advanced ECC
Power Conscious

Intel Xeon Nehalem processor ranges:

3400 ("Lynnefield") UP
3500 ("Bloomfield") DP
5500 ("Gainestown") DP
7500 ("Beckton"?) MP. Not released (Q4-2009)

Notes:

The FSB has gone and is now replaced by Memory Controller Hubs (MCH) and QuickPath Interconnect (QPI). The word 'interleaving' is not used.

Each MCH can control just 3 DIMM's. But Dell's DDR3 bus clocks slower as you add more DIMM's... 1333 for 1 DIMM, 1066 for 2, 800Mhz for 3 DIMM's.
There are Single-, Dual- and Quad-rank DIMM's. Max 8 ranks per MCH.
[Connection charts exist that visualise the rules]

The QPI allows a chip to access memory via off-chip MCH's.
Of course, if you only install one processor chip in a two-proc board,half the DIMM slots are unusable. Need the MCH's of the 2nd chip to get to them.

RDIMM's use more power, but can be larger, faster and support 'address parity' and are required for more esoteric modes.

The Xeon 3400 series has 2 MCH's and often only 2 DIMM's per MCH on the motherboard.

The 5500 series has 3 MCH per chip. Some motherboards have only 2 DIMM's/MCH.
For a two-chip system, max 18-slots - with the caveat only 9 accessible if a single-chip is used.

Upshot:

If you want 'cheap', you'll use UDIMM's@1066
If you want 'fast', you'll use 1 RDIMM@1333 per MCH and a CPU to match.
For 'max memory' - you're down to 800MHz and fill all slots.
for high reliability, you might use 'Mirror mode' or SDCC.

Speed: Throughput vs Latency:
There is around a 20% memory throughput improvement by going with 1333Mhz memory where possible.

This piece on a SUN blog has a table of memory throughputs for Intel Xeon 5500 series with DDR3:

"Table 3 - Relative Bandwidth Comparisons"
"Table 4 - Relative DIMM Power Comparisons"

It lines up ranks, bus speed, DIMM speed and number of channels populated against 'throughput'.
[Not access speed. High-performance mode is 1*DIMM per channel]

I've not been able to get clear info of the effect of DDR3 "ranks" on performance.
From the tables in the SUN piece, it seems to imply that memory controllers understand 'ranks' and can utilise them to improve throughput (not latency).

Another caveat is that SUN claim to be able to run their memory bus @ 1333Mhz even if 2 or 3 slots per channel are filled. The generic and Dell doco I've found says the bus derates to 1066Mhz for 2 slots and 800Mhz for 3 slots.

SUN don't give the bus throughput for "100%". A snippet from their table.

1xDR @ 1066 = 36%
1xDR @ 1333 = 44%
2*DR @ 1066 = 68%
3xDR @ 800 = 74%

Quotes from the SUN conclusions:

Key takeaways from the above (Table 3 on bandwidth) are:

for DIMM configurations that support both speeds, memory bandwidth is up to 12% higher with 1333 DIMMs than with 1066 DIMMs (which is not obvious from chart since it shows bandwidths relative to 1x DR per channel and not a direct comparison between 1333 and 1066 for a given configuration)
for a given capacity, one dual rank DIMM per channel provides higher bandwidth performance than two single rank DIMMs per channel

and

From the table (4: Power) it can be determined that:

for a particular DIMM configuration and bus speed, DDR3-1333 DIMMs consume up to 6% less power than DDR3-1066 modules
for a given DIMM configuration, the incremental power required to operate DDR3-1333 DIMMs at 1333MT/s data rate vs. 1066MT/s is 8% or less (also not obvious from chart since it shows power relative to 1x DR per channel and not a direct comparison between 1333 and 1066 for a given configuration)
and somewhat less obvious but equally important:
a dual rank DIMM operating at 1333MT/s consumes less power than two single rank DIMMs at 1066MT/s

My Life as an Inventor

2007-06-02T22:45:00.000-07:00

I always carry a notebook and have filled up more than I recollect... Ideas and invention has always been a fascination for me.

My work in I.T./Communications - Systems and programming - have always been an outlet for my creativity.

1972-1979

Study at Uni of NSW (Computing Science and some Chemical Engineering), 2 years as Analytical Chemist plus 2 years in mainframe Operations at CSR, 2 years full-time study plus clerical and Computing work.

One of the defining events for me at Uni was being taught "Operating Systems" by John Lions. I was on the second ever Unix kernel course, the first with The Book: the Lions Commentary. As a matter of passing usefulness, we had to learn 'C' in ~4 weeks.

1979-1984

Worked in I.T./Computer in Communications and National Security.
Demonstrated good design and troubleshooting skills.
Worked on projects, real-time systems maintenance, database development and high-performance/high-availability systems.
Ran the International email service for O.T.C.

1986
At Softway, with Greg Rose, one of the principals, I put the following 4 ideas to the board:

RAID - we were a Unix systems and kernel house and could write drivers.
Multi-protocol Routers as an extension to the CSIRO micro-node we were programming
Multi-function printer/fax/copier
Digital photocopier/printer

The ideas were turned down without comment. 3 out of 4 went on to become major products.
A good hit-rate.

I'd taken it on myself to go off and find contracts to keep the company afloat - around $100k/yr. Enough to pay $30k salary and at least one of the directors. This initiative didn't help my stock there.

In 1988, I got to know Ken Thompson reasonably well when he visited Sydney University and taught for a year.

1989
I'd worked too hard, become 'burnt out' left Softway and moved 'to the country', the South Coast of NSW.

I approached the Telecom Product Development Fund with a proposal to build a single TCP/IP based network, using commodity Intel hardware running Unix, to switch all their 'Text Services" [Telex, Teletex, Fax, email, file transfer].

I'd worked for 7 years on exchanges at O.T.C. and knew a bunch about Telco services and digital networking and messaging. This was before Telecom/Telstra had an email service.

I'd run the O.T.C. commercial e-mail offering, "Minerva", as well as multiple in-house Unix systems using Sendmail, ACSnet/MHSnet and even some uucp.

My proposal included a store-and-forward network and 3 possible variants of fax machines - that would use DNS-style human readable addresses, not just phone numbers.

For ordinary fax machines, this meant having a human-readable cover-sheet - eliminating wrong numbers.

Telstra's comments were:

We don't know what you're proposing. It's not one thing.
We don't have any one person who can evaluate this.
Nobody else in the world is doing this.
Your cost estimates are too low. [commodity hardware vs. 'telco' pricing]
Your cost/benefit estimates are wrong. [no help on what would be right]
From the chief of Text Services in Melbourne on consolidating all text switching:
"Why would we want to do that?"

That was the year that AARNET brought the Internet to Australia.

1992
I built the first widely available, complete, Australian daily weather database while at CSIRO.
Created a browsable index, redefined the file structure, performed full-coverage testing on the dodgy code I'd inherited - and got the product released at the annual ABARE 'Outlook' conference admist much fanfare.

1994
Designed a VPN for linux. Implemented by David B Deaves.
Worked between Canberra and Perth offices over public Internet.

1995/6
Filed, had examined and granted my 'netserver' petty patent.
A domestic router, firewall and internet connection. Also allowed USB devices to be connected.
The petty patent lasted 5 years and was only for Australia.
Unable to find a VC interested in pursuing this.

1999
For Federal Govt. "Business Entry Point" administered the initial ABN registrations.
Forecast peak demand to within 5% and designed the first web 'busy tone' to handle (massively) excessive loads.
Final peak load was 25-times the design load, with consistent 5 second response times.

2001
Filed an international PCT application for a Power Change-over device [allows servers or a rack of equipment to be switched to another power outlet without disruption.]
Examined and granted in Australia.

2005/6
A new take on Internet Security - probably provable secure.
Introduces a replacement for VPN's: Virtual Air Gapped Networks and Controlled Secure Links.
By adding other elements it is suitable for Military/Government grade security as well as normal commercial operations and domestic/e-commerce use.
Global market for VPN and related devices is currently estimated around US$25Bn.
Replacing and upgrading VPN technology would probably create a market 10-20 times larger.

Patent application filed in Australia on 27-September-2006.
Refiled in 2007, now lapsed.

Historical Storage Prices - Raw Data

2007-05-27T18:10:00.001-07:00

These are the raw data I collected from some Australian PC magazines from 2002-2007.

There are problems with the data:

- these are not 'best retail price' - just what I could find advertised

- inconsistent markup and discount schedule. Couldn't find a single advertiser.

- not same-same. RAM was DDR 3200, then PC133

- Disks. Ignored cache size, and SATA.

- all drives assumed to be 7200rpm

- didn't collect data on 2.5" drives and SCSI/fast drives.

RAM

Year	Size	32	64	128	256	512	1024
2002	512			$117	$238	$510
2003	512			$51	$96	$202
2004	512					$149
2005	1024					$168	$189
2006	1024						$135
2007	1024						$159

Flash

Year	Size	32	64	128	256	512	1024	2048	4096	8192	16384
2002	256	$53	$87	$147	$305
2003	512		$63	$109	$187	$350
2004	1024	$39	$49	$75	$129	$200	$400
2005	1024			$40	$60	$90	$158
2006	2048					$55	$95	$175
2007	4096					$25	$45	$69	$99
MSY.COM	$					$15	$15	$25	$45	$92	$177
	$/Gb					30	15	12.5	11.25	11.5	11.0625

Disk

Year	Size	40	60	80	120	160	250	320	400	500	750
2002	120	$205	$275	$375	$375
2003	160	$165	$190	$220
2004	250	$95		$115	$155	$215
2005	320	$80		$90	$129	$135	$206
2006	500			$75		$109	$155
2007	750			$86		$100	$133	$133	$149	$186	$345

DVD - 50 pieces

Year	Size	4.7 Gb
2002		$135 (guess)
2003		$90
2004		$60
2005		$40
2006	8x	$40
2007	16x	$35

RAM

Year	Best $/Gb	Yr/Yr Ratio
2002	936	2.4375
2003	384	1.2886
2004	298	1.5767
2005	189	1.4000
2006	135	0.8491
2007	159
Avg		1.510373357

Flash

Year	Best $/Gb	Yr/Yr Ratio
2002	1176	1.68
2003	700	1.75
2004	400	2.53164557
2005	158	1.805714286
2006	87.5	3.535353535
2007	24.75	2.237288136
Avg		2.260542678

Disk

Year	Best $/Gb	Yr/Yr Ratio
2002	3.125	1.136363636
2003	2.75	2.129032258
2004	1.291666667	1.567556634
2005	0.824	1.329032258
2006	0.62	1.666666667
2007	0.372
Avg		1.565730291

DVD

Year	Best $/Gb	Yr/Yr Ratio
2002	0.574468085	1.5
2003	0.382978723	1.5
2004	0.255319149	1.5
2005	0.170212766	1
2006	0.170212766	1.142857143
2007	0.14893617
Avg		1.328571429

Summary

The Yr/Yr ratios are used for forward projections.

The 'Est Flash' column uses the current 'best price' (MSY) for flash memory and a Yr/Yr ratio of 3.25.

Flash memory is now very much cheaper than RAM - forward projections not done.

Year	RAM $/Gb	Flash $/Gb	Est Flash $/Gb	Disk$/Gb	DVD $/Gb	Max flash	Max Disk
2002	936.00	1176.00	1176.00	3.13	0.57	256	120
2003	384.00	700.00	700.00	2.75	0.38	512	160
2004	298.00	400.00	400.00	1.29	0.26	1024	250
2005	189.00	158.00	158.00	0.82	0.17	1024	320
2006	135.00	87.50	87.50	0.62	0.17	2048	500
2007	159.00	24.75	11.06	0.37	0.15	4096	750
Yr/Yr ratio		2.26	3.25	1.57	1.33
2008		10.95	3.40	0.24	0.11
2009		4.84	1.05	0.15	0.08
2010		2.14	0.32	0.10	0.06
2011		0.95	0.10	0.06	0.05
2012		0.42	0.03	0.04	0.04

Blogging Tools

2007-05-19T18:17:00.000-07:00

Google's 'Blogger' is cheap and cheerful and a reasonable way to get started... But it doesn't work well for me in a number of ways.

'Safari' on Mac isn't well supported
The edit window is very small
I can't compose off-line and upload previous work simply
My articles aren't stored on my computer making it easy for me to re-use and organise them.
The dinky little input/edit window isn't resizable
Checking final layout is painful - flipping in and out of 'preview' mode
Using Blogger and it's AJAX interface kicks up Safari's memory use - and kills my machine

So I went searching for some tools. Their are Mac clients that look good - but it's pay, pay, pay all the way on Mac :-)

Downloaded 'BloGTK' - it might work in the X-11 environment, but that uses too much memory. I 'only' have 512Mb - which once upon a time would've seemed unlimited.

So I found 'flock'. It's a modified Firefox browser which understands blogging and actually has resizable windows and stores new material on my disk... But it doesn't allow me to import what I've already written. And the publish interface is incredibly annoying if you editing/updating.

And there's a Firefox addon 'Firescribe' (formerly Performancing). It does some stuff reasonably well. Not sure about saving posts locally.

Still searching for tools I can work with ;-)

News-May-11

2007-05-11T18:20:00.000-07:00

Microsoft Naysayer bandwagon gets crowded: http://blogs.zdnet.com/BTL/?p=4820

Sarah Friar of Goldman Sachs:
It's over after Vista.
Microsoft needs markets that can move the revenue needle.
Microsoft will be hurt by Linux, Apple and software as a service.
Microsoft can't be nimble enough to compete with Google.
Microsoft's messaging stinks. [what's "Live"?]

"It all seems a bit premature. Sure Google may be sexier–just ask the Microsoft employees that are defecting and burning bridges on the way out–but can you really say a company with $28.8 billion in cash is on the downswing? I can't."
[http://blogs.zdnet.com/microsoft/?p=380]
-> ROE is 35+%. Should be able to repeat that
-> $17B 2006/7 spent on share buy-backs => propping up the share price

Chunks: hidden key to RAID Perf. http://blogs.zdnet.com/storage/?p=130
* Cache
* Striping
* Chunk Size [size of stripe on each drive]

Web DNA - Attributor: http://blogs.zdnet.com/micro-markets/?p=1321
-> Copyright Control, Monetisation On-line

Typical Linux Desktop User
http://blogs.zdnet.com/hardware/?p=397
-> http://www.desktoplinux.com/news/NS9755856281.html

2008 Year of Mobile Linux? http://blogs.zdnet.com/mobile-gadgeteer/?p=359

RedHat takes OLPC knowledge corporate: (one Laptop per child) Global Desktop http://blogs.zdnet.com/BTL/?p=5022

New Rules for Software VC. http://www.sandhill.com/opinion/editorial.php?id=132&page=4

Here is a list of must-haves to “jump start a startup” today and make it a candidate for venture investing.
* Begin with a first-class, seasoned management team

* Find a market undergoing demand growth where some form of disruption is creating a space for a new business

* Buy the components of the business – technology, distribution, customer base - that you can; only build what you have to (which usually does include building innovative technology

The Enterprise Software “Threatdown” http://www.sandhill.com/opinion/editorial.php?id=131 [Top 10 threats]

1. Buyer Ennui
2. Looming Skills Shortage
3. Software Security
4. Usability
5. India’s IT Services Firms
6. Private Equity Plays
7. New ERP Challengers
8. The On-Demand Model
9. Mobile
10. Software Appliances

SandHill: The State of the Software Industry – 2007
http://www.sandhill.com/opinion/editorial_print.php?id=133
Innovation Accelerating as a Top Priority
Internal Funding is Healthy -> revenue growing 9-13%
+ Margins are equally healthy. Megavendors (>$10B Rev) margins 35-40% EBITDA.
+ Margins rest 12-18%
+ software share of the customers’ IT budget will continue to increase
++ 2006: 29.6% expected 35.7% in 2009
External Funding Driven by Private Equity
+ Compound Annual Growth (CAGR) 26% in VC investment
Implications for Software Vendors and Customers
+ Customers: Customers have a paradox in front of them – how can a company that is increasingly spending their budget with the largest software vendors ensure they get access to smaller vendors who are driving innovation?
+ Vendors: Software vendors need to adapt to the bottom-up and center-out diffusion model within this innovation wave.

Sand Hill 30 - The stocks driving the next generation of software and services.
http://www.sandhill.com/finance/sandhill_index.php

Megavendors
Pure-play software vendors whose every move shakes the industry.
HP
IBM
Microsoft
Oracle
SAP

Big Shots
Well-positioned, large vendors in healthy sectors.
Adobe
BEA
CA
EMC
Intuit
Symantec

Specialists
Vendors with deep expertise and a strong market position.
Autodesk
Blackbaud
Business Objects
Cognos
Hyperion Solutions
Informatica
Lawson Software
Sybase
Tibco Software

New Big 5
The most powerful offshore IT services providers in the world.
Cognizant Technology Solutions
Infosys Technologies Ltd.
Satyam Computer Svcs.
Tata Consultancy Services
Wipro Ltd.

New Agers
Next-generation vendors leveraging new technologies and models.
Red Hat
RightNow Technologies
Salesforce.com
Taleo
WebEx Communications

What Price a Life? II

2007-04-22T23:01:00.000-07:00

How do we individually and collectively survive and thrive?
What do we need to know from the past to avoid the mistakes of our forefathers?
How will our future depend on what we do in the present?

Reframing what looks like a question about death and mortality into a much deeper, broader question that can surprise and sometimes delight the viewer but always takes unexpected paths. There is enough variety in the question to make informative, provactive media for more than 10 years. Real information on our collective choices and their consequences will help shape the political debate and agenda in a unique way.

Having a Life

In 1901 Australia had a population of 3.8M. In 2006, it was well over 20M.
In 1911, 30% of the population worked in Agriculture, which accounted for 20+% of GDP. Over 40% of the population lived in rural areas. Most families were >4 children.
In 2006 this was around 4% and 3% of GDP. 90+% of the population lived in urban areas. Families are <=2.
ABS - 2001 Yearbook, "A century of population change in Australia".

AIHW life expectancy data:
1900 M:55 F:58
2000 M:77 F:82

In the 1950's the average family, with 4 children, was supported on a single wage or income:
- the average wage in a city
- 40 acres (16ha) in a 'mixed farm' (dairy, pigs, poultry, crops)
- or an author writing "one book a year" [no citation]

In the 2000's, two above average wages are needed to purchase a house, women are deferring children until age 32, 400ha is not economic for broad-acre or grazing enterprises - and authors need 8-10 books 'in print' at one time to make a living.

We are all better off - we must be. But at what price? Why does it take so much more to pay for our lives? Where does the money go...

Taking a Life

The headline meaning, "what does it cost to end a life?" can be examined in many ways.

For example comparing and contrasting the dollar cost per life in warfare.

the monetary cost of killing a person in War over centuries - the American Civil War, The Boer War, WWI, WWII, Korea, Vietnam, Gulf War I & II.
What was the cost the USA of bombing Afganistan "back into the Dark Ages"?
- and what about the cost to the Taliban and the Afhanhi community there and around the world?
And the low-tech version, How much money did it cost in Mogadisheu to wield machtees?
- and what was the collectve cost of Ethnic Cleansing to their country and the countries that have provided aid for decades?

What is the effect on the global and individual economies of the Arms Suppliers (the USA, UK, France & Germany), the arms users and the Arms trade blackmarket?

But the world doesn't need just another War Documentary, even if it does take a different view.

And their is an individual, personal side to this equation:
What is the time, energy and opportunities passed up for a single man to fight for the US military?

If he does come home in one piece, what are the long-term effects on him, his family, his community?
And for those that increasingly come back as 'damaged goods', what of their lives afterwards?

Saving a Life

Every year hospitals, paramedics and rescue crews save thousands of road accicent victims.

There is a very real dollar cost in saving, or trying to save each of these people. It's part of our culture that we want to offer the same helping hand to everyone. But are we that equitable and elgalitarian? Are there places we quietly just don't serve because the people are somehow 'not as worthy' or 'the costs are too high'. Did the community make those decisions - or have them made for us?

But the victims and their families have to live after 'the accident'. Head injuries are often life-changing. Personalities change - people become violent, abusive and irrational. It's hard for them, and harder for their carers.

How do the paid professionals cope, the families and friends and those that used to depend on them?

And how do we individually and collectively deal with the knowledge that many were preventable?
We are better at not killing people on the roads than China and South Africia. But can we do better?
Commercial Aviation shows what can be done - if there is a will.

But do we want to pay that price? Discipline, rigour and attention to detail. And maybe not everyone automatically has 'the right to drive, to handle a lethal weapon'. How many people are we happy to see die needlessly by our collective inaction?

Politicians won't frame that question. But if they don't, who will?

Supporting us in the lifestyle to which we've become accustomed

We sit on the terrace and sip a nice latte or fine chardonay - but there are consequences for the rest of the world.

All the people who laboured to bring a simple thing together.
And to keep us safe while there and getting there.
We have water to drink, electricity to heat and light, phones to chat upon and buildings that stand up.
But most of all, we are blessed with abundance. We don't toil in the fields or labour in factories. Never before in history have so many had such a fine life. Many things we take for granted - even as a right - were once not even the domain of Kings. Refrigeration, detergents, clean running water, safe food, antibiotics, microsurgery, phones, TV, computers, the 'Net...

These we take for granted... It wasn't always so - and isn't so for the majority of the Earth's population.

And we are borrowing from the future, from our childrens' inheritance, to support this lifestyle.

"Water security" is now Real News. Wasn't it always? Folks in the bush have always been there.
Soon we'll have to consider the consequences of the drought - erosion, loss of soil and unavailability of food and goods.
And the profiteering that will go alongside scarity.

And where do our farmers go when the fertiliser runs out?

Oil is too precious to be burn like trash - look around you and find something that is not composed at least partly of plastics. It is either there because of petro-chemicals used in its production/manufacture or in its transport and use. When Oil runs out, the true cost will not be 'oh no, I can't drive', but 'where's all the consumer products?'.

Conclusion

The concept is to present difficult questions about how we live our lives and the implicit and explicit choices we make - in a sensitive and insightful manner. To ask questions to which there are no simple answers - only choices. Where 'living in the question' is the only right-minded response.

To provoke informed debate. To say things and address issues in areas politicians fear to tread.
And hopefully grab an audience and entrall them. To take them on an unpredictable, exciting journey.

What price a Life?

2007-04-22T18:07:00.000-07:00

This is an idea for a probably infinite set of media programs... That could inform, entertain and may provoke (shock, horror!) some thought/introspection.

In 3 or 4 generations we in the Western World have gone from a farming and manufacturing economy and society, to a 'service economy'. Just one generation saw - electricity reticulation, street lighting, radio, automobile, aircraft, the 'production line', industrial scale War [WW I] and the end of domination of European Royalty and Nobels.

We've gone onto create new life forms and manipulating DNA. To knowing most of the building blocks of matter and creating nuclear bombs capable of extinguishing everyone on the planet many times over. And to generations living off social welfare, using drugs and escaping into the 'virtual worlds' of TV, computer games and on the 'Net.

We didn't get here by accident, but we could lose everything in a flash with an accident - or through others intentionally destroying it.
Appreciating what we have and why we have it is essential to preserving our lives.

"What Price a Life?" It's ambiguous and can be taken many, many ways. Such as:

Price

'price' has many dimensions:

Dollars
Time
Energy
Resources (animal, mineral, vegetable)
Opportunity Cost - the cost of choosing 'this', not something else
Environmental
Human - personal, family, friends, group, community, town, country, ethic grouping
Human - physical, mental, emotional, spiritual
Sociological
Consequences - what are the on-going results of decisions, actions and inaction?

a Life - I

"What price for Taking a LIfe?" With a gun, a knife, .... Intentionally and accidentally. In War, in peace and at home.

Or:

Saving a Life - hospital, road safety, industrial safety, nuclear
Extending Lives - good health and nutrition
Prolonging - last month, 3 months, 12 months
Living a Life. e.g. a monk, a celebrity, a cleaner, a guard, a doctor, a teacher, a prostitue, a farmer, ...
Creating a Life - IVF, surrogacy, Gene Therapy...
Selling/Trading a Life - slave traders, 'black-birding', white slavery, sex slavery, ..
Preventing/Containing a Life - jail, detention, ...
Seeking a New Life - immigrants, refugees, ...
Freedom, Self-Determination, Self-Expression, Creativity
Justice, Equity, Fairness

a Life - II

"What Price a Life here and now?"
Or:

Over time
In this town, community, country
For this sub-group - aboriginals, Native Red Indians, Inuit
By religon/religous belief
By Sex (M/F/other)
By sub-culture - terrorists, drug-pushers, regime...

a Life - III

"What Price a Human Life?"
Or:

A wild animal
A pet
A domestic animal (food, fibre, fleece, milk, eggs, ...)
An endangered species
A not 'cute and cuddly' thing
Insects, parasites, pests...

Request to Twinings Tea Company: Create more business through the Net.

2007-04-04T23:01:00.000-07:00

How to INCREASE your profits and keep old fogeys like me happy...

PLEASE bring back your wonderful variety of leaf tea!

History

Since the early 1970's I've been discovering and drinking your teas. There are other brands - but I like yours :-)

In 1986 at an IT startup company I helped found we collected & drank over 30 varieties of your teas - as well as our own Espresso machine. Wonderful times!

Around 1988 you changed the formulation of "Earl Grey" and the colour of the packaging. Tasted like Burgamont oil from mint not oranges.

And since you've increasingly focused on what can be easily sold in the high-street and supermarkets. Limited range and culling slow-sellers. Migrating to tea-bags.

ALL GOOD THINGS!!
All great, not merely good, business sense.

New Business Model

How can you increase your profits and cater to old fogeys like me???

A. The Net - direct sales with minimum qty.
B. Regular "speciality packs" into supermarkets.

Internet sales

The people who want to buy your specialist leaf tea are buying a luxury item. A higher price marks extra value. And yes, there would be a price-point sales will decline.
you already have many national distributors & can supply locally (by mail) to most of the world.
You get to charge double for smaller batches that you make less often.
Selling direct means you cut out the distributor/reseller - and double your margin again.
And you don't have to do anything much different. Every country now has Internet Sales 'Fulfillment' organisations that charge a modest fee and do everything else...
minimum quantity sales [like 1Kg or $25] means your profits don't get sucked away by transaction fees.

In the Supermarkets

Look at Aldi and their approach to 'special items'. They make them available for a limited period and then swap out the stock for new specials. You can do a different speciality pack every 3 months for 4 weeks. Savvy buyers will stock up.
In Australia, well known brands have occasional 'nostalgia' packaging. Tins printed to celebrate an anniversary usually. They aren't real 'collectables' - but a nice promotional vehicle that in today's recycling conscious world don't get trashed for quite some time. They are a long-term in-home promotional display - that you make an upfront profit on!
[I have a 250g Earl Grey tin bought around 1990 that sits on my counter top & I see everyday.]

Generation X & Y effects

It's true the Baby Boomers [1946-1964] have a lot stronger connection to Ye Olde Teas - and all the ceremony surrounding them. They also have very high purchasing power now, like what they like (are brand loyal) and are generally comforted by links to their 'roots' - especially pleasant/fond memories.

Their children, Gen X, are surprising - they're turning 40 and have taken on many of their parents preferences and patterns. Why do the Beatles and Rolling Stones still sell so well?? Gen X & Y know and share many of their parents preferences.

And you could be tapping that market as well...

Market Testing

Australia is a great place to test :-) Two large supermarket chains with 60%+ coverage, 80%+ internet access/use [home or office] and a bunch of cashed-up baby boomers - and a reliable & cheap parcel post system.
Won't cost you much to market test and establish the vectors/relationships.

I suspect there is a HUGE pent-up demand for some of your now premium product, from a small but loyal band of 'believers'.
Surveys are unreliable - Dollars at the cashpoint are the only survey that matters.

Summary

You can have both the new market of limited choice, fast/easy prep., high turnover + lower margins
AND
Ye Olde World "Slow Food"-style premium-product with high-margins and spectacular range.

The Internet has changed everything. Let it change your business too.

New Blogger Interface

2007-03-23T02:06:00.000-07:00

Previously, wrote notes on the Blogger Interface. Was it ATOM? (Probably)

These are not meant to be instructions! They are notes on the process of using the API manually

New API doco:

http://code.google.com/apis/blogger/gdata.html#Get_Feed_Manual

"To get a feed of a blog's entries, you send the following HTTP request to Blogger" [Everything in XML!]

curl -o blog.1 http://itilopia.blogspot.com/feeds/posts/default

"After you send the GET request, Blogger may return a redirect, depending on various factors. If so, send another GET request to the redirect URL." Curl takes care of this...

Posting: Client Login [URL encoding needs to be done in the POSTS]

Then authenticate the user. To do that, send a POST request to the following URL:
https://www.google.com/accounts/ClientLogin

From http://code.google.com/apis/accounts/AuthForInstalledApps.html

POST /accounts/ClientLogin HTTP/1.0
Content-type: application/x-www-form-urlencoded

Email=johndoe@gmail.com&Passwd=north23AZ&service=blogger&source=Gulp-CalGulp-1.05

[CAPTCHA example]
POST /accounts/ClientLogin HTTP/1.0
Content-type: application/x-www-form-urlencoded

accountType=HOSTED_OR_GOOGLE&Email=johndoe@gmail.com&Passwd=north23AZ&service=blogger&
source=Gulp-CalGulp-1.05&logintoken=DQAAAGgA...dkI1LK9&logincaptcha=brinmar

Where

logintoken	(optional) Token representing the specific CAPTCHA challenge. Google supplies this token and the CAPTCHA image URL in a login failed response with the error code "CaptchaRequired".
logincaptcha	(optional) String entered by the user as an answer to a CAPTCHA challenge.

curl's POST doesn't work... This URL has a working example for calendar using wget...

http://denizyuret.blogspot.com/2006/05/google-calendar-how-to-delete-multiple.html


$ wget --no-check-certificate -O blog.2 --post-data 'Email=stevej098@gmail.com&Passwd=xxxxxxx&service=blogger&source=Gulp-CalGulp-1.05' https://www.google.com/accounts/ClientLogin
$ grep Auth blog.2

To use the 'Auth' string:
wget -q -O- --header='Authorization: GoogleLogin auth=XXX' http://...

And this piece from the instructions is cracker-jack! You gotta go look at the

Now send the POST request to the appropriate Blogger URL:
http://www.blogger.com/feeds/blogID/posts/default
Note: This URL is the same as the URL in the <link rel="service.post"> tag that appears in the <head> section of the human-readable version of the blog.

It also appears in the XML downloaded:
<code><entry><id>tag:blogger.com,1999:blog-3110200838296430371.post-4190105953841900395</id>

And this is how you push it in :-)

In the body of the new POST request, place the Atom element you created above, using the application/atom+xml content type.

Final command:

a=$(grep Auth blog.2)
wget -q -O- --header="Authorization: GoogleLogin $a" --header 'content-type: application/atom+xml' --post-file=blog-entry.xml http://www.blogger.com/feeds/3110200838296430371/posts/default

List all Blogs owned by user

Save XML description of all users blogs & tags into file 'blog.4'
wget -q -Oblog.4 --header="Authorization: GoogleLogin $a" http://www.blogger.com/feeds/default/blogs

neat. Something worked as advertised...

Query a Blog

Save XML of all blog entries made in March, 2007 (in blog.5)

wget -O blog.5 'http://itilopia.blogspot.com/feeds/posts/default?updated-min=2007-03-01T00:00:00&updated-max=2007-03-31T23:59:59'

Cute quote from doco (I wonder where to find *that* list?):

To query using other standard GData query parameters, just change the URL to use other parameters. (Note that Blogger doesn't support GData category queries or full-text search queries.)

Which might be here:

http://www.google.com/base/api/demo/html/demo.html#query

Deletes and Update/Edits

Too hard... Not clear. no examples of fully generated URI's. Tried to generate a URI for an item/article - failed.

here's the suggestions. untested...


Note the edit URI's of the events you'd like to delete:
wget ... | tidy -xml -wrap 999 | grep edit

Delete the event by using its edit URI; this is a two step process, because wget does not redirect correctly:

wget -nv -O- --header='Authorization: GoogleLogin auth=XXX' --header='X-HTTP-Method-Override: DELETE' --post-data='' 'URI'

Watch the output and note the redirect URI with gsessionid attached, then execute:

wget -nv -O- --header='Authorization: GoogleLogin auth=XXX' --header='X-HTTP-Method-Override: DELETE' --post-data='' 'URI?gsessionid=YYY'

Demonacracy - Putting the Demons back into Democracy

2007-03-20T19:29:00.000-07:00

In this posting on "Why all Monopolies become Corrupt", I was flying without a parachute - using a browser that didn't support Google's spell-checker.

So I banged it out and published - only to get pinged by a friend on this great typo: "Demoncracy".

I love the term - it does represent what many small-l liberals of both Left and Right think is happening these days with "machine men" politics feed by Big Money Political Parties.

In Australia, we are governed by under the 5% view. With ~10M voters, there are fewer members of the main political parties than your average good sports club. [Described with a little more colour in the last few years in Adelaide, South Australia.]

There are fewer than 500,000 - and probably under 100,000 - members of the main political parties.
The Parties represent themselves and the views of their members, the rest of us have the option of "take the full package - you'll love it" or not... It's often Hobson's Choice - neither is remotely close to what you might want if you had the choice. And professional politicians (an oxymoron?) consistently rank as more disliked/untrusted than any other profession.
Exactly the opposite applies to indepentents elected to Federal Parliment - they are seen are highly principled and ethical.

Australians being "very laid back" (what used to be called apathetic) just go along with this state of affairs. We're not lotus eaters nor univolved and uncaring - just wear the wrong clothes to a weekend football match!

We're mostly 'comfortable' and just getting on with our lives - Politics is often seen as an a three-yearly annonyance. A view I'm sure the people of Eastern Europe that recently put their lives on the line to have a Democracy don't share... We have bills to pay and places to go - and active lives to lead. Politics can take care of itself - it has done so all this time!

Our current Political Party Machine system is a direct result of the statistical work of George Horace Gallup in the 1930's that led to his correct prediction, against all other pundits, of the 1936 US Presidential election by Franklin Roosevelt.

For fascinating, first hand accounts of this, see Alistair Cookes' "Letter from America" of Monday, 7 October, 2002 - What the American people want and Monday, 17 July, 2000- Now read on...

The system of Polls driving elections, speeches and policy results in very little difference between parties. It looks like "Tweedle Dee and Dumber".

All Monoplies tend to become corrupt

2007-03-20T18:11:00.000-07:00

This is not about I.T., but we have to work in the real world :-) It does affect us and our work.

That's a Big Statement: ALL Monoplies tend to become corrupt.
The codasyl is, given enought time.

The reasoning/explantion is very simpe - comes from feedback theory.
For any effective control system you need a controller, input and output sensors and two sets of actuators. These form a feedback system that attempts to maintain the system at a 'set-point'. There's the whole open/closed loop control thing. 'Open' means you do no control. 'Closed' systems are what I'm considering here.

The input and output sensors are needed to feed the model embedded in the controller.
- the output is the state you want near the 'set-point'. You have to know if you are on-target or not.
- input levels/rates/values are needed to drive the model the controller embodies.

The actuators adjust the controllable input - one set to increase, another to set decrease. There may be some inputs to the process that you can't control. Such environmental inputs - like air or water. You can't control their pressure, temperature, contaminants, ...

The key here is that you need two or more counter-opposing forces. If you have only one, then you can only hold a system where it is (if it's very stable) or drive it in one direction. The limiting case is the system fails or "max's out" - it goes to a maximum value and stays there. The stuck throttle problem - one that no driver wants to encounter.

Applying Control System Theory to Organisations

Management is supposed to be the internal control sytem for organisations. But who keeps them honest? Who checks the output/operations of any organisation?

The feedback system here is external to the organisation:
How do the behaviour and attitudes/thinking of organisations get held to acceptable values?

There has to be external one or more 'counter-vailing' forces that can enforce change - as well as transparent or obvious 'outputs' of the organisation.

Demoncratic Governments enshrine these principles with the "Separation of Powers". There are at least 3 opposing forces: the Parliment, the Executive and the Judicary. Within the Parliment, there are often two 'houses' - offering the possiblility of opposing forces. Within the process of Government, there are many 'checks and balances' - various bodies, such as the Audit Office, Way and Means committee and Productivity Commission, that can both observe the outputs of agencies and enforce changes.

Outside of both public and private sectors is the Media. Their role, beside pedalling 'shock and awe', is to make the 'outputs' of organisations transparent and visible/open. Democracies then rely on the sensitivity of politicians to 'the ballot box' to enforce change - eventually. Taking away/limiting 'free speech' by fiat or threat (of police/military action) or removing the effectiveness elections removes any consequences - short of armed rebellion/uprising.

Publicly held companies are required to report their financial results - and woe betide any executive that tries to lie about these. This provides some level of 'transparent outcomes' with regulatory control. The counter-vailing forces are the executives versus the regulators.

In a market with many suppliers, customers vote with their feet - they dump suppliers who's cost, quality or service are poor.
The counter-vailing forces are the supplier management versus the customer choice. The customer is free to withdraw from or got to a particular supplier. The management can manipulate the internals of the company to maximise performance against their goals in the context of market preferences and constraints.

In a business monopoly, customers are not free to choose another supplier. The external feedback system, the lose customers actuator, is removed. A commercial monopoly can do what it likes with impunity - until some natural limiter operates or another effect, such as government intervention, occurs.

But there are very large Private Sector monopolies:
- Police
- Military
- (public) Health
- (public) Education, primary, secondary, tertiary
- Local Councils
- 'Secret Squirrel' agencies - like Military Intelligence and other "National Interest Classified" organisations.
- Government bureaucrasies

These organisations are notionally audited ('opposed') by Ombudsmen and Auditors.

But the one essential management output is never reviewed independently - managerial decisions.

All these organisations tend to become overtly corrupt - money or power & influence - or covertly - lassitude, perverse behaviours, institutionalised bullying, harrasment and discrimination, structural inefficiences and ineffectiveness, and completely ignoring or disrepecting those whose service they are in (their ultimate employers) - the public.

For secret organisations, the lack of transparent outputs is even worse. The 'independent' reviewers are impossible to find - only people from within the closed community (they are 'cleared') are allowed to view the evidence and ask questions. A free-floating monoculture very different from the enclosing Society and the organisations commission, can and will develop.

Summary

Monopolies are bad.
Unrestrainted monopolies are worse - eventually becoming a Law unto Themselves - with either over or covert corruption.
Private Sector monopolies are the worst. There are no effective mechanisims to control them.

Effective, frequent independent external reviews of both managerial decisions and 'Corporate Culture' [the behaviour, values, morals and ethics shown, not espoused] are essential to prevent 'free floating monocultures'.

Ideally, three counter-vailing forces are needed, in case any one becomes too strong or one of the 'checkers' becomes too weak.

Then you are left with the problem of: "Who watches the Watchers?"
Us, via the media, transparent/open government and 'free speech'.

Memory Tags and JVM's

2007-03-16T17:42:00.000-07:00

Came up with this talking to my mate Stanley over a coffee...

Why do Java Virtual Machines have a hard time finding/locating pointers? They need to do this as a precursor to doing a "Garbage Collection". At CMGA conference I talked to an admin from a bank (St George?) who said SUN talked about these "Stop The World" events. With more memory, they take longer - e.g. 2-5 seconds of clock time. And nothing can happen in the JVM while garbage collection is going on... The events may not be able to be scheduled, and their occurrence will be unpredictable.

One of the good points is that more CPU's get the job done faster.
Each n-byte block of memory is read to see if it couldbe a pointer, i.e. has the value in the range of the heap addresses. Then that block has to be examined to see if it's an object, and does it contain pointers, and so on. They are looking for objects that are no longer used and can be returned to the 'free pool'... It sounds a very hard way to get things done.

The Burroughs B5000 used Memory Tagsand descriptorsto describe to the hardware the contents/use of each 'word'. Descriptors allowed real Virtual Memory around a decade ahead of IBM 360/370.

And memory 'tags' eliminated a whole class of problems. The 'Mark Stack' tag prevented stack manipulation [negative stack offsets weren't allowed either] and accidental corruptions. A whole class of security problem and coding faults were avoided.

So my question: JVM's are in complete control of their environment. They know when an object is created. Why don't they 'tag' each object and pointers to objects? This makes the garbage problem simple. You know the location of all objects and pointers. I guess it could be extended to a 'reference count', but I don't know enough about implementing these things...

Won't it take memory space?
yes - this is a space/time tradeoff.

Because we don't have hardware memory tags, then we have to do it for ourselves...
Like the 'free block list' in some file systems, a bit-map of each "object sized" memory region need be kept. I'm thinking that objects can't be less than 64-bits and they'd be allocated on those boundaries.

And the same for pointers - when an object pointer is allocated/created, set the tag for it.

No idea if this has already been tried... Just seems like A Good Idea :-)

Test Scan

2007-03-01T13:18:00.000-08:00

This is a page scanned from my notebook today.
The image should be on the left. The text on the right. (default)

I orginally uploaded the image to another blog and have copied the HTML here.

They are stored on PIXCA I think.
Have seen mention of 300Mb total.

Some limits on size of individual image to upload. in the 50Kb or Mb range.

Finally - a working expandable summary, but not happy

2007-02-27T18:04:00.000-08:00

Not happy with blogspot "help" :-(

In a previous post I wrote of my frustrating experience wasting many hours of my time following (precisely) the instructions on blogspot "help"....

Last night I wasted another 4-5 hours and got something working.

It's really simple - blogspot just can't do it. And there's a post on the Blogger forums that says this...
The help instructions are confusing - they run together instructions for two different types of templates (old and 'layout'), without letting you know how to tell which you've got, nor that you have to 'expand widgets' and with an error in the so called working code... And when you finally get to the end of their example so it parses - it can't be made to work. There is a bug that's been reported to them and they're not fixing...

The really annonying part of the help page - that you cannot send them a message! I'd like to wring their necks... They ask "was this helpful" ? Doesn't matter how ofter you hit 'No', you've got no way to tell 'em what's going on.

Finally found a Blogger forum post that pointed to 'hackoshpere' - in amongst a whole slew of people having exactly the same problems as myself over 6-9 months.

What I don't like about the solution:
It relies on a javascript file pulled of someone elses site. It's a big security hole...

And Google with all their resources, could easily fix this.
I wonder what their business objectives are with 'blogspot'??

NOTE: These URL's are not links. You have to copy and paste them to use them.
That's so you know where they go :-)

Javascript:
http://www.anniyalogam.com/widgets/hackosphere.js

Hackosphere notes:
http://hackosphere.blogspot.com/2006/09/expandable-posts-with-peekaboo-view.html
http://www.anniyalogam.com/widgets/peekabooposts.html

RTFM!! or Read Blogger Help :-(

2007-01-09T16:13:00.000-08:00

There's a compatibility matrix in Blogger Help for browsers...

Unfortunately, 'Safari', doesn't do a bunch of useful stuff. Like support the wysiwyg editor.

Firefox does. Fired up the version I had (1.5.6?), loaded the update (2.0.1).
Installed and running... With the wysiwyg editor :-)

Have to check whether the auto-install replaces the '.app' in Applications.

Adding summary/full article to a blogspot

2007-01-09T14:17:00.000-08:00

Saw this on a blogspot blog and thought I should do it as well :-)

Looked up Blogger Help - and there seemed to be two mechanisims.
Then there is the 'Classic' vs. 'widget' (? - anyway the _new_ format).

Banged on the format, couldn't find the 'style' tags they talk about.
Couldn't see where to modify the 'post template' they refer to.
Couldn't see where to add the 'span type=fulldisplay' tags either.

Didn't break the CSS :-)

Now, where do I go??

So I bought a Palm...

2007-01-08T14:36:00.000-08:00

I've had a PSION for maybe a decade. Started with a Series 3 and upgraded to a Series 5 (B/W).
The '5' uses batteries faster than I'd like. It's got lots of great things standard - including external buttons to use as a voice 'note taker', standard "CF" (Compact Flash) cards, sound (play & record), infrared and a touch screen. For most people the price (~A$1,000) was *not* right.

As a 'consultant' for a company, I ran my life from the Psion. It allowed me to separate my work and personal lives, capture critical information (like billing times, tasks, notes) and was pretty much small enough to go with me everywhere. I never did it working with my Nokia 7110. Shoulda, but the replacement handset had old firmware and I baulked at shelling out ~A$100 for the upgrade. I'd rigged up a battery pack for an old land-line modem (Maestro 28.8kbps) and with a few adaptors, actually *did* check my e-mail once or twice when travelling...

My data was well organised and I knew where things were. But I'd stopped using the '5' - it churns 'AA' batteries compared to the '3'
Psion provides an actual operating system - so you have multiple programs (and different instances of them) open at the one time - and you can just put the thing to sleep. I have a standard set of programs open - it's quick to zip around and do regular things. On the '3' I pretty much wore the writing off the 'command' key used for most keyboard shortcuts.

And you need MS-Windows to run the Nokia phone suite *or* the Psion.
Having a serial interface, it's possible to backup & upload/download 'stuff' to the Psion - *iff* you have a serial or IR interface on your box.

This year I moved my working desktop to Mac OS/X (a mac-mini, so I can share KBD/Screen with other computers). I bought an IR-USB interface. Works like a charm on Windows :-) Mac doesn't seem to understand serial over USB. Google-ing didn't show an answer :-(

Early 2006 I started some projects at home, including formal study at the ANU (university). After realising that I don't have the personality/character structure to be an accountant, I found I missed *two* important deadlines. My organisation/processes failed me...

Never happened with the Psion - time to get a new PDA. Rechargable, small & light, 802.11.
Couldn't get another Psion, no longer produced. But the code lives on in Symbian.

Looked at PDA-phones - permium pricing, compromise user interface and poor keyboard.

Wasn't interested in a Windows-CE device. Just don't trust MSFT code to absolutely not let me down.

So it was a Palm...

Generally happy with it - but.

Doesn't provide a real O/S, but 'fast start' applications - and just *one* is running at a time.
The 802.11 is sorta integrated into the applicaitons, but not uniformly.
Bluetook 'works' with the Mac - but is so much slower (10 or more times) than USB, it's painful.

Bought a fold-out keyboard - bit bulky & uses batteries. Billed as 'universal wireless keyboard' - sorta. It's infrared, and you need to download the driver to the Palm, and if your mobile phone has no driver, tough... Have used it to sit and write notes/document. Not entirely a dud :-) Not nearly as effective as the Psion, but close enough.

All the address lists and calendar docs from the Psion - tried to download them...
Mixed success. The Psion program exports in CSV format. It's seen by the Palm Desktop on Mac as Excel files - and the Palm itself says they are Excel, but barfs when opening one. From memory, it requires a hard-reset to recover.

Too many times for my liking, I've found myself being forced to do hard resets...

'palmsource' (?) have announced a Linux for Palms - and it seems to be opensource. No idea on how to convert my T|X :-(
My major concern is giving up the 'graffiti' touch screen input. And 'bricking' it.

Blogging interfaces

2006-06-21T09:30:00.000-07:00

Google/Blogger have good doco on-line:
http://code.blogger.com/archives/atom-docs.html

And there are PERL modules on CPAN.

O'Reilly has a discourse: http://www.xml.com/pub/a/2003/10/15/dive.html
[and many others]

I'd like to find some free application to edit the XML or post the blog locally.
O'Reilly have a thing that publishes to "iDisk" on .MAC - not blogger and atom.

Here's some command line posting stuff. Uses "curl".

What BLOGS do I have?
Auth='stevej_098:password'
curl -u "$Auth" -o blog.1 https://www.blogger.com/atom

Which will give for each blog:
<link href="https://www.blogger.com/atom/29583699" rel="service.post" title="stevej-in-oz" type="application/atom+xml"/>
<link href="https://www.blogger.com/atom/29583699" rel="service.feed" title="stevej-in-oz" type="application/atom+xml"/>
<link href="http://stevej-in-oz.blogspot.com" rel="alternate" title="stevej-in-oz" type="text/html"/>

Download a blog
Get the "service.post" tag.
curl -u "$Auth" -o blog.2 https://www.blogger.com/atom/29583699

Post an entry
The hard bit here is the formating of the XML..

curl -u "$Auth" -o blog.4 -H "Content-type: application/xml" -d @blog-post.1 https://www.blogger.com/atom/29583699

some useful searches
grep service.feed blog.1
grep service.edit blog.1
grep service.post blog.[12]
grep service.delete blog.[12]

The Data Dungeon

2006-06-17T20:24:00.000-07:00

This idea came from reading "I, Cringely" of 17 Nov 2005. Cringely says Google is creating a "datacenter in a box" - shipping container, really - "5000 Opteron processors and 3.5 petabytes of disk". That's pretty impressive.

People have been buying servers and building datacenters for years - why should this be exciting? Because it has the potential to lower the cost of the whole datacenter radically - without doing any calculations, 5 or 10 times.

I thought how I'd do it - no fans (they break) means liquid cooling, no UPS - direct DC, single A/C unit, no walk spaces, no cases needed for equipment unless it's for cooling, RFI or containment. OH&S doesn't occur when it's working - it's sealed.

And you throw away the key. It's the next logical move from "lights out" or "dark datacenters"...
[You may even weld the doors shut.]

A reasonable technology and physical life, without maintenance, would be 3-4 years.

And people like SUN, Dell, IBM and HP could make these things - either sell or lease. And being the owners, could

Google have a very special workload - it scales linearly and generally is short transactions that can be rerun.

Normal commercial workloads are things like:
- webserver (Transaction based, restartable, load-balancer friendly)
- database (long-connection time, persistent, need clustering for high-reliability/availability)
- filer/file server (more like a database)
- email - client or server. Both need reliable data storage, but can take restarts.
- and probably way more...

The whole point of a "Data Dungeon" is replicating at the systems level, not the component level...
You don't need hot-swap power supplies, dual-NICs yadda-yadda-yadda if you have two complete systems that hot-swao.
Commodity hardware is *cheap*. You have to be inventive with your software/systems to design around break-able parts.

And all parts don't have to be the same - you'd want some really low-power fanless CPU's for some types of service, and enough top-end high-power CPU's in the mix for those times when too much grunt is not enough...
It's not going to be a box full of just the one thing...

So a "Data Dungeon" - would you ever just have ONE? Nope - the breakable design dictates at least two... Which you can stack in a car-space out the back [shipping containers, rememer?]. And when it's time to upgrade, wheel in another one or two, mirror the data, migrate the persistent processes and take away the old ones - all done live in prime-time...

Part of the scheme is running everything in Virtual Machines: Only one service to a virtual machine (ebserver, email, DB, ...)
It's easy to migragte a service onto a different physical processor - if you have load or servicability problems.
[VMware have some neat new Enterprise tools to do this now.]

And with Mac on Intel, running VM's means you get to run all the major commercial apps:
- all flavours of Windows deskop - via VNC or Citrix remote client to host legacy Apps.
- Windows server
- Mac OS/X
- z/OS [IBM mainframe]
- Solaris, BSD, and Linux [for those who need a Unix]

The Challenge

2006-06-17T20:16:00.000-07:00

From: NeilG
Subject: The Data Dungeon - Notes from the Lab

As a side-effect of our conversation last night, I mentioned you should start blogging this idea ("throw away the key data center") as a way of promoting it. This is my reminder.

I'm pretty sure you said it wasn't a patent item, but you weren't sure how to convince people it was a good idea. Blog it! :)

In that same vein, I was trying to think how to differentiate your blog. Most blogs are just text, and very unappealing visually (to me). I think a paradigm that would work well for you is, a "lab notes" format. We were talking about keeping a notebook as part of the chem and eng and comp sci disciplines. Take that same format (you already have the base text) and blog it with visuals.
E.g., This link

Personally, I think it's cool if you can look inside someone's mind as ideas are being wrought and I think a lot of geeks would agree. That's also the antithesis of the western philosophical tradition, and a good match for your style.