APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Sloppy data presentation

© September 2006 Anthony Lawrence

I have a client for whom I massage U.S. government supplied weather data. There is a bewildering amount of data available; I haven't a clue what most of it means: I just go get what they want and slice and dice it to their specs.

The folks who produce this data of course think nothing of changing locations, formats, or anything else. That's not unexpected and there's no point griping about it - it is what it is.

What does cause me to make unpleasant faces is sloppiness in data representation. For example, here's a data line from something I was working on yesterday:

DUAN6 200609191320 65.4 0 -996 9 42.03 0 0.00 0 -996.0 9 -996 9 -996.00 9 -996.00 9 -996 0 -996 9 -996 9

It looks like space separated values, easily parsed in Perl with "split /\s+/", but is it? Here's another sample from the same data source.

ABGV1 200609221150 -999.0 1 -996 1 -996 1 0.00-9999 1-996-9999 1-996-9999 1 -996 1 -996 1 -996 1 -996 1 -996 1

What the heck is that? Values are run together. Let's align the two lines to make it easier to see:

DUAN6 200609191320 65.4 0 -996 9 42.03 0 0.00 0 -996.0 9 -996 9 -996.00 9 -996.00 9 -996 0 -996 9 -996 9
ABGV1 200609221150 -999.0 1 -996 1 -996 1 0.00-9999 1-996-9999 1-996-9999 1 -996 1 -996 1 -996 1 -996 1 -996 1

Hmmm. So it's not fixed length.. but it's not space delimited either. It's just a mess.

This particular mess is easy enough to fix. The only time we get this overlap is with negative numbers, so:

s/-/ -/g;
@line=split /\s+/;

That works. It's annoying, but hardly a "big deal". It is however, in my opinion, real sloppiness in the original programming. If there was no care for being human readable, why space anything out at all? Why not just run it together in pure record form? DUAN6200609191320000065.4 etc.? Apparently there was some intent at presentation, so why not make ALL data lines consistent?

Yeah, I know: grumpy old Unix guys. Ayup, that's me.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> Sloppy data presentation

Inexpensive and informative Apple related e-books:

Take Control of Apple Mail, Third Edition

El Capitan: A Take Control Crash Course

Take Control of Pages

Take Control of the Mac Command Line with Terminal, Second Edition

Take Control of Numbers

More Articles by © Anthony Lawrence

Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

Printer Friendly Version

The camel has evolved to be relatively self-sufficient. (On the other hand, the camel has not evolved to smell good. Neither has Perl.) (Larry Wall)

Linux posts

Troubleshooting posts

This post tagged:



Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode

SCO Unix Sales, Support, & Service

Phone:  707-SCO-UNIX (707-726-8649Toll Free: 833-SCO-UNIX (833-726-8649)