APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Too much traffic

© February 2009 Anthony Lawrence

This website limps along serving up a little less than 200,000 pages a month. If you divide that up, it's a bit over 6,000 a day, which comes down to 4 or 5 pages per minute. That's hardly a lot to ask of any webserver.

Of course sometimes we get peaks. There's not much happening at 3:00 AM so obviously there's more happening at noon. Every now and then some article gets a little burst of popularity despite my best efforts to discourage such things. Those cause peaks too.

But lately we've been getting some plain old peaks. I mean that they don't come because a particular page has attracted some attention; they come just because a lot of people came at once. For example, we had one just a few minutes ago and it hit just about one page view per second. As each page also has to pull in other stuff, the load on the server was three times that and it started throwing errors:

"unable to include /cgi-bin/blah-blah.pl", "Premature end of script headers:", "Permission denied".. not good.

Well, you can sympathise. If that load kept up all day and all night, I'd be serving over 2 million pages per month and that might be a little too much to ask of a shared web server.

So what to do? Well, I could use mod_perl. That's annoying, but it's probably what I have to do. Unfortunately, my code is far from ready for mod_perl so that will take some time and effort.

I could create static pages. That would be even more annoying.

I could ask more of you to use RSS. Go ahead and do that. I don't think it will help that much.

I could switch to a more expensive server. I really don't want to do that either.

Any other suggestions?

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> Too much traffic


Inexpensive and informative Apple related e-books:

Are Your Bits Flipped?

Photos for Mac: A Take Control Crash Course

Take Control of Preview

Take Control of Upgrading to El Capitan

Take Control of IOS 11

More Articles by © Anthony Lawrence

Thu Feb 12 21:40:58 2009: 5397   NickBarron

How odd, I was just thinking about this yesterday.

How popular this site had become over the last year or so, even with your best efforts to discourage it ;)

Thu Feb 12 21:48:55 2009: 5398   TonyLawrence

Funny, isn't it: everybody else is begging for more RSS subscribers, asking people to vote up their posts on Digg, sucking up to anyone who might give them a Tweet.. and I want exactly the opposite: keep the unwashed masses away and I'll be happy..

Really, traffic is fine and I will have to deal with it. Mod_perl is the simplest way I guess.. I just have to clean up my sloppy programming :-)

Fri Feb 13 13:52:17 2009: 5401   MikeHostetler

If you didn't have so many things to say . . . .

mod_perl would work, but if your traffic keeps going up, you may have to deal with a "caching" solution that periodically generates static pages. This has become the accepted WordPress solution:

Fri Feb 13 13:57:24 2009: 5402   TonyLawrence

Yeah, after sleeping on it I started thinking that rewriting to make static pages would be easier than rewriting everything for mod_perl. I'll start working on that today.

Fri Feb 13 17:10:12 2009: 5408   TonyLawrence

OK, just finished up the code. I'm not going to convert everything at one - I'll test a few pages first to make sure I didn't bollix it.. so I'll test with few high volume pages and then roll it out..


Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

Printer Friendly Version

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming. (Donald Knuth)

Linux posts

Troubleshooting posts

This post tagged:



Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode

SCO Unix Sales, Support, & Service

Phone:  707-SCO-UNIX (707-726-8649Toll Free: 833-SCO-UNIX (833-726-8649)