Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Slashdot Posting Bug Infuriates Haggard Admins

CmdrTaco posted more than 7 years ago | from the this-is-never-good dept.

262

Last night we crossed over 16,777,216 comments in the database. The wise amongst you might note that this number is 2^24, or in MySQLese an unsigned mediumint. Unfortunately, like 5 years ago we changed our primary keys in the comment table to unsigned int (32 bits, or 4.1 billion) but neglected to change the index that handles parents. We're awesome! Fixing is a simple ALTER TABLE statement... but on a table that is 16 million rows long, our system will take 3+ hours to do it, during which time there can be no posting. So today, we're disabling threading and will enable it again later tonight. Sorry for the inconvenience. We shall flog ourselves appropriately. Update: 11/10 12:52 GMT by J : It's fixed.

cancel ×

262 comments

Only one thing to say (5, Funny)

bernywork (57298) | more than 7 years ago | (#16786073)

*Clap clap clap*

Sorry, could not resist. (2, Insightful)

Noryungi (70322) | more than 7 years ago | (#16786075)

we shall flog ourselves appropriately


Please do.

So does a first post ... (5, Funny)

AppHack (622902) | more than 7 years ago | (#16786079)

... roll over to be a last post?

D'oh. (5, Insightful)

NeuralAbyss (12335) | more than 7 years ago | (#16786089)

Anyone could have made the mistake.. good to keep us all in the loop though :)

And let this be a reminder to the kids - RTFM, twice!

Disable posting!!11oneone (1)

vogon jeltz (257131) | more than 7 years ago | (#16786091)

Oh, ...... never mind.

fantastic (1)

kook44 (937545) | more than 7 years ago | (#16786093)

d'oh

I for one (5, Funny)

MrBulwark (862510) | more than 7 years ago | (#16786097)

welcome our 2 to the power of X overlords.

Last post! (5, Funny)

buro9 (633210) | more than 7 years ago | (#16786103)

Last post!

who we should REALLY blame (5, Funny)

admdrew (782761) | more than 7 years ago | (#16786117)

Alright, who's the joker who posted the 16,777,216th comment?

Thanks for breaking slashdot, jerk :D

Nice (0)

Anonymous Coward | more than 7 years ago | (#16786119)

Great work.

Oh Noes! (5, Funny)

The Mysterious X (903554) | more than 7 years ago | (#16786125)

Its like y2k, only worse!

Oops (2, Funny)

Irish-DnB (161087) | more than 7 years ago | (#16786129)

As Nelson Muntz would say "HA HA"

Congrats taco (5, Interesting)

LiquidCoooled (634315) | more than 7 years ago | (#16786133)

Does this mean that comment id#16777215 has the longest thread in history?

Can anyone actually find it to see - I tried but could only get to 16777217 [slashdot.org] , its likely to be in a journal or just a reply to an older article.

I sense a disturbance in the force (5, Funny)

Anonymous Coward | more than 7 years ago | (#16786141)

As if a thousand geeks all made the same damn "last post!" joke at once. . . . . .

Better make it longer (3, Funny)

The_Wilschon (782534) | more than 7 years ago | (#16786147)

I mean, look how quick we got to 16M comments. 4.1 Gigacomments will come in hardly any time at all. I predict we'll be doing all this again in merely a few weeks!

So who's the killer? (5, Insightful)

Rob T Firefly (844560) | more than 7 years ago | (#16786159)

I wonder who posted comment #16777216. That person should win some sort of "I borked Slashdot!" award.

Re: Oh Noes! (4, Funny)

PFI_Optix (936301) | more than 7 years ago | (#16786169)

"it's like y2k but worse"

I know what you mean. Y2K was supposed to put an end to civilization, but at least we'd have been able to post on slashdot.

seems strange (5, Funny)

jihadi_schwartz (989888) | more than 7 years ago | (#16786171)

...why wasn't this problem discovered on the dev system in advance?

They should have used PostgreSQL (-1, Troll)

Anonymous Coward | more than 7 years ago | (#16786173)

Subject says it all. PG does a much better job giving you so much rope to hang yourself with.

Dupe! (0, Insightful)

Anonymous Coward | more than 7 years ago | (#16786175)

Given the amount of duplicate articles, why not just let it overflow and we can roll the same stories around again. Saves all the hassle of making submissions.

GSM phones to monitor traffic problems, anyone ?

It's true that... (5, Funny)

0racle (667029) | more than 7 years ago | (#16786183)

Anyone could have made the mistake
But it's so much funnier when that anyone is not you.

Point 2.4 of Resign Patterns: Detonator (2, Informative)

Smoking (24594) | more than 7 years ago | (#16786185)

Taken from http://franksworld.com/blog/archive/2005/01/04/600 .aspx

Chapter 2: Destructional Patterns

2.4 Detonator

The Detonator is extremely common, but often undetected. A common
example is the calculations based on a 2 digit year field. This bomb
is out there, and waiting to explode!

TubgirL (-1, Offtopic)

Anonymous Coward | more than 7 years ago | (#16786193)

Check out... (3, Interesting)

BJH (11355) | more than 7 years ago | (#16786203)

...comment 16777215 [slashdot.org] .
Mmmm... CT, are you sure the parent index was your only problem?

Would PostgreSQL be faster? (-1, Troll)

Anonymous Coward | more than 7 years ago | (#16786213)

Would PostgreSQL have performed this operation any faster than MySQL?

2^24 (5, Funny)

Anonymous Coward | more than 7 years ago | (#16786219)

"2^24 comments ought to be enough for anyone" -- CmdrTaco

Comment 16,777,216 does not exist (5, Informative)

jamie (78724) | more than 7 years ago | (#16786231)

Some of you are asking which comment it was that got the cid 16,777,216. The answer is that none did. For redundancy, Slashdot is now running multiple-master replication which skips values for auto-increment [mysql.com] . Our db-1 assigns odd-numbered primary key IDs, and db-2 assigns even-numbered. Right now writes are going to db-1 so newly created rows will have only odd IDs.

The comment that got 2**24-1 was this one [slashdot.org] , if anyone cares :)

Sorry about the inconvenience, everyone.

Digg? (5, Funny)

Afrosheen (42464) | more than 7 years ago | (#16786235)

That's cool, I'll just pretend I'm on Digg, with its 1981 Commodore 64 BBS-style threading.

  Wait..sorry Commodore fans. I know it had better threading than Digg.

OMG!!!! (1)

toonworld (838479) | more than 7 years ago | (#16786241)

Posting comments is my entire life!! What will I do?? :-(

Haggard? (5, Funny)

Anonymous Coward | more than 7 years ago | (#16786249)

Haggard admins? Does this mean that the Admins will go buy some meth and get a massage?

My Reply to the Funny Comment Above This (4, Insightful)

eldavojohn (898314) | more than 7 years ago | (#16786251)

Uh, this is a reply to the 8th post down from the top (remember to use this like an array and zero reference). Yes, I'm talking to you, admdrew.

You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced, and it would actually be the one after that one that would break it. You laugh but these kinds of problems plague a lot of coders?

If you don't agree with me, please respond below and reference my comment ID.

What will i do?? (0)

Anonymous Coward | more than 7 years ago | (#16786255)

If I can't post comments??

16 million posts ... (5, Funny)

DrJimbo (594231) | more than 7 years ago | (#16786273)

... should have been enough for anyone.

Holy Chit! (5, Funny)

RedCard (302122) | more than 7 years ago | (#16786289)

No threading? Welcome to Farkdot.

Old discussions (1)

DeadCatX2 (950953) | more than 7 years ago | (#16786293)

So let me guess...the parent index has been lost forever?

Hopefully it just rolled over, and you can add 2^24 to every value to get the parent links back.

Though I somehow doubt that will work.

Feature missing in MySql (1)

cucucu (953756) | more than 7 years ago | (#16786301)

to filter out comments by mod

Re:OMG (2, Funny)

jZnat (793348) | more than 7 years ago | (#16786305)

Posting comments is my entire life!! What will I do?? :-(
I know how you feel; reading them is my life, too.

we've come a long way since VAXclusters... (1, Funny)

Anonymous Coward | more than 7 years ago | (#16786309)

...and the philosophy of zero downtime.

Oh wait, we haven't.

But at least we can use processors several orders of magnitude faster to produce sufficient eye candy to catch what little attention span is left in today's users.

Psssh! (-1, Redundant)

Anonymous Coward | more than 7 years ago | (#16786321)

16,777,216 comments ought to be enough for any forum.

And? (5, Funny)

Lars T. (470328) | more than 7 years ago | (#16786325)

Sorry for the inconvenience. We shall flog ourselves appropriately.
And post the YouTube link?

Can't count. (-1, Troll)

Anonymous Coward | more than 7 years ago | (#16786329)


> Last night we crossed over 16,777,216 comments [snip] > .. but on a table that is 16 million rows long

That's nearer 17 million YTC

Disclosure (0)

diersing (679767) | more than 7 years ago | (#16786341)

Its a conspiracy, the truth is Natalie Portman poured hot grits on the beowulf /. cluster. In Soviet Russia, all your threads belong to our table altering overlords.

Ok.. (1)

Awod (956596) | more than 7 years ago | (#16786357)

I admit it, it was me.. all your slashdot are belong to me.

It's Like Digg without the nesting. Where's the (0)

Anonymous Coward | more than 7 years ago | (#16786363)

thumb?

We just have to use the @dimwit syntax.

It's a joke, kids (5, Funny)

Temuar Skylari (1008449) | more than 7 years ago | (#16786369)

Dupe! I TOTALLY posted this story like, last WEEK man! (I laugh, but I betcha someone might post this in seriousness)

Arbitrary Limits (0)

RAMMS+EIN (578166) | more than 7 years ago | (#16786377)

And this is why you should not have arbitrary limits in your programs, ladies and gentlemen. Not even limits on the values your numbers can represent - unless you have _proven_ that no values outside the representable range will ever occur.

Did somebody say... (4, Funny)

camusflage (65105) | more than 7 years ago | (#16786381)

Flogging and Haggard in the same sentence? If we can get "crystal meth" in, we'll hit the trifecta!

Sounds Familiar (5, Informative)

old_skul (566766) | more than 7 years ago | (#16786385)

I used to work at Comair. Remember, that airline that stranded about 10,000 people in the airport a couple of Christmases ago? Same deal. Program was capable of handling only a certain number of changes. Hopefully your president won't have to resign.

Gay meth orgies (1)

pHatidic (163975) | more than 7 years ago | (#16786387)

Did anyone else read the headline and think that's what it was about?

So was it fixed in Slash? (4, Interesting)

tlhIngan (30335) | more than 7 years ago | (#16786391)

So is the bug still in the CVS revision of Slash, or was it fixed 5 years ago and Slashdot never applied the patch?

Re: Can't count (1)

LiquidCoooled (634315) | more than 7 years ago | (#16786393)

Its not a problem with counting, its the fact he used the lower case c for comments.

16777216 comments = 16 million rows.
16777216 Comments = nearly 17million rows.

Also, he could have used mibicomments to get the proper units and would have totally avoided this confusion.

Give Slashdot Subscriptions to the borkers (5, Interesting)

davidwr (791652) | more than 7 years ago | (#16786397)

Give a 2^0-year Slashdot subscription to the guy who hit the limit and one to the the first non-administrator guy who successfully posted after the fix.

If you can find the first guy who COULDN'T reply due to the limit, give him one too. He deserves something for his trouble.

EldavoJohn cid=16786251 reply (4, Informative)

DeadCatX2 (950953) | more than 7 years ago | (#16786399)

Actually, comment 16,777,217 couldn't break it, because that comment's parent cid could have only been 16,777,215. Up until then, there wouldn't have been an overflow value put into the db.

There's no telling which comment it is, because (16,777,217 + 2n) might not have been a reply, meaning it would come up correctly.

Fark? (0)

Anonymous Coward | more than 7 years ago | (#16786403)

Suddenly Slashdot seems more like Fark.com.

Dorks. (3, Funny)

morgan_greywolf (835522) | more than 7 years ago | (#16786419)

Does this mean that Slashdot is going to denegrate into Digg now?

Thanks (1)

arun_s (877518) | more than 7 years ago | (#16786427)

I'd like to extend a big thanks for letting us know, and also for explaining how the problem arose in the first place.
(I'm not smart enough to jump into slashcode and feel at home there, so its pretty interesting to see the kinda stuff you people have to face every now and then)

Reply to 16786251 (4, Informative)

RAMMS+EIN (578166) | more than 7 years ago | (#16786431)

Reply to comment number 16786251 [slashdot.org] :

``You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced'' ...but it's probably not zero-referenced. Typically, ids in SQL start at 1.

Re: Can't count. (1)

brunascle (994197) | more than 7 years ago | (#16786435)

comment id 16786329:
> Last night we crossed over 16,777,216 comments [snip] > .. but on a table that is 16 million rows long

That's nearer 17 million YTC
i'd be willing to bet it's much below 16 million actual comments, since the first few were probably tests, and then deleted. he means the comment with a primary key of 16777216; that doesnt necessarily mean there are that many comments in the table.

Today is the first day I've had to type subjects. (3, Interesting)

admdrew (782761) | more than 7 years ago | (#16786443)

@eldavojohn, #16786251 [slashdot.org] (god, this feels like digg now)
Uh, this is a reply to the 8th post down from the top (remember to use this like an array and zero reference). Yes, I'm talking to you, admdrew.

You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced, and it would actually be the one after that one that would break it. You laugh but these kinds of problems plague a lot of coders?

If you don't agree with me, please respond below and reference my comment ID.

I certainly admit I wasn't thinking 0-based when I wrote that. The question is, though, should we blame the person who wrote the last valid comment (therefore ruining the fun for the rest of us), or whoever wrote the first broken comment?

Also, is everyone going to add the obligatory 'parent' link on their posts today?


[ Parent [slashdot.org] ] - [ Reply to this [slashdot.org] ]

Like, Dude! (0)

Anonymous Coward | more than 7 years ago | (#16786445)

Unfortunately, like 5 years ago we changed our primary keys..
Who knew that CmdrTaco was 12 years old?

Access to the Database? (5, Interesting)

Shadow Wrought (586631) | more than 7 years ago | (#16786449)

Any thoughts on making the DB publicly accessable other than through teh Dot? Not sure what I'd do with all that data, but I'm sure these's a grad student somewhere who'd love the opportunity...

When I Saw "Haggard" In The Title... (0)

Anonymous Coward | more than 7 years ago | (#16786461)

I thought some Slashdot bug had taken down an evangelical website or something.....duh.

That's what you get... (-1, Troll)

NineNine (235196) | more than 7 years ago | (#16786473)

... for using MySQL. I hate to say it, but it's the truth. Remember kids: if MS Access can't handle what you're doing, MySQL probably can't handle it, either. I know that at least Oracle can re-index on the fly. Probably MS SQL, DB2, and even PostgreSQL, too.

Only on slashdot... (1)

beavt8r (919284) | more than 7 years ago | (#16786479)

would a story like this actually be posted. And with such hilarity. But really, it seems many times we computer people give us more room for future expansions and inevitably end up hitting it (y2k 'bug', this, 128K should be enough for everybody, that kinda thing). Yes i know, we hit limits eventually. But, you would think that we'd prepare better for this and increase beforehand. (the admins did as mentioned, just missed the other part) Or are we lazy? Or all have A.D.D.? Perhaps we just do this "OOhh...a new piece of hardware/software/whatever. Um....yeah, we can wait to raise that limit."

16777216 (1)

Little Brother (122447) | more than 7 years ago | (#16786487)

eldavojohn (898314)

(http://slashdot.org/~eldavojohn/journal/ | Last Journal: Tuesday April 04, @01:06PM) Uh, this is a reply to the 8th post down from the top (remember to use this like an array and zero reference). Yes, I'm talking to you, admdrew. You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced, and it would actually be the one after that one that would break it. You laugh but these kinds of problems plague a lot of coders? If you don't agree with me, please respond below and reference my comment ID.

Yes, but you forgot to start counting with 0

/Hey, this whole no threading thing reminds me of something.
//Obscure?

reply to comment 16786251 (1)

lazarusdishwasher (968525) | more than 7 years ago | (#16786493)

You seem to be thinking like I was when I first read about this problem. but I think I have a better way of addressing who we are replying to http://slashdot.org/comments.pl?sid=205731&cid=167 86251 [slashdot.org] hyperlinks are easier to follow than trying to count as the number of comments rise.

May i be the first to say... (5, Funny)

zepo1a (958353) | more than 7 years ago | (#16786501)

Brillant!

I always wondered where Paula Bean ended up...

So what if you delete a few? (1)

Alaria Phrozen (975601) | more than 7 years ago | (#16786503)

Why not delete all the posts that have the words: insensitive clod, I for one welcome our [a-z]+ overlords, Soviet Russia, Libraries of Congress, ...profit!, frist psot, why is this on slashdot, dupe, or have any mention of porn.

You'll free up 2^24 - 2^15 of the comment IDs at least...

RE: 16777217 GET (5, Funny)

Stalyn (662) | more than 7 years ago | (#16786505)

mod parent up

Re: Slashdot Posting Bug Infuriates Haggard Admin (1)

JymmyZ (655273) | more than 7 years ago | (#16786525)

Mod parent down, funny my ass. It's obviously flamebait

Mod parent up (4, Funny)

foniksonik (573572) | more than 7 years ago | (#16786527)

Uhhhh who's your daddy?

Taco is an illiterate blowjob merchant (0, Troll)

WisC (963341) | more than 7 years ago | (#16786535)

Hey taco, the slashdot diet of crack and male cum must really be getting to you, it is spelt haggered you dumb illiterate fuck. How can you claim to be news for nerds when you can't even spell properly, spit out that dick and use spell check you lazy fuck. Oh hang on it is spelt right, oh well the above still applies anyway

Old articles? (1)

TheThiefMaster (992038) | more than 7 years ago | (#16786537)

Does this mean that all replies to comments after the 2^24-1th one were attached to much much older comments instead? Can anyone find one and see?

Perhaps some sql command that adds 2^24 to a comment's parent entry if the comment's own id is >2^24 and it's parent is less than 10,000?

Why are all 16 million+ comments in a single table (4, Interesting)

poot_rootbeer (188613) | more than 7 years ago | (#16786543)

Slashdot being a news (for nerds) site, I would expect that the usage patterns are such that a huge majority of the content accessed by users is very recent -- say, perhaps, 90% of the database hits are for stories and comments that were posted in the last week.

So why, pray, is this usage pattern not accounted for in the database design?

MOD parent up! (4, Funny)

bazorg (911295) | more than 7 years ago | (#16786545)

Mod parent ... wait..

I think you mean... (2, Funny)

rjstegbauer (845926) | more than 7 years ago | (#16786557)

LOST Post!

How is this possible? (1)

Stonent1 (594886) | more than 7 years ago | (#16786561)

I thought you guys were like Gods or something. Maybe I should start praying to someone else instead now?

Re:reply to comment 16786251 (1)

Thansal (999464) | more than 7 years ago | (#16786567)

> #16786493 [slashdot.org]

Exactly, just link to their comment, and if you want you can even follow the standard "Re:" format if you want :D

This was fortold a few months ago... (4, Informative)

Anonymous Coward | more than 7 years ago | (#16786577)

http://slashdot.org/~themusicgod1/journal/137880 [slashdot.org] ...ok, so it was obvious...

Should have used PostgreSQL instead.. (2, Insightful)

eamacnaghten (695001) | more than 7 years ago | (#16786583)

Would not have happened if Slashdot used PostgreSQL.

Let the flamewars begin...

No post but mod!! (0)

Anonymous Coward | more than 7 years ago | (#16786593)

So I got mod points instead of being able to post....

Digg (0, Offtopic)

franksands (938435) | more than 7 years ago | (#16786595)

I'm sorry to inform you, but Digg has nested posting for a while now.

Take all the time you need (5, Funny)

deaton (616663) | more than 7 years ago | (#16786597)

Take all the time you need, I'm more than willing to refrain from posting durin.... Oh shit!

Trolls (2, Funny)

Jaseoldboss (650728) | more than 7 years ago | (#16786609)

At least nobody can feed the trolls now!

Slashed Eyeballs (3, Insightful)

Doc Ruby (173196) | more than 7 years ago | (#16786611)

If Slashdot released the Slashcode more frequently, with more/better comments/docs, and encouraged some of the many of us who complain about bugs/features to help the project, then it's more likely that someone would have debugged this bug earlier.

Open source - it's not just a buzzword, it's a way of life.

What's impressive (0, Troll)

FirmWarez (645119) | more than 7 years ago | (#16786623)

Is how quick you figured out what the problem was. If this were some huge corporate entity, the PHBs would still be running around trying to set up meetings upon meetings...I think the answer to all technical problems is smart people with little to no be bureaucracy.

To: Donald Rumsfeld +1, Patriotic (-1, Offtopic)

Anonymous Coward | more than 7 years ago | (#16786627)


Go fuck yourself and Osama bin Laden's employer [whitehouse.org] .

Patriotically as always,
Kilgore Trout.

Reply to 2569161991 (0)

Anonymous Coward | more than 7 years ago | (#16786649)

You creep up to the window and, in the soft, muted lights, you see a tall woman with long, blond hair. She sits before a mirror and brushes her hair, then stands and walks over to the sunken tub off to her left. She kneels and her blue, silken robe drops to the floor. She turns the water on and steam slowly fills the air.
You watch in fascination as she reaches down into the tub, whirls, and points an Uzi in your direction. "Stop reading paragraphs you're not supposed to read, creeps." She sighs deeply. "Next time I'm going to demand they put me in a Bard's Tale game, this Wasteland duty is dangerous."

A real Slashdotter! (4, Funny)

QuickFox (311231) | more than 7 years ago | (#16786661)

And this is why you should not have arbitrary limits in your programs, ladies and gentlemen. Not even limits on the values your numbers can represent

Now this is a real Slashdotter! This guy knows how to build an infinite computer!

Re:Old discussions (1)

Control Group (105494) | more than 7 years ago | (#16786671)

So let me guess...the parent index has been lost forever?

Hopefully it just rolled over, and you can add 2^24 to every value to get the parent links back.

Though I somehow doubt that will work.


Or they could just use a DB backup. Does MySQL support point-in-time recovery?

Parent [slashdot.org]

Re: Old discussions (1)

Control Group (105494) | more than 7 years ago | (#16786699)

So let me guess...the parent index has been lost forever?

Hopefully it just rolled over, and you can add 2^24 to every value to get the parent links back.

Though I somehow doubt that will work.


Or they could just use a DB backup. Does MySQL support point-in-time recovery?

Parent [slashdot.org]

Let teh flogging commence... (0)

Anonymous Coward | more than 7 years ago | (#16786751)

... and by the way, you SUCK.

LMAO, my vword: "cheeks"

We're awesome! (0)

Anonymous Coward | more than 7 years ago | (#16786767)

We're awesome!

No, "Taco", you're just stupid. But we already knew that.

Misleading Title! (1)

bshroyer (21524) | more than 7 years ago | (#16786789)

I had a hard time imagining how the admins of Ted Haggard's site [tedhaggard.com] could be miffed at Slashdot.

I also didn't recollect Slashdot discussing our favorite non-homosexual, non-meth-using ex-pastor recently...

24 bit? (4, Insightful)

voidptr (609) | more than 7 years ago | (#16786795)

Why on earth does MySQL have a 24 bit integer datatype? On what platform does it even remotely make sense to use that in the first place? It's going to get cast to 32 bits for any arithmetic operations anyway, and on most platforms today alignment requirements are going to pad the extra byte in memory and disk, so you're not even saving any space. Why even give someone the option over choosing between 16 bit and 32 bit integers?

Re: Why are all 16 million+ comments in a single t (5, Informative)

jamie (78724) | more than 7 years ago | (#16786801)

poot_rootbeer asks why all the comments are in one table, when the data access pattern is such that 90% of our hits are on only the most recent entries in that table.

The answer is that we used to do it this way but it's a huge pain. In 2000 we converted from having two tables for 'stories', recent and archived, and merged them together. The performance hit was not big, and it made the code so much simpler it was a no-brainer.

It's the database's job to cache properly whether we split the table or not, and the database does that just fine. The only performance problem could be when there is a rush of inserts, or updates to the same sets of rows, spanning both newer and older portions of the table, and that just doesn't happen.

If we did want to do this we wouldn't split the tables manually; the code complexity is too high a price to pay. In MySQL 5.0 we would use a MERGE [mysql.com] engine, which has issues of its own but would involve smaller changes to our code. That's still not worth it for us. What we're probably going to do is wait for MySQL 5.1 to get out of beta and then do some performance testing on tables partitioned [mysql.com] by date and see if that gains us anything. For example, a SELECT on our comments table could be limited with a WHERE clause to only retrieve rows with a date >= the discussion object's date, which for 90% of our queries MySQL 5.1 could optimize to only look at the most recent partition. If the gains turn out to be significant, then since partitioning involves very limited code changes, we'll probably do that. Generally speaking, though, database performance is not a problem for us. So far our main bottlenecks have been CPU and RAM on the webheads. As long as we don't do anything stupid our database performance has been fine, though, as today proves, we are quite capable of being stupid.

[ Parent [slashdot.org] ]

The wize (1)

Psychotic_Wrath (693928) | more than 7 years ago | (#16786803)

16,777,216 comments in the database. The wise amongst you might note that this number is 2^24 Only would a slashdotter look at that number and think OH thats 2^24 :D

Here's... (2)

BJH (11355) | more than 7 years ago | (#16786829)

....yet another non-existent comment numbered 16777215 [slashdot.org] . And another one [slashdot.org] . And another one [slashdot.org] .
Normally, accessing a non-existent comment gets you either the "nothing to see here [slashdot.org] " message or the "can't find that comment in this discussion [slashdot.org] " message. Where are the ghost comments coming from?

Re: Access to the database (2, Insightful)

Control Group (105494) | more than 7 years ago | (#16786841)

Any thoughts on making the DB publicly accessable other than through teh Dot? Not sure what I'd do with all that data, but I'm sure these's a grad student somewhere who'd love the opportunity...

Not just grad students; as a DBA by profession, I'd love a crack at the DB. If nothing else, it would give me a great place to play around with MySQL. Not to mention the ability to maybe extract some interesting user-level statistics.

Of course, the odds of this happening are pretty damn low - there'd have to be an awful lot of work and review done to scrub the DB of information that is entrusted to /. that people didn't plan on having released to the internet at large. Passwords, for example (even if they're stored only as hashes, getting the whole DB would make it feasible to crack them); real email addresses, real names...I assume that the subscription process doesn't involve actually storing credit card information in the DB (I don't know; I've only used PayPal), but that might be another concern.

Just the email addresses would be a huge deal - can you imagine the market value of such a targeted list of addresses?

In short, it would be fantastically cool for them to release the DB, but it would be a lot of work on their part for no particular return. Not to mention that if they released it once, they'd no doubt be pestered to keep releasing periodic updates...then there's the bandwidth issues...and, even, the potential copyright issues (/. doesn't own the copyright on posted comments, the poster does)...then the copyright issues for stuff they do own; releasing the DB would make it trivial for a bad actor to post a mirrored slash. A little bit of domain typosquatting and some ad deals, and you could be talking about real money.

If I were them, there's no way in hell I'd even think about doing it.

But it would be cool.

Parent [slashdot.org]
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...