Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Leap Second Bug Causes Crashes

samzenpus posted more than 2 years ago | from the slip-it-in-there dept.

Businesses 230

An anonymous reader writes in with a Wired story about the problems caused by the leap second last night. "Reddit, Mozilla, and possibly many other web outfits experienced brief technical problems on Saturday evening, when software underpinning their online operations choked on the “leap second” that was added to the world’s atomic clocks. On Saturday, at midnight Greenwich Mean Time, as June turned into July, the Earth’s official time keepers held their clocks back by a single second in order to keep them in sync with the planet’s daily rotation, and according to reports from across the web, some of the net’s fundamental software platforms — including the Linux operating system and the Java application platform — were unable to cope with the extra second."

Sorry! There are no comments related to the filter you selected.

All of my servers were fine (5, Insightful)

Anonymous Coward | more than 2 years ago | (#40512075)

And I didn't do anything special, just kept their software up-to-date.

Re:All of my servers were fine (4, Informative)

Sir_Sri (199544) | more than 2 years ago | (#40512085)

That can be hard for some people.

Re:All of my servers were fine (3, Informative)

Anonymous Coward | more than 2 years ago | (#40512105)

Agreed. Patches that aren't required to solve an ongoing incident impacting customer traffic require about 2 weeks advance notice to pass through change control, and that's if everything is perfect. A single error in a ticket can push that ticket out another week, and another, and so on.

Generally, we shoot for 3 weeks before we are allowed to install a patch. On average, it's about right.

Re:All of my servers were fine (5, Informative)

Anonymous Coward | more than 2 years ago | (#40512129)

the patch was posted back in March.

https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d

You probably don't do much Java, then (5, Informative)

burne (686114) | more than 2 years ago | (#40512301)

As it turns out my biggest problems was customer-supplied software which uses their own java jre's. We install a jre by default and update it whenever possible, but some software (Adeptia, VLTrader, Alfresco) comes with their own ancient jre and scripts to call that over system-supplied java.

Not a single machine crashed (we are very explicitly in charge of what OS-version there's running) but a lot of java locked up and had to be restarted.

I can even see a small bump in the power-usage around two o' clock (0:00 GMT).

Re:You probably don't do much Java, then (1)

Guy Harris (3803) | more than 2 years ago | (#40512793)

As it turns out my biggest problems was customer-supplied software which uses their own java jre's. We install a jre by default and update it whenever possible, but some software (Adeptia, VLTrader, Alfresco) comes with their own ancient jre and scripts to call that over system-supplied java.

Not a single machine crashed (we are very explicitly in charge of what OS-version there's running) but a lot of java locked up and had to be restarted.

So are you saying that, in addition to the Linux kernel glitch in question (which appears to cause some userland processes to spin [lkml.org] ), there are purely-userland problems? Or, if you're running on a Linux that doesn't have John Stultz's fix, is it that some JREs are vulnerable to the Linux kernel glitch and others aren't?

Re:All of my servers were fine (0, Troll)

Anonymous Coward | more than 2 years ago | (#40512551)

It's pretty sad that a protocol as well defined and well known as UT* times even requires to be patched in any kernel.

Re:All of my servers were fine (2)

0123456 (636235) | more than 2 years ago | (#40512313)

One of ours (running Java on Linux) started throwing out NTP alarms at 10 seconds after midnight, but it seems to have stayed up. However, the software on that particular system is especially vulnerable to leap second issues so we'd tested it pretty well beforehand.

Otherwise no-one has complained about any other systems going down so I presume they're OK.

Re:All of my servers were fine (-1)

Anonymous Coward | more than 2 years ago | (#40512703)

(running Java on Linux)

Well there's your problem right there. A double whammy of suck.

Re:All of my servers were fine (4, Insightful)

nmb3000 (741169) | more than 2 years ago | (#40512453)

And I didn't do anything special, just kept their software up-to-date.

That's a nice ideal, but the reality is that many up-to-date "stable" distribution releases are still using kernels which are susceptible the leap second problem (and haven't had the patch back-ported to them). Ubuntu 8.04 LTS server is supposed to be supported until April 2013, and on my (updated!) system,

# uname -r
2.6.24-28-server

I like the idea of stable releases, but this is a glaring problem with the entire idea. Everyone extolls the wondrous virtues of package managers for Linux-based systems, but the dirty secret is that unless you stay bleeding-edge (which is usually the opposite of "server"), you'd better be happy with the 4-year old version of Apache, PHP, MySQL, and the Linux kernel you're running. Sure, it's possible to manually download and install packages from a newer release (assuming you can get past the dependency hell usually associated with it). Sure, it's possible to try and splice in (or "pin" packages using Debian parlance) from a newer repository. Sure, it's possible to install from source, compiling and installing everything by hand. But once you do any of these you've given up 90% of what makes the package manager useful and are just asking for dependency problems in the future.

And, all that aside, do you even know if the patch released to fix this problem is included in your distribution-released kernel? If you're not rolling your own kernel it can be nigh to impossible to know what's included and what's not -- in that case it doesn't even matter if it's up-to-date.

Re:All of my servers were fine (3, Informative)

lister king of smeg (2481612) | more than 2 years ago | (#40512541)

And, all that aside, do you even know if the patch released to fix this problem is included in your distribution-released kernel? If you're not rolling your own kernel it can be nigh to impossible to know what's included and what's not -- in that case it doesn't even matter if it's up-to-date.

Well you could read through the change log and release notes to find out.

Re:All of my servers were fine (2)

RNLockwood (224353) | more than 2 years ago | (#40512747)

NASA's Astronomy Picture of the Day, http://antwrp.gsfc.nasa.gov/apod/astropix.html [nasa.gov] , has apparently been down all day; wonder if this is the cause.

Anyone heard from the Space Lab today?

Re:All of my servers were fine (3, Interesting)

thePowerOfGrayskull (905905) | more than 2 years ago | (#40512749)

Our problem was with a third party monitoring solution - its daemon process brought every single one of our servers to a near halt by consuming all available cpu cycles at the stroke of gmt midnight.

The OS itself was fine.
This monitoring software is common enough that it likely was behind a lot of the issues seen around the 'net.

Re: (2)

bleedingsamurai (2539410) | more than 2 years ago | (#40512087)

Interesting. I wonder what conditions had to have been met for a crash to happen, none of my servers had so much as a hick-up.

Re: (5, Funny)

Anonymous Coward | more than 2 years ago | (#40512101)

>hick-up.

The hick up watching the servers when the leap second came was you.

Re: (0)

Anonymous Coward | more than 2 years ago | (#40512113)

For starters, you need a kernel no more recent than 2.6.28, a kernel so old my Debian stable box is four revisions past it!

Re: (1)

bleedingsamurai (2539410) | more than 2 years ago | (#40512163)

Well that explains it. I'm running nothing less then 3.3.8

Re: (0)

Anonymous Coward | more than 2 years ago | (#40512189)

Not true.
I had a number of boxes running 2.6.32 getting bit by this bug.

Re: (2, Informative)

Anonymous Coward | more than 2 years ago | (#40512249)

The hard system lock bug due to a leap second was patched in 2.6.29, so either you've got some weird related bug, or something is very wrong.

Re: (0)

Anonymous Coward | more than 2 years ago | (#40512181)

From the looks of it the kernel must be running on multiple cpus for the livelock to occur. This is probably one of the reasons why none of my servers had any issue.

Re: (2)

Admiral Justin (628358) | more than 2 years ago | (#40512285)

Configuration of the system to only accept 23:59:59 and not 23:59:60

Linux (4, Informative)

Anonymous Coward | more than 2 years ago | (#40512097)

I'm a Linux admin at a fairly large hosting company. The only thing that I personally aware of happening this time around was that the time change triggered a bug in the OpenManage software on Dell servers causing it to use 100% CPU. The solution was to resync the time and restart OpenManage. It wasn't really a fault of Linux itself, but in OpenManage on Linux. Lots of datacenters use Dell hardware and I'm sure most use OpenManage, so I'm sure the problem was widespread.

Re:Linux (5, Informative)

Anonymous Coward | more than 2 years ago | (#40512165)

What you describe is a bug in the Linux kernel that causes problems for the Java VM that OpenManage uses.
It is not a bug in OpenManage at all.

Our Red Hat servers had no issues at all (4, Insightful)

93 Escort Wagon (326346) | more than 2 years ago | (#40512117)

I'm uncertain why these reports keeps referring to some monolithic "Linux" that is supposed to have had issues - Red Hat's the biggest Linux vendor, and certainly their "Linux" handled it just fine.

What distros had issues?

Re:Our Red Hat servers had no issues at all (4, Informative)

Nutria (679911) | more than 2 years ago | (#40512169)

TFA mentioned that the RHE6 kernel had the bug, but not RHE5.

It appears also that system load was a big factor, so if your systems aren't busy on Saturday then they might not have crashed even if running an affected kernel.

Re:Our Red Hat servers had no issues at all (1)

93 Escort Wagon (326346) | more than 2 years ago | (#40512209)

TFA mentioned that the RHE6 kernel had the bug, but not RHE5. -- It appears also that system load was a big factor, so if your systems aren't busy on Saturday then they might not have crashed even if running an affected kernel.

Ah, ok - thanks, I managed to miss that. Most of our servers are still on RHEL 5 because of some odd issues we've experienced with LDAP under RHEL 6.

I've got a test/catch-all machine on RHEL 6, but that doesn't generally have to work very hard.

Re:Our Red Hat servers had no issues at all (1)

Anonymous Coward | more than 2 years ago | (#40512199)

The bug is related to kernel version, IIRC (introduced somewhere in the 2.6 series, resolved in 3.2 or somesuch). So it depends what kernel the distros ran.

Re:Our Red Hat servers had no issues at all (-1, Redundant)

Anonymous Coward | more than 2 years ago | (#40512203)

I'm uncertain why these reports keeps referring to some monolithic "Linux" that is supposed to have had issues - Red Hat's the biggest Linux vendor, and certainly their "Linux" handled it just fine.

What distros had issues?

Which distros?

Gobbuti...no, what's it called? Ubooni? No that's not it. Hmm...Urmbumpy?

Sorry can't remember the name. It's the one that takes the credit for the work of others.

Re:Our Red Hat servers had no issues at all (4, Funny)

MightyMartian (840721) | more than 2 years ago | (#40512279)

Sorry can't remember the name. It's the one that takes the credit for the work of others.

Windows?

Re:Our Red Hat servers had no issues at all (-1)

Anonymous Coward | more than 2 years ago | (#40512679)

Seriously? +5 Funny for faggotry?

Re:Our Red Hat servers had no issues at all (4, Funny)

drinkypoo (153816) | more than 2 years ago | (#40512291)

Sorry can't remember the name. It's the one that takes the credit for the work of others.

You must be talking about SCO, but if you're still running CND you should probably upgrade.

Re:Our Red Hat servers had no issues at all (4, Informative)

Anonymous Coward | more than 2 years ago | (#40512273)

Red Hat had a lot of issues.
https://access.redhat.com/knowledge/articles/15145
https://access.redhat.com/knowledge/solutions/154713

It depended entirely on your load. The buggy kernal code ran every 17 minutes for the 24hr period leading up to the leap-second insertion.
If you had enough load, your chance of dead-locking your system increased significantly.

Solution, strip the leap-second flag by manually setting your time.

Re:Our Red Hat servers had no issues at all (1)

antdude (79039) | more than 2 years ago | (#40512325)

I think my Debian stable box's latest rbot [ruby-rbot.org] build's launch_here.rb was acting weird from the leap bug because the CPU was going high even when idled. I rebooted after 55 days of uptime and it was fine.

Re:Gentoo (1)

miknix (1047580) | more than 2 years ago | (#40512653)

Clock: inserting leap second 23:59:60 UTC

No problem whatsoever on my Gentoo server, with a 3.3.1 hardened (Linux) kernel.

Why now? (1)

Anonymous Coward | more than 2 years ago | (#40512119)

Considering leap-seconds happen every now and then, it seems odd that such fundamental things as Linux and Java can not handle it. AFAIK, it was just about for years ago since we last had a leap-second.

Re:Why now? (2, Interesting)

Znork (31774) | more than 2 years ago | (#40512265)

We will keep having these kinds of issues for as long as some people who fail to understand that time of day is an arbitrary number whose main utility lies in it being composed of predictable periods and divided into homogenous units. It should have no relation whatsoever to whatever time the sun happens to rise or set at any particular location and above all it should not be changed to accomodate fluctuations in the orbit of a rock circling an arbitrary star. Abominations like leap seconds or daylight savings make the whole system less useful by merely existing.

But personally I wouldn't be surprised if people off the equator were to get summer minutes composed of 120 seconds during daytime (or even better, a scale!) to ensure the sun rises and sets at the same time year around. Or, hey, why not simply make the seconds longer? Or a combination of both plus we can define pi to be 3 to make things simpler.

Re:Why now? (4, Insightful)

vux984 (928602) | more than 2 years ago | (#40512379)

and above all it should not be changed to accomodate fluctuations in the orbit of a rock circling an arbitrary star.

That is precisely the point of keeping track of the time of day, or day of the year.

time of day is an arbitrary number whose main utility lies in it being composed of predictable periods and divided into homogenous units.

You do not need a complex system like date time comprised of minutes hours, seconds, months, weeks, and years if you just want to measure time in a convient homogenous unit then define a time-zero, and just count milliseconds from that to whatever arbitrary distance into the past and future you want from that. Measure it kilo-seconds, mega-seconds, giga-seconds... etc.

The entire point of date/time is because we do in fact care a lot about how that "arbitrary counter" lines up with when we will be awake or asleep or eating at various points -- that's what makes it useful.

What we should have is what I've described above, time-zero and a counter. And translations from that to localized date time should be handled by a library.

What about Windows and Mac? (4, Interesting)

kthreadd (1558445) | more than 2 years ago | (#40512121)

So far all I've heard about is affected Linux systems, did Windows and OS X just fine?

Re:What about Windows and Mac? (1)

Anonymous Coward | more than 2 years ago | (#40512133)

Doesn't Windows just sync its clock once a day? I don't remember it having a proper NTP daemon.

Re:What about Windows and Mac? (1)

93 Escort Wagon (326346) | more than 2 years ago | (#40512141)

So far all I've heard about is affected Linux systems, did Windows and OS X just fine?

No problems on a couple of OS X machines that were on during the leap second - one running 10.7 Lion, the other 10.6 Snow Leopard (my laptop, which I was actively using).

Re:What about Windows and Mac? (1)

godrik (1287354) | more than 2 years ago | (#40512177)

well, none of my machines (all running Linux) were affected by the problem. I guess the bug only appeared in some systems.

Re:What about Windows and Mac? (1)

0123456 (636235) | more than 2 years ago | (#40512335)

Ditto. My Ubuntu server is still running fine (the only indication of the leap second is a message in dmesg output) and my Ubuntu laptop had no problems either.

Re:What about Windows and Mac? (2)

Nutria (679911) | more than 2 years ago | (#40512205)

And apparently neither did any desktop Linux systems.

Re:What about Windows and Mac? (1, Funny)

Alrescha (50745) | more than 2 years ago | (#40512719)

"And apparently neither did any desktop Linux systems."

There are desktop Linux systems?

(ducks)

A.

Re:What about Windows and Mac? (0)

Anonymous Coward | more than 2 years ago | (#40512161)

So far all I've heard about is affected Linux systems, did Windows and OS X just fine?

The glitch mostly affected POSIX compliant operating systems as POSIX specifies a day as 86400.

Re:What about Windows and Mac? (1)

Anonymous Coward | more than 2 years ago | (#40512185)

Mac OS X is POSIX compliant and certified UNIX. None of my OS X systems or Solaris (also POSIX and certified UNIX) had any problem with the extra second. This appears to be a problem for some Linux (not a certified UNIX system) systems.

Re:What about Windows and Mac? (2)

Guy Harris (3803) | more than 2 years ago | (#40512497)

So far all I've heard about is affected Linux systems, did Windows and OS X just fine?

The glitch mostly affected POSIX compliant operating systems as POSIX specifies a day as 86400.

So you're saying the glitch could affect OS X [opengroup.org] (or, at least, OS X Snow Leopard - although Leopard was also registered - but I'll bet Lion behaves, and Mountain Lion will behave, the same way)?

Re:What about Windows and Mac? (0)

Anonymous Coward | more than 2 years ago | (#40512167)

Maybe they are not that accurate at all.

Re:What about Windows and Mac? (1)

bickerdyke (670000) | more than 2 years ago | (#40512173)

My guess ist that Windows simply ignored it, so there never was a 61st second in a minute.

Beeing correct, on the other hand, might come as a surprise to more than one pieces of software.

Re:What about Windows and Mac? (4, Informative)

Guy Harris (3803) | more than 2 years ago | (#40512771)

My guess ist that Windows simply ignored it, so there never was a 61st second in a minute.

Well, if Microsoft's documentation of the SYSTEMTIME structure [microsoft.com] reflects the implementation, GetSystemTime() [microsoft.com] , the claim in that man page^W^WMSDN page that "The system time is expressed in Coordinated Universal Time (UTC)" nonwithstanding, cannot acknowledge the existence of a 61st second in a minute ("The second. The valid values for this member are 0 through 59.", as the SYSTEMTIME page says).

But, just as on UN*X, you have "counter" and "human-style label" times (time_t, struct timeval, struct timespec are examples of the former, and a struct tm as returned by, for example, gmtime() is an example of the latter, on UN*X), with the Windows versions of those being SYSTEMTIME and FILETIME [microsoft.com] respectively. That page on FILETIME says nothing about leap seconds - does it just keep counting over a positive leap second or does it stop or what? And, if it doesn't just keep counting over a positive leap second, does it just freeze for a while second, or does it slow down over some period of time so that it eventually syncs up, or what?

As for NTP, Microsoft has a page on "How the Windows Time service treats a leap second" [microsoft.com] , which says

When the Windows Time service is working as a Network Time Protocol (NTP) client

The Windows Time service does not indicate the value of the Leap Indicator when the Windows Time service receives a packet that includes a leap second. (The Leap Indicator indicates whether an impending leap second is to be inserted or deleted in the last minute of the current day.) Therefore, after the leap second occurs, the NTP client that is running Windows Time service is one second faster than the actual time. This time difference is resolved at the next time synchronization.

(the author of which needs to be told what "inserted or deleted" implies - do they mean that, regardless of whether a leap second is inserted or deleted, the NTP client that is running Windows Time service is one second faster than the actual time?)

And then there's one more question: if there's anything in the NT kernel that deals with leap seconds, does any version have a glitch, as some versions of the Linux kernel do [kernel.org] ?

If not, then many of the other problems might not exist on Windows. This email from John Stultz [lkml.org] , the author of the fix linked to in the previous paragraph, seems to indicate that at least some of the problems, if not all of them, stem from a kernel bug, so it might be that Java and company might be Just Fine on systems that don't have a kernel glitch of that sort (so they might work fine on at least some non-Linux systems, as well as on Linux systems with the bug fixed).

Re:What about Windows and Mac? (1)

gman003 (1693318) | more than 2 years ago | (#40512187)

As far as I can tell, all current operating systems handled it fine. It's applications that have problems, mainly server-type apps that actually use the clock for important things.

Linux being heavily affected is just a side-effect of most servers running Linux (although apparently some older versions don't handle leap seconds so cleanly - maybe that has something to do with it?).

Re:What about Windows and Mac? (0)

Anonymous Coward | more than 2 years ago | (#40512417)

So far all I've heard about is affected Linux systems, did Windows and OS X just fine?

Yes, as did any other BSD system (which OS X is a variant of), as did the commercial Unixes (AIX, Solaris (and its variants, e.g., Illunimos)).

Re:What about Windows and Mac? (0)

Anonymous Coward | more than 2 years ago | (#40512867)

Yes. Apparently, many eyes on the sources is not the only way to avoid bugs. (happy dance)

Linux kernel unable to cope? I think not. (1)

Anonymous Coward | more than 2 years ago | (#40512135)

"some of the net’s fundamental software platforms — including the Linux operating system and the Java application platform — were unable to cope with the extra second."

No opinion about java, and no doubt there's plenty of dodgy software running on Linux, but the part about Linux not coping is BS.

From last night's logs....
  Jun 30 19:59:59 thabto kernel: Clock: inserting leap second 23:59:60 UTC

Re:Linux kernel unable to cope? I think not. (4, Informative)

Anonymous Coward | more than 2 years ago | (#40512227)

There was a Linux kernel bug. See
http://news.ycombinator.com/item?id=4183122
http://marc.info/?l=linux-kernel&m=134110635328824&w=2
and
https://lkml.org/lkml/2012/6/30/122

Re:Linux kernel unable to cope? I think not. (1)

lister king of smeg (2481612) | more than 2 years ago | (#40512577)

yes, an old one that was patched before this became an issue. the issue if for un-updated/unpatched versions of Linux and shoddily written apps and java

Re:Linux kernel unable to cope? I think not. (0)

Anonymous Coward | more than 2 years ago | (#40512297)

Possibly refers to some of the issues covered here: https://access.redhat.com/knowledge/articles/15145?amp

Re:Linux kernel unable to cope? I think not. (0)

Anonymous Coward | more than 2 years ago | (#40512319)

Why is it even the kernel's business to manipulate the clock this way?

Re:Linux kernel unable to cope? I think not. (5, Interesting)

Anonymous Coward | more than 2 years ago | (#40512383)

I run Arch Linux with kernel 3.4.4 and it went haywire. My machine was very heavily loaded at the time and when the leap second happened mysqld, firefox, and ksoftirq processes started consuming 100% CPU. The load factor was well over 10 and the machine was grinding along. It didn't actually fail but it was loaded down.

Even restarting the processes didn't fix it. The high load would go away once I stopped the processes but as soon as I started them again the load would come right back. I had Firefox open on a blank page not doing anything and it was slammed at 100% CPU and had a could ksoftirq tasks slammed at 100% CPU each too.

I had to reboot the machine to get it back to normal.

I have Ubuntu and Debian servers that for whatever reason did not add the leap second so they were fine. Their time was a second off today though (at least until ntp slowly corrected it or I manually intervened).

Re:Linux kernel unable to cope? I think not. (1)

burne (686114) | more than 2 years ago | (#40512413)

You would have been fine if you stopped ntpd before restarting the offending processes.

Re:Linux kernel unable to cope? I think not. (5, Informative)

kwardroid (1466409) | more than 2 years ago | (#40512617)

Restarting ntp wasn't enough for me, I had to reset the date with:
date -s "`date`"
Only one machine went haywire though.

FUD? (1, Insightful)

jimshatt (1002452) | more than 2 years ago | (#40512145)

I don't know, but the article reads as FUD. Sure, there might have been problems, but then, aren't there always problems, everywhere? It's just a matter of picking the right ones and you've got a 'Linux and Java = bad' artice? Or am I being a fanboy now?

Re:FUD? (1)

sjames (1099) | more than 2 years ago | (#40512183)

It was a genuine bug but that doesn't make Linux or Java 'bad', all software has bugs. Good or Bad will depend on how many bugs, how often they bite and how badly.

The bug has already been fixed for months now, so the systems having trouble were the ones that weren't kept up to date.

Re:FUD? (0)

Anonymous Coward | more than 2 years ago | (#40512259)

> It was a genuine bug but that doesn't make Linux or Java 'bad', all software has bugs.

#!/bin/sh
echo "hello universe"

Look! It's an example of bug-free software!

Re:FUD? (2)

dotgain (630123) | more than 2 years ago | (#40512643)

Have you read the source for /bin/sh ?

Re:FUD? (0)

dissy (172727) | more than 2 years ago | (#40512261)

Ironically I have an old server down in the basement running a vulnerable 2.6.18.8 kernel, where the only reason I leave it even running is the uptime being at
16:34:29 up 832 days, 21:00

It survived the leap second just fine and is still running along nicely.

I hear a lot of the problems might have been around a vulnerable NTPd on a vulnerable kernel, and I do not run nptd.

I also have another machine here that started off life as a 2.6.18 kernel but has been kpatched to the latest version, and it too survived the leap second, and it does run nptd. I can only assume the fix was already kspliced in shortly after release.

All in all the whole thing seemed like a nonissue.

Re:FUD? (0)

Anonymous Coward | more than 2 years ago | (#40512293)

Well, if you're not running ntpd, then there's no way for the kernel to register the leap second. These things aren't on a regular schedule, so the kernel can only know about them when it gets the information from the outside. No leap second, no problem.

Re:FUD? (1)

Anonymous Coward | more than 2 years ago | (#40512267)

It was a genuine bug but that doesn't make Linux or Java 'bad', all software has bugs.

I seem to recall Microsoft suffering from some leap year bugs. Boy, the Slashdot comments section lit up about how even a kindergarten programmer would catch the mistake and how it's indicative of the software quality coming out of Microsoft. Now that Linux hit the same type of hurdle, we're all of a sudden being very nuanced about the definition of code quality? Typical.

Re:FUD? (2)

burne (686114) | more than 2 years ago | (#40512339)

Now that Linux hit the same type of hurdle, we're all of a sudden being very nuanced about the definition of code quality? Typical.

Wow. You're still pissed over Azure failing, your Xbox disabling itself, your Zune crashing for a full day and your Outlook manhandling your appointments (on more than one occasion)?

Talk about carrying a grudge..

Re:FUD? (-1)

Anonymous Coward | more than 2 years ago | (#40512657)

The consistent person's recipe for fun fun fun:
1. Go to a site full of two-faced liars
2. Point out one of their many glaring double standards
3. Watch them get all butthurt about it
4. PROFIT!

Re:FUD? (1)

sjames (1099) | more than 2 years ago | (#40512385)

You'll need to talk with the people who made those comments. Of course, I did provide a few more criteria you might want to consider.

Re:FUD? (1, Insightful)

lister king of smeg (2481612) | more than 2 years ago | (#40512615)

I seem to recall Microsoft suffering from some leap year bugs. Boy, the Slashdot comments section lit up about how even a kindergarten programmer would catch the mistake and how it's indicative of the software quality coming out of Microsoft. Now that Linux hit the same type of hurdle, we're all of a sudden being very nuanced about the definition of code quality? Typical.

the difference being this bug was patched already it only affected systems the were not kept up to date. Microsoft however did not patch or fix their software until after the problem had already occurred. the fault lies not in Linux here it lays with system/server admins not updating their servers. where with Microsoft case it was their direct fault for not handeling a well known date issue. where with the Linux bug no one had heard of leap seconds until yesterday yet it did have a patch already out there not their fault or problem.

Re:FUD? (0)

Anonymous Coward | more than 2 years ago | (#40512447)

And there I was thinking that my 3.2.19 kernel was fairly up to date...

Actually I didn't realize I was affected by this bug until a few minutes ago, when I used strace to see why firefox was using up all the time on one of my cores.
I think there are lots of people that didn't realize as well.

Re:FUD? (4, Funny)

kasperd (592156) | more than 2 years ago | (#40512513)

Actually I didn't realize I was affected by this bug until a few minutes ago, when I used strace to see why firefox was using up all the time on one of my cores.

You don't need a leap second in order for that to happen. Firefox does that regularly.

Extremely weird (5, Informative)

Anonymous Coward | more than 2 years ago | (#40512147)

From my own machines and comparing notes with some other people (all in all, about 3k servers) the bug seems to affect machines randomly. Known facts:

There's a kernel patch that fixes the supposed issue: https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d

Affects Debian stable a lot.

Affects Java and Virtualbox (starts using too much CPU).

Affected my browser (iceweasel on debian testing).

Affects SOME mysql installs (5.1 and 5.5, but not all, and of two identical installs one might be affected, the other not).

The fix has been posted at lot of places: /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; /etc/init.d/ntp start

(I'm all for switching unix time to a simple counter and leaving it to the calendar libs to put the leap seconds where necessary)

Re:Extremely weird (4, Informative)

burne (686114) | more than 2 years ago | (#40512387)

It's a race-condition, either crashing your ancient kernel or causing software using certain kernel-calls to effectively lock up. In both cases load seems to be a factor.

Over here the race-condition coincided with the actual leap-second and the start of the first batch of cronjobs at 02:00 local time.

(I'm all for switching unix time to a simple counter and leaving it to the calendar libs to put the leap seconds where necessary)

Bad idea. It would have prevented kernels affected by the race-condition from crashing, but would have meant most of your running software would have been either hit by this bug or would have been on the mercy of a 17 year old pimple-faced coder.

I think I prefer a crash over the mayhem caused by banking-software not handling a leap-second correctly. That could bankrupt whole countries.

Re:Extremely weird (0)

Anonymous Coward | more than 2 years ago | (#40512817)

timezone data is already added or subtracted from the time after
time() is called. so i don't buy this argument.

unfortunately it doesn't matter. due to clock drift and the need to
peg the clock representing local time to a standard, we can't prevent
the local clock from running fast or slow.

so the real solution, if you're interested in a time interval, is to use
a local clock that is not adjusted to fit a master.

Second story in one day? (0)

Anonymous Coward | more than 2 years ago | (#40512151)

Why has this "story" been posted twice in one day?

Do you guys think we are incapable of remembering things that have happened in the last 24 hours?

Re:Second story in one day? (1)

Teresita (982888) | more than 2 years ago | (#40512247)

I want justice. Next time they take away a second from the day, I want one of these "stories" to be expunged.

Re:Second story in one day? (1)

houghi (78078) | more than 2 years ago | (#40512255)

24 hours? I assume you mean 1 day and that is NOT 24 hours, but a second more.

Your E-Book Is Reading You (-1)

Anonymous Coward | more than 2 years ago | (#40512155)

Your E-Book Is Reading You

http://online.wsj.com/article/SB10001424052702304870304577490950051438304.html [wsj.com]
- http://si.wsj.net/public/resources/images/WK-BC359_COVER__DV_20120628153540.jpg [wsj.net]

"Digital-book publishers and retailers now know more about their readers than ever before. How that's changing the experience of reading."

Updated June 29, 2012, 1:39 p.m. ET | By ALEXANDRA ALTER

"It takes the average reader just seven hours to read the final book in Suzanne Collins's "Hunger Games" trilogy on the Kobo e-readerâ"about 57 pages an hour. Nearly 18,000 Kindle readers have highlighted the same line from the second book in the series: "Because sometimes things happen to people and they're not equipped to deal with them." And on Barnes & Noble's Nook, the first thing that most readers do upon finishing the first "Hunger Games" book is to download the next one.

In the past, publishers and authors had no way of knowing what happens when a reader sits down with a book. Does the reader quit after three pages, or finish it in a single sitting? Do most readers skip over the introduction, or read it closely, underlining passages and scrawling notes in the margins? Now, e-books are providing a glimpse into the story behind the sales figures, revealing not only how many people buy particular books, but how intensely they read them.

For centuries, reading has largely been a solitary and private act, an intimate exchange between the reader and the words on the page. But the rise of digital books has prompted a profound shift in the way we read, transforming the activity into something measurable and quasi-public.

The major new players in e-book publishingâ"Amazon, Apple and Googleâ"can easily track how far readers are getting in books, how long they spend reading them and which search terms they use to find books. Book apps for tablets like the iPad, Kindle Fire and Nook record how many times readers open the app and how much time they spend reading. Retailers and some publishers are beginning to sift through the data, gaining unprecedented insight into how people engage with books.

Publishing has lagged far behind the rest of the entertainment industry when it comes to measuring consumers' tastes and habits. TV producers relentlessly test new shows through focus groups; movie studios run films through a battery of tests and retool them based on viewers' reactions. But in publishing, reader satisfaction has largely been gauged by sales data and reviewsâ"metrics that offer a postmortem measure of success but can't shape or predict a hit. That's beginning to change as publishers and booksellers start to embrace big data, and more tech companies turn their sights on publishing.

Barnes & Noble, which accounts for 25% to 30% of the e-book market through its Nook e-reader, has recently started studying customers' digital reading behavior. Data collected from Nooks reveals, for example, how far readers get in particular books, how quickly they read and how readers of particular genres engage with books. Jim Hilt, the company's vice president of e-books, says the company is starting to share their insights with publishers to help them create books that better hold people's attention.

The stakes are high for the company as it seeks a greater share of the e-book market. Sales of Nook devices rose 45% this past fiscal year, and e-book sales for the Nook rose 119%. Overall, Nook devices and e-books generated $1.3 billion, compared to $880 million the previous year. Microsoft recently invested $300 million for a 17.6% stake of the Nook.

Mr. Hilt says that the company is still in "the earliest stages of deep analytics" and is sifting through "more data than we can use." But the dataâ"which focuses on groups of readers, not individualsâ"has already yielded some useful insights into how people read particular genres. Some of the findings confirm what retailers already know by glancing at the best-seller lists. For example, Nook users who buy the first book in a popular series like "Fifty Shades of Grey" or "Divergent," a young-adult series by Veronica Roth, tend to tear through all the books in the series, almost as if they were reading a single novel.

Barnes & Noble has determined, through analyzing Nook data, that nonfiction books tend to be read in fits and starts, while novels are generally read straight through, and that nonfiction books, particularly long ones, tend to get dropped earlier. Science-fiction, romance and crime-fiction fans often read more books more quickly than readers of literary fiction do, and finish most of the books they start. Readers of literary fiction quit books more often and tend skip around between books.

Those insights are already shaping the types of books that Barnes & Noble sells on its Nook. Mr. Hilt says that when the data showed that Nook readers routinely quit long works of nonfiction, the company began looking for ways to engage readers in nonfiction and long-form journalism. They decided to launch "Nook Snaps," short works on topics ranging from weight loss and religion to the Occupy Wall Street movement.

Pinpointing the moment when readers get bored could also help publishers create splashier digital editions by adding a video, a Web link or other multimedia features, Mr. Hilt says. Publishers might be able to determine when interest in a fiction series is flagging if readers who bought and finished the first two books quickly suddenly slow down or quit reading later books in the series.

"The bigger trend we're trying to unearth is where are those drop-offs in certain kinds of books, and what can we do with publishers to prevent that?" Mr. Hilt says. "If we can help authors create even better books than they create today, it's a win for everybody."

Some authors welcome the prospect. Novelist Scott Turow says he's long been frustrated by the industry's failure to study its customer base. "I once had an argument with one of my publishers when I said, 'I've been publishing with you for a long time and you still don't know who buys my books,' and he said, 'Well, nobody in publishing knows that,' " says Mr. Turow, president of the Authors Guild. "If you can find out that a book is too long and you've got to be more rigorous in cutting, personally I'd love to get the information."

Others worry that a data-driven approach could hinder the kinds of creative risks that produce great literature. "The thing about a book is that it can be eccentric, it can be the length it needs to be, and that is something the reader shouldn't have anything to do with," says Jonathan Galassi, president and publisher of Farrar, Straus & Giroux. "We're not going to shorten 'War and Peace' because someone didn't finish it."

Publishers are only just beginning to mull over the potential uses for e-reading data. Many are skeptical that analytics can aid in the industry's ongoing battle to woo consumers who are increasingly distracted by games and social media. But at a time when traditional publishers are losing ground to tech giants like Amazon and Apple, better analytics seem to offer tantalizing possibilities.

Amazon, in particular, has an advantage in this fieldâ"it's both a retailer and a publisher, which puts the company in a unique position to use the data it gathers on its customers' reading habits. It's no secret that Amazon and other digital book retailers track and store consumer information detailing what books are purchased and read. Kindle users sign an agreement granting the company permission to store information from the deviceâ"including the last page you've read, plus your bookmarks, highlights, notes and annotationsâ"in its data servers.

Amazon can identify which passages of digital books are popular with readers, and shares some of this data publicly on its website through features such as its "most highlighted passages" list. Readers digitally "highlight" selections using a button on the Kindle; they can also opt to see the lines commonly highlighted by other readers as they read a book. Amazon aggregates these selections to see what gets underlined the most. Topping the list is the line from the "Hunger Games" trilogy. It is followed by the opening sentence of "Pride and Prejudice."

"We think of it as the collective intelligence of all the people reading on Kindle," says Amazon spokeswoman Kinley Pearsall.

Some privacy watchdogs argue that e-book users should be protected from having their digital reading habits recorded. "There's a societal ideal that what you read is nobody else's business," says Cindy Cohn, legal director for the Electronic Frontier Foundation, a nonprofit group that advocates for consumer rights and privacy. "Right now, there's no way for you to tell Amazon, I want to buy your books, but I don't want you to track what I'm reading."

Amazon declined to comment on how it analyzes and uses the Kindle data it gathers.

EFF has pressed for legislation to prevent digital book retailers from handing over information about individuals' reading habits as evidence to law enforcement agencies without a court's approval. Earlier this year, California instituted the "reader privacy act," which makes it more difficult for law-enforcement groups to gain access to consumers' digital reading records. Under the new law, agencies must get a court order before they can require digital booksellers to turn over information revealing which books their customers have browsed, purchased, read and underlined. The American Civil Liberties Union and EFF, which partnered with Google and other organizations to push for the legislation, are now seeking to enact similar laws in other states.

Bruce Schneier, a cyber-security expert and author, worries that readers may steer clear of digital books on sensitive subjects such as health, sexuality and securityâ"including his own worksâ"out of fear that their reading is being tracked. "There are a gazillion things that we read that we want to read in private," Mr. Schneier says.

There are some 40 million e-readers and 65 million tablets in use in the U.S., according to analysts at Forrester Research. In the first quarter of 2012, e-books generated $282 million in sales, compared to $230 million for print, the Association of American Publishers recently found.

Meanwhile, the shift to digital books has fueled an arms race among digital start-ups seeking to cash in on the massive pool of data collected by e-reading devices and reading apps. New e-reading services, which allow readers to purchase and store books in a digital library and read them on different devices, have some of the most sophisticated reader tracking software. The digital reading platform Copia, which has 50,000 subscribers, collects detailed demographic and reading dataâ"including the age, gender and school affiliation of people who bought particular titles, as well as how many times the books were downloaded, opened and readâ"and shares its findings with publishers. Copia aggregates the data, so that individual users aren't identifiable, and shares that information with publishers that request it.

Kobo, which makes digital reading devices and operates an e-reading service that stocks 2.5 million books and has more than eight million users, has recently started looking at how readers as a whole engage with particular books and genres. The company tracks how many hours readers spend on particular titles and how far they get. Kobo recently found, for example, that most readers who started George R.R. Martin's fantasy novel "A Dance With Dragons" finished the book, and spent an average of 20 hours reading it, a relatively fast read for a 1,040-page novel.

Some publishers are already beginning to market test books digitally, before releasing a print edition. Earlier this year, Sourcebooks, which publishes 250 titles a year, began experimenting with a new model of serial, online publishing. Sourcebooks has released early online editions for half a dozen titles, ranging from romance to young adult to nonfiction books, and has solicited questions and suggestions from readers. Eventually, readers' feedback will be incorporated into the print version.

Scholastic, which publishes popular young-adult fiction such as Harry Potter and "The Hunger Games," created online message boards and interactive games connected to its popular series "39 Clues." The online game and message board, which has 1.9 million registered users, allows the publisher to track which story lines and characters are resonating with young readers. David Levithan, Scholastic's publisher and editorial director, says the online feedback has shaped the ongoing "39 Clues" series and helped to turn it into a global franchise with more than 15 million copies in print.

"You very rarely get a glimpse into the reader's mind," he says. "With a printed book, there's no such thing as an analytic. You can't tell which pages are dog-eared."

Few publishers have taken the experiment as far as Coliloquy, a digital publishing company that was created earlier this year by Waynn Lue, a computer scientist and former Google engineer, and Lisa Rutherford, a venture capitalist and former president of Twofish, a gaming-analytics firm.

Coliloquy's digital books, which are available on Kindle, Nook and Android e-readers, have a "choose-your-own-adventure"-style format, allowing readers to customize characters and plot lines. The company's engineers aggregate and pool the data gleaned from readers' selections and send it to the authors, who can adjust story lines in their next books to reflect popular choices.

"Data and analytics, we've seen how it revolutionized certain industries like mobile apps and gaming," says Mr. Lue. "With reading, we don't yet have that engagement data, and we wanted to provide a feedback mechanism that didn't exist before between authors and readers."

Coliloquy developed its software through Amazon's Kindle data developer program, which allows outside companies to create interactive content for Kindle. Their proprietary data platform draws on complex algorithms, similar to gaming software, that lets readers choose from different narrative pathways.

The company hired six editors and five technology and product developers and began recruiting authors from a range of genres, including romance, nonfiction, young adult fantasy and erotica. Since launching this past January, the company has released eight titles, and is expanding into crime fiction, legal thrillers and experimental fiction. Mr. Lue and Ms. Rutherford declined to provide sales figures for Coliloquy's titles, citing a nondisclosure agreement with Amazon. But they say more than 90% of readers who buy Colloquy's books, which range from $2.99 to $7.99, finish reading them, and 67% reread the books.

In "Parish Mail," Kira Snyder's young adult mystery series set in New Orleans, readers can decide whether the teenage protagonist solves crimes by using magic or by teaming up with a police detective's cute teenage son. Readers of "Great Escapes," an erotic romance series co-written by Linda Wisdom and Lynda K. Scott, can customize the hero's appearance and the intensity of the love scenes. A recent report from Coliloquy showed that the ideal hero for "Great Escapes" readers is tall with black hair and green eyes, a rugged, burly build and a moderately but not overly hairy chest.

In Tawna Fenske's romantic caper "Getting Dumped"â"which centers on a young woman who finds work at a landfill after getting laid off from her high-profile job at the county's public relations officeâ"readers can choose which of three suitors they want the heroine to pursue. The most recent batch of statistics showed that 53.3% chose Collin, a Hugh Grant type; 16.8% chose Pete, the handsome but unavailable co-worker; and 29.7% of readers liked Daniel, the heroine's emotionally distant boyfriend.

Ms. Fenske originally planned to get rid of Daniel by sending him to prison and writing him out of the series. Then she saw the statistics. She decided 29.7 % was too big a chunk of her audience to ignore.

"So much of the time, it's an editor and agent and publisher telling you, 'This is what readers want,' but this is hands-on reader data," says Ms. Fenske, 37, who lives in Bend, Ore. "I've always wondered, did that person buy it and stop after the first three pages? Now I can see they bought it and read it in the first week.""

Write to Alexandra Alter at alexandra.alter@wsj.com

Copyright ©2012 Dow Jones & Company, Inc. All Rights Reserved

Dreaded S60 bug... (2)

mschaffer (97223) | more than 2 years ago | (#40512229)

It's like the Y2K bug, but every few years.

Re:Dreaded S60 bug... (1)

PolygamousRanchKid (1290638) | more than 2 years ago | (#40512483)

Hey, it's an excellent opportunity to drum up bogus consulting work!

"Are your old C programs able to handle a leap second! Think of how much money your company will lose when that one extra second of interest gets calculated on your bank accounts! You need me to check your code for you!"

"Thanks, see you around, for the next leap second!"

The IT industry definitely needs for leap seconds.

RHEL + JBOSS = LOAD (0)

Anonymous Coward | more than 2 years ago | (#40512309)

We had many servers with this issue, mostly RHEL 6 servers running JBoss. The only symptom is high load. If you are not actively monitoring your server load, you may not even know that there's an issue yet.

I always thought leap seconds were stupid (2, Interesting)

circletimessquare (444983) | more than 2 years ago | (#40512375)

Why not bundle them and apply them every 10 or 20 years?

And apparently I'm not alone:

http://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds [wikipedia.org]

Hogwash, Astronomers can find coping mechanisms, it's either that or these ridiculous levels of stress for systems admins.

Re:I always thought leap seconds were stupid (2)

burne (686114) | more than 2 years ago | (#40512429)

Hogwash, Astronomers can find coping mechanisms, it's either that or these ridiculous levels of stress for systems admins.

TAI [wikipedia.org] doesn't know about leapseconds, and it's the coping mechanism of choice for astronomy.

Re:I always thought leap seconds were stupid (1)

circletimessquare (444983) | more than 2 years ago | (#40512437)

so there you go. there's no good reason for leap seconds. bundle them and apply them once a decade or more.

Re:I always thought leap seconds were stupid (3, Interesting)

at10u8 (179705) | more than 2 years ago | (#40512837)

except that BIPM, the providers of TAI, have published this http://www.bipm.org/cc/CCTF/Allowed/18/CCTF_09-27_note_on_UTC-ITU-R.pdf [bipm.org] wherein the CCTF "stresses that TAI is the uniform time scale underlying UTC, and that it should not be considered as an alternative time reference." This appears to indicate that the CCTF and BIPM are not comfortable with the notion that operational systems might be employing TAI as their time scale. At the end of that paper they also discuss the possibility that TAI could cease to exist.

Re:I always thought leap [years] were stupid (0)

Anonymous Coward | more than 2 years ago | (#40512435)

Hogwash, Astronomers can find coping mechanisms, it's either that or these ridiculous levels of stress for systems admins.

The same can be said for leap years. They'v been around for a few hundred years and people still can't cope with them. Why don't we just go back to the Julian calendar and drop the Gregorian one?

Ditto for Daylight Saving Time which, IMHO, is completely arbitrary and not tied to any physical need or phenomenon, and we're still "stuck" with nonetheless.

Only Linux affected? (4, Interesting)

cpghost (719344) | more than 2 years ago | (#40512399)

I'm managing a cluster of 2,400 nodes running FreeBSD, and AFAICS, none was tripped off by leap second NTP adjustments. On the other hand, 4 out of 180 Linux nodes crashed simultaneously at that very moment. All this is exceedingly weird, but may indeed point to a subtle bug in the Linux kernel (only?). I've never witnessed this behavior in the past.

Re:Only Linux affected? (-1)

Anonymous Coward | more than 2 years ago | (#40512621)

Probably because the 2400 FreeBSD nodes were not doing anything of value.

Re:Only Linux affected? (1)

Anonymous Coward | more than 2 years ago | (#40512791)

IIRC, one of the main FreeBSD dudes is a bit of an atomic timekeeping hobbyist..

I'd actually _expect_ that OS at least to work properly re. leap seconds :-)

Debian + Java = Issues (2)

thatskinnyguy (1129515) | more than 2 years ago | (#40512415)

About 5 seconds after midnight GMT a Java server app running on my Debian Squeeze server decided it was going to eat-up ALL THE THINGS and for some reason, the server rebooted itself. Glad to know I wasn't alone in shitting myself over odd behaviours.

No program I ever write will be able to cope (0)

Anonymous Coward | more than 2 years ago | (#40512439)

I will never write a program that correctly handles seconds=60. Period. EVAR!

Google on how they fixed that.. (3, Interesting)

Barryke (772876) | more than 2 years ago | (#40512517)

Google official blog: "Time, technology and leaping seconds" (sept 2011)
http://googleblog.blogspot.in/2011/09/time-technology-and-leaping-seconds.html [blogspot.in]

I wonder if the leap second has anything to do with the labs Chubby paper / site currently being offline..

Re:Google on how they fixed that.. (0)

Anonymous Coward | more than 2 years ago | (#40512589)

That blog post could have been about 1% the size it was:

We implemented slew mode in our NTP servers.

fp bfitch (-1)

Anonymous Coward | more than 2 years ago | (#40512525)

they're gone Mac Usenet. In 1995, BUWLA, or BSD Whether to repeat AMERICA) might be from the FreeBSD are almost You have a play

hmm (0)

Nocturnal Deviant (974688) | more than 2 years ago | (#40512571)

Nobody has posted that it also took thepiratebay down, something that the MAFIAA has been trying to do for the last umpteen years?

MySQL had issues only. (1)

Zombie Ryushu (803103) | more than 2 years ago | (#40512581)

MySQL started spiking my CPU when the leap second hit. Only MySQL, and nothing else. It was odd.

Hmm, could this have been the cause of my issues? (1)

psm321 (450181) | more than 2 years ago | (#40512781)

I had a lot of programs (none Java-based though) taking up an inordinate amount of CPU, and high system CPU usage. Couldn't figure out the cause, and a reboot fixed it. In retrospect, I think it was around midnight UTC.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?