Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

GitHub Back Online After Service Outage

timothy posted about a year ago | from the you-were-asleep-face-it dept.

Bug 55

The Next Web reports that GitHub — home to many open source projects — suffered (and quickly recovered from) a service outage this morning, starting around 14:00 UTC. Other than that the problem "appears to have been caused by its database server," the cause isn't clear.

cancel ×

55 comments

Sorry! There are no comments related to the filter you selected.

Sponsoring a High Availability solution? (1)

DF5JT (589002) | about a year ago | (#43889097)

We are seriously considering sponsoring github with a free platinum support and maintenance contract for the Linux cluster stack, i.e. DRBD, Heartbeat, Pacemaker, Corosync.

Would that help?

Re:Sponsoring a High Availability solution? (0)

Anonymous Coward | about a year ago | (#43889139)

No. It wouldn't prevent outages (nothing will), and it sounds like they recovered quickly. For all you know the source of the outage was a meteor strike. Having said that, it might allow them to reduce some operational expenses so maybe it would be worthwhile.

Re:Sponsoring a High Availability solution? (0)

Anonymous Coward | about a year ago | (#43889261)

meteor strike is easy to plan for -- put the data in two data centers very far apart -- East/West coast of the US is good enough.

fat finger / bad database schema changes are much more difficult to catch every time.

free platinum support won't really help there.

Re:Sponsoring a High Availability solution? (2)

monkeyhybrid (1677192) | about a year ago | (#43889469)

meteor strike is easy to plan for -- put the data in two data centers very far apart -- East/West coast of the US is good enough

That's fine for most meteors, but what about asteroids that could destroy everything on Earth? That's why you should backup to the cloud, on another planet.

Re:Sponsoring a High Availability solution? (0)

Anonymous Coward | about a year ago | (#43889589)

What if the meteor strikes in places that make many american backbones out of power or destroyed? What if verizon gets hacked? What if the system itself gets hacked? What if a solar flare causes a blackout in the US?

Outages are simple out of our control.

Re:Sponsoring a High Availability solution? (4, Funny)

Mitchell314 (1576581) | about a year ago | (#43890065)

Do what Starbucks does. Use portals to other dimensions. Why do you think they're all over the place?

Re:Sponsoring a High Availability solution? (4, Funny)

Shimbo (100005) | about a year ago | (#43890771)

Do what Starbucks does. Use portals to other dimensions.

I thought that was just to avoid tax.

Re:Sponsoring a High Availability solution? (2)

hobarrera (2008506) | about a year ago | (#43895807)

If everything on earth is destroyed, all remaning github users will have no issues with the new uptimes anyway.

Re:Sponsoring a High Availability solution? (1)

Cammi (1956130) | about a year ago | (#43889293)

HA and DR will ... That's their sole purpose.

Re:Sponsoring a High Availability solution? (2, Funny)

Anonymous Coward | about a year ago | (#43889491)

If the meteors go on strike I think we'd have known about it.

Re:Sponsoring a High Availability solution? (2)

Cammi (1956130) | about a year ago | (#43889281)

Actually, it would help if it was a simple database outage. However, it sounds like someone over at Github did something stupid .. .like, deleted the database itself. In that case, nothing can help.

Re:Sponsoring a High Availability solution? (1)

slashmydots (2189826) | about a year ago | (#43889437)

Oh, that's not a fun kind of problem because then if you check the site with downforeveryoneorjustme.com it will show as up even though the site isn't "working."

Re:Sponsoring a High Availability solution? (1)

aflag (941367) | about a year ago | (#43890429)

Then downforeveryoneorjustme.com should consider the HTTP status response. I'm sure it's not 200 when something bad happens.

Re: Sponsoring a High Availability solution? (0)

Anonymous Coward | about a year ago | (#43890629)

Get rid of your Opstards.

Re:Sponsoring a High Availability solution? (3, Informative)

Anonymous Coward | about a year ago | (#43889525)

In my opion their system already handles everything quite well. I've noticed few outages (this one I only heard about because I was reading slashdot, it's sunday, I'm not even working on my project right now) and, so far, they all have recovered quickly. Moreover, due to the nature of git, the outages could even be longer and more frequent that it would still not be a big issue. The issue tracking system is a bit more critical, but I hardly think anything of great value will be lost due to the problems I've seen so far. From the perspective of one of their paying users, the only change I'd care about is lower price. The system itself is already available enough that I don't consider that an issue at all.

The most worrisome thing on github so far was when it got hacked. However, having high availability would not help in this situation. That was a one time thing. That seems to have happened once to every major service out there. If it was a regular thing, then it would be a big issue.

Re:Sponsoring a High Availability solution? (2)

dkf (304284) | about a year ago | (#43890097)

The issue tracking system is a bit more critical

They should put each project's issues in a git repository, so that you can trivially keep them replicated on your own systems.

I know we like to sleep late (5, Funny)

MLBs (2637825) | about a year ago | (#43889255)

But describing 14:00 as morning is a bit too much.

Re:I know we like to sleep late (1)

uberbrainchild (2860711) | about a year ago | (#43889349)

Well it was morning in the US

Re:I know we like to sleep late (1)

MLBs (2637825) | about a year ago | (#43889407)

It's always morning somewhere

Re:I know we like to sleep late (5, Informative)

Anonymous Coward | about a year ago | (#43889455)

GitHub is based in San Francisco. The time given is UTC. 14:00 UTC is 7am in San Francisco. "Morning" is a local time reference.

Re:I know we like to sleep late (2)

loufoque (1400831) | about a year ago | (#43889663)

It's morning until you've had lunch.

Re:I know we like to sleep late (1)

serviscope_minor (664417) | about a year ago | (#43894271)

It's morning until you've had lunch.

It's morning until you have your first beer. Morning ends at about 8:45am around here.

Re:I know we like to sleep late (1)

ZDroid (2938715) | about a year ago | (#43898245)

But describing 14:00 as morning is a bit too much.

Here's The World Clock [timeanddate.com] . ;)

Github DB failure following govt announcement (1)

Anonymous Coward | about a year ago | (#43889283)

Not soon after the US government announced it would distribute info via github, github suffers failures. Coincidence? You decide.

Re:Github DB failure following govt announcement (2)

game kid (805301) | about a year ago | (#43889907)

Maybe Github shut down to...voluntarily add law enforcement backdoors. Yeah. Completely voluntarily. Totally not due to legislation by BSA bribe or anything.

how the fuck (-1)

Anonymous Coward | about a year ago | (#43889413)

is this even remotely newsworthy, even for slashdot, news for nerds and stuff that matters?

there is NO online service, web site, or server, that has absolute 100% uptime and availability. NONE. claim otherwise all you want, but you (and us) know you're a fucking liar if you do.

Re:how the fuck (0)

Anonymous Coward | about a year ago | (#43890037)

Maybe it was just slow news day.

This is front page? srsly? (4, Insightful)

dbc (135354) | about a year ago | (#43889453)

So, sure, I use github. But... it goes down for a couple of hours and SlashDot panics? This isn't news.

Re:This is front page? srsly? (1)

Anonymous Coward | about a year ago | (#43889545)

News for nerds, that used to mean something around here. I think this sits pretty squarely within that, even if it doesn't affect you at at all. Certainly better than much of the news on here that's been not even slightly nerd related.

Re:This is front page? srsly? (1)

Anonymous Coward | about a year ago | (#43889571)

No, this counts as news. It was definitely an "oh, sh!t" moment for many people, and requires an explanation. Only in hindsight is it known that the outage was temporary.

Re:This is front page? srsly? (3, Interesting)

Antique Geekmeister (740220) | about a year ago | (#43889651)

It's news. I've corporate partners who rely heavily on gitbub.com for access to their open source tools and even for their corporate git repositories, since they're more reliable than almost any in-house source code repository I've dealt with. This especially includes the hand-built, written by the CIO source control systems, that are surprisingly common in startups before they mature. I know companies whose automated software continuous build environments because of this, so it's certainly news.

Re:This is front page? srsly? (1)

greg1104 (461138) | about a year ago | (#43890057)

I've corporate partners who rely heavily on gitbub.com

gitbub is the best there is at what they do, but what they do best isn't very nice.

Re:This is front page? srsly? (2)

VortexCortex (1117377) | about a year ago | (#43893699)

I just use Git. You know, that DECENTRALIZED source control thing.... yeah, might want to think about using it in a decentralized way, since, well, you know? That's what it is? I mean, unless you mean that all of the systems you're writing code with are collectively less reliable. Seems like PEBKAC to me. Hell, you can comment out one line in a .git config file to enable a post-update hook, and you're pretty much done setting up an "in-house source code repository".

Put it this way: If any one part of your DISTRIBUTED source control goes down for a few hours, and that's a big deal.... Then you're a fucking idiot.

Re:This is front page? srsly? (1)

serviscope_minor (664417) | about a year ago | (#43894277)

Put it this way: If any one part of your DISTRIBUTED source control goes down for a few hours, and that's a big deal.... Then you're a fucking idiot.

Or, truly ingenious. I have a paid up github account for work purpouses I wouldn't actually have any idea how to make a github outage become a big deal.

Re:This is front page? srsly? (0)

Anonymous Coward | about a year ago | (#43894315)

Lol. Hi idiot. Let me explain what decentralization means within the context of Git. It lets one person who has one cloned the repository upload it to a new host, and be ready to rock. Any of the other contributors can add it as a remote, then push their changes and you're good to go. This means they have an "in-house source code repository" times the amount of people who have ever contributed to the project.

However what Antique Geekmeister primarily described was the companies depending on Github for retrieving open source tools, which an in-house repository would not help with... Unless you mean to clone *all* of github? :)

Re:This is front page? srsly? (1)

Antique Geekmeister (740220) | about a year ago | (#43894561)

Actually, I referred both to open source repos and to corporate sponsored, private repositories which are the _reference_ clones for other git users to update or clone from. this is particularly important for automatic build systems, which should only be _pulling_ changes form the common repository, never publishing changes to that reference repository.

git does not force this approach, you can switch to other repositories, but it is nonetheless extremely common and just how most people wind up using git.

Re:This is front page? srsly? (1)

Antique Geekmeister (740220) | about a year ago | (#43894663)

> config file to enable a post-update hook, and you're pretty much done setting up an "in-house source code repository".

This kind of thing is _precisely_ why many developers,and many IT departments, don't get along well. For example, any developer can instlal sendmail: or Apache or a file server. Running a 24x7 critical high availability service with backup, account management, and user support is a larger task, and the IT department really has to think in those terms if they're skilled.

github has been a critical central repository for thousands of open source projects, hundreds of which I've had to work with in the last few years. Your personal home repository doesn't do me any good, nor will it collect and merge the changs from other developers.

Re:This is front page? srsly? (1)

bill_mcgonigle (4333) | about a year ago | (#43889961)

It was down on Thursday for a little while too - if the story were about a pattern, perhaps it would be noteworthy if not newsworthy.

But, hey, I appreciate big sites being down every once in a while. When my systems have better uptime than those that Amazon runs, it's at least an understandable point of reference for PHB's.

Re:This is front page? srsly? (0)

Anonymous Coward | about a year ago | (#43890909)

A client's very large company pays GitHub for commercial grade service. The outages of varying levels have been frequent and getting in the way of team development efforts. The outages have cost us days of work with many dozens of developers being stuck and unable to work.

Re:This is front page? srsly? (0)

Anonymous Coward | about a year ago | (#43890979)

Github is down constantly. Enough that if you use it full time for your dev work it will impact you several times a year. They need an ops person badly.

Re:This is front page? srsly? (0)

Anonymous Coward | about a year ago | (#43893235)

Actually, I've seen github down 4 times already this year when I needed it. And when it went down, there's just basic status that really says, "wait". The last time this happened for me, I waited 5min and gave up... but sure put be behind work half a day (there is such thing as work schedules and NON-work schedules).

With this high availability around us, I'm really considering going back to a local git or hg. I mean the whole reason for github is availability... for critical projects (I have a paid account). I know some will beg to differ, aka it's for social coding, but there's so many other ways to share code nowadays (e.g. googlecode).

Re:This is front page? srsly? (0)

Anonymous Coward | about a year ago | (#43895615)

You can still push/pull from a secondary remote if you like. Doesn't make a jot. If you don't trust github's reliability use github and your own solution.

Re: This is front page? srsly? (0)

Anonymous Coward | about a year ago | (#43894009)

"frontpage" lol

Centralization is risky (1)

Bing Tsher E (943915) | about a year ago | (#43889715)

For years, going back to the days when SourceForge (forgery of source code??) was closely associated with Slashdot, I have been nervous of centrally-located sites for massive numbers of projects. Yes, locate resources there as a robust distribution front end. But have an independent presence on the net as well.

Centralization of something that is otherwise as free-wheeling and independence minded as Open Source Software, just seems contradictory.

Re:Centralization is risky (1)

aflag (941367) | about a year ago | (#43889933)

At first sight centralization looks like a great way to reduce costs, however, there are problems that are likely to make the cost actually rise. I'm unaware of any studies showing when it's good to centralize and when it's bad. It would be a very interesting study to read. While you may need fewer machines to run 20 websites together than you'd need to run them separately, but you'll also need more qualified staff. An avarage 16 years old with a bit of time in his hands and an interest for technology can easily configure a machine capable of handling hundreds of requests per second. That's more than most sites need. As you strive to share more and more resources, you'll need more and more qualified staff, capable of figuring out how to manage the resource sharing. Resource sharing is one of the most difficult problems in computer science. With machine prices being so low, is it really worthy to solve those problems for all but really big, google sized, sites?

Re:Centralization is risky (1)

VortexCortex (1117377) | about a year ago | (#43889963)

Centralization of something that is otherwise as free-wheeling and independence minded as Open Source Software, just seems contradictory.

Yep. I never understood Giant Websites. We give the world a distributed information network with fault tolerance and a self healing structure designed to withstand entire cities disappearing (censorship, nuclear war, etc), and what do folks do? Centralize the shit out of everything. Data Silos?! CLIENT SEVER architecture?! That was the whole point. There IS NO CLIENT on the web. EVERYONE is supposed to be a server. It's a shame that greed has firmly entrenched the essentially prototype protocol IPv4 (which was never meant to be used for this long, hence the 32bit IP addr...) and reinforced the divide between content providers and consumers via NAT.

This is why we can't have nice things. Long Live The Internet, but Fuck the Web.

Re:Centralization is risky (1)

aflag (941367) | about a year ago | (#43890061)

Some centralization is required, though. There must be at least an index, like google, to find the IPs of relevant servers with the information you need. How would you solve information retrieval problem without a server-client platform. I think it's likely to be the cheapest solution for the problem. That been said, there is really no need for everyone sharing the same email server, for instance. I agree with you that there are more client-server designs than needed, but they are needed. They can also be the simplest technical solution, which is good by itself. A social network, like facebook, for instance. Didn't need to be centralized, and it would be a lot better if it wasn't.

Re:Centralization is risky (1)

anyanka (1953414) | about a year ago | (#43890753)

Well, funnily, the index (DNS) that allows you to find IPs of relevant servers (and anything else) *is* decentralized. At least for many values of "decentralized".

Re:Centralization is risky (1)

aflag (941367) | about a year ago | (#43891219)

Well, it's more delegated than it is decentralized. In the end you still have an authority dictating the rules. Anyway, DNS has a very simple query and a very simple and straightforward answer. However, if you want to find some content on the Internet, the query and the answer are much more sofisticated. When you take into account spam, algorithm improvement and so on. It looks very hard to think of a decentralized index that could possible work and provide better results than google.

Re:Centralization is risky (0)

Anonymous Coward | about a year ago | (#43890937)

You're looking at it the wrong way. This isn't about centralization of machines, it's about specialization of people.

I've maintained and hosted my own VCS, SMTP, HTTP, NNTP, and SSH servers for well over a decade now. I'd never rely on third parties for this stuff, because I don't trust them, and I don't have to trust them. Not only can I admin my own systems, I can understand and hack on all the relevant source code. But most other people can't do this, including supposed "geeks".

Once upon a time the badge of cool was running your own server, now it's just having an account on some website. Which is sad, because people will never truly understand how these systems work, and how they _should_ work, unless they actually spend time creating and maintaining them themselves.

It's doubly sad because as the number of people like me dwindle, at some point government and big corporations will start blocking access to my mail server and my web server. My freedom will be taken away because you all are lazy.

Also, FWIW, one nice thing about a distributed VCS is that nothing is centralized. It's entirely possible to use Github and maintain your own master (or multiple master trees) separately. I doubt many people do this, but it's possible. So Github is slightly less evil this way.

Re:Centralization is risky (1)

Anonymous Coward | about a year ago | (#43890073)

For years, going back to the days when SourceForge (forgery of source code??) was closely associated with Slashdot

Back to the days? They're both still owned by the same company (currently Dice Holdings).

Re:Centralization is risky (0)

Anonymous Coward | about a year ago | (#43890801)

Good thing that everybody who clones the repository has all the history, so it's trivial to migrate to another hosting platform or set up your own.

Isn't this the point behind git? (1)

Anonymous Coward | about a year ago | (#43890661)

Isn't the benefit of git that you keep the repository on your local machine, not on a central server like svn? Can't a git project absorb a brief outage?

Re:Isn't this the point behind git? (1)

Oceanplexian (807998) | about a year ago | (#43892165)

As someone working at a small, relatively new startup using a 'hip' stack, unfortunately, no.

Most new shops heavily tie into Github for code deployment, configuration management, and most importantly, Agile development practices that require continuously querying Github to test code (CI) plus frequent pulls and branching.Github being down basically means development grinds to a halt and startups' ability to run their own stack diminishes.

Like the Amazon outages, hopefully this serves to teach people that infrastructure isn't just something you can outsource to the 'cloud' and expect to work perfectly without doing your homework.

The reality is that Github is just a hosting provider and no different than any other hosting provider in the last 20+ years.

and a thousand brogrammers (-1)

Anonymous Coward | about a year ago | (#43891399)

all cried out as their branch of some shitty unknown open source project was suddenly unavailable to the world.

Pull this -- high availability!

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>