Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Nine Traits of the Veteran Network Admin

samzenpus posted about a year ago | from the man-behind-the-cable-curtain dept.

Businesses 142

snydeq writes "Born or made, network admins share certain defining characteristics. Deep End's Paul Venezia offers nine: 'I hope that this insight into the extremely logical, yet consistently dangerous world of the network admin has shed some light on how we work and how we think. I don't expect it to curtail the repeated claims of the network being down, but maybe it's a start. In fact, if you're reading this and you are not a network admin, perhaps you should find the closest one and buy him or her a cup of coffee. They could probably use it.'"

cancel ×

142 comments

Sorry! There are no comments related to the filter you selected.

Will read later (5, Funny)

Anonymous Coward | about a year ago | (#44315509)

The network is down.

Re: Will read later (1)

Anonymous Coward | about a year ago | (#44315569)

#10: Scragly neck beard that makes users as uncomfortable as his "you just don't understand" rants.

Re:Will read later (5, Funny)

Razgorov Prikazka (1699498) | about a year ago | (#44315687)

Have you tried turning it off and on again?

Re:Will read later (1)

Ghjnut (1843450) | about a year ago | (#44317031)

At least 3 times

Re:Will read later (0)

Anonymous Coward | about a year ago | (#44317185)

best video ever

Re:Will read later (1)

hvm2hvm (1208954) | about a year ago | (#44315871)

It gives me 403 forbidden... so that's ironic

Re:Will read later (1)

Anonymous Coward | about a year ago | (#44315949)

Your admin blocks Infoworld articles? Thank him sometime for looking out for you.

Re:Will read later (1)

somersault (912633) | about a year ago | (#44316133)

No, because a 403 is a response from the webserver. A proxy would probably give a different message, and if it was blocked via firewall rules then it just would give no response whatsoever..

Re:Will read later (3, Informative)

SuricouRaven (1897204) | about a year ago | (#44316793)

A proxy delivers whatever code the admin wants it to deliver. Squid returns 403 on ACL blocks, but it's trivial to alter.

Re: Will read later (4, Insightful)

nosfucious (157958) | about a year ago | (#44316103)

These "traits of" or "habits of" articles are the modern equivalent of horoscopes. Lots of feel good stuff, but not much actual advice. I can agree and disagree with every point to some extent.

Nice article that generates a lot of page views. For real advice get 1 or more O'Reilly books, or better yet, find a competent admin and become his/her apprentice.

Re:Will read later (2, Funny)

Anonymous Coward | about a year ago | (#44316173)

Call to Admin: "Network is slower today than usual". Network Admin to caller: The weather is hot today, bits expand and it's harder for them to squeeze through thin wires.

Re:Will read later (2)

fuzzywig (208937) | about a year ago | (#44316911)

"You could try straightening out your network cable, just go down behind your computer and massage out any kinks in it that might be slowing down the data."

By the time they've finished doing that, you'll probably have worked out what was causing the slowdown, and either fixed it, or allowed it to fix it's self.

Re:Will read later (0)

Anonymous Coward | about a year ago | (#44316235)

Nuhuh.

The network is the computer.....

The spice of life (4, Funny)

cold fjord (826450) | about a year ago | (#44315527)

For many reasons there are few things add excitement to life like working with someone who habitually answers the phone with, "I didn't do it."

Re:The spice of life (-1, Troll)

Anonymous Coward | about a year ago | (#44315601)

God, I sure want to dip my nutsack into your anus! Why don't I just do it? Now, open your anus! Open it! I demand that you grant my nutsack entry into your underhole!

Obligatory xkcd (4, Funny)

vikingpower (768921) | about a year ago | (#44315541)

devotion to duty [xkcd.com]

Re:Obligatory xkcd (2, Funny)

L4t3r4lu5 (1216702) | about a year ago | (#44316167)

Holy fucking balls, you're such a moron! That's SYSTEMS! You know, the guys that make sure that the monkeys who think knowing PowerShell makes you some kind of IT god? NOTHING. They are FILTH.

We're talking about Networks. Systems can do NOTHING without my massaging of the thinwire, caressing of the switch configuration. Systems are USELESS, DUMB without us. Systems is the heart of the network? Well, Networks is the vascular system. What is the heart without the blood vessels? Just another lump of useless meat.

Excuse me for a moment. Latency just jumped three orders of magnitude on the fiber trunk to another building. I can hear the packets screaming already...

I'll save you. I'll save you all.

Re:Obligatory xkcd (2)

Anonymous Coward | about a year ago | (#44316465)

The REAL Sysadmin is the Network/Server/Software/Hardware person. A long time ago in a Galaxy far, far away.... You did everything and anything involved with IT. You even had to manage PBX and telco equipment, not just the routers and switches. You took care of the tape backups and user printers. You had to write code and compile kernels. And you had to know about electronics to repair outdated equipment critical to the workings of the company.

Re:Obligatory xkcd (2)

ixidor (996844) | about a year ago | (#44316653)

and this would be a the job description of where i am now, add in gym tv repair man, light-fixture-changer, air-filter-changer, and 100's of other things.

Re:Obligatory xkcd (0)

NatasRevol (731260) | about a year ago | (#44316833)

Only in companies that are ~ 100 people

Re:Obligatory xkcd (0, Insightful)

Anonymous Coward | about a year ago | (#44316785)

Speaking of lumps of useless meat, sys admins are just CS dropouts YO!

Re:Obligatory xkcd (1)

Anonymous Coward | about a year ago | (#44317065)

no.

CS dropouts that become sysadmins are reformed sysadmins.

And they usually have a dozen CS guys under them.

The true neck beards never went to college. No need to be institutionalized.

Re:Obligatory xkcd (0)

Anonymous Coward | about a year ago | (#44317209)

Word. First internship in college, the network went down. Work could not get done. The pro sysadmin "called a guy". Obese, pimple faced guy with a neck beard comes in wearing one shoe one flip flop tshirt that barely covers his belly with one neon green word on it "Linux". Bangs on keyboard for a while, swaps out some cabling, makes a few calls and leaves. I asked the sysadmin what company that guy worked for, and he just said it was someone he met in the computer isle at borders. He works for old computer parts and band width.

Re:Obligatory xkcd (0)

Anonymous Coward | about a year ago | (#44318357)

Sysadmins play with software, developers understand software.

Re:Obligatory xkcd (3)

zildgulf (1116981) | about a year ago | (#44317109)

In spite of all of the times I wired numerous double-wide and single-wide mobile units, big enough to hold 48 or 18 people respectively, the VoIP phones that trash their own firmware upon reboot, full recoveries in parking lots, fields, and any open spot after a hurricane or storm, all of the EIGRP and BGP failures, MPLS installations, numerous satellite equipment failures, bugs in switch, router, and firewall software, multicast software packages having dubious compatibility with firmware versions of our equipment, numerous firewalls each with its own way to make configurations and access lists, PBX systems going haywire, PRI, SIP, T1, and Fiber link outages, corrupted or destroyed MX and DNS records, e-mail services crapping out, lost connections to the cloud services, the DoS attacks, and such, there is nothing like being a field network engineer for disaster recovery. I LOVE my job!!!

Re:Obligatory xkcd (1)

Penguinisto (415985) | about a year ago | (#44317155)

I'll save you. I'll save you all.

...until your workstation goes splat. Then you'll be over at the help desk, begging them to get your laptop back up. Therein lies the problem with over-specialization. ;)

(...meanwhile, a real sysadmin will have will have told you which router you screwed up the configs on, and would you please either fix the damned thing, or at least cough up the supe password so that he can do it and get on with the day.)

I RTFA (5, Insightful)

19thNervousBreakdown (768619) | about a year ago | (#44315545)

Why do I feel like I just watched somebody jerk off?

Re:I RTFA (5, Insightful)

Anonymous Coward | about a year ago | (#44315557)

Because you can't handle the sheer awesomeness that is the veteran network admin!

Re:I RTFA (1)

bickerdyke (670000) | about a year ago | (#44315593)

To me, it was like watching someone jerk of while insulting everyone around as idiot.

(Really, the first 5 bullet points were nothing but "all mere mortal users are just stupid" ranting)

Re:I RTFA (1)

1s44c (552956) | about a year ago | (#44315665)

To me, it was like watching someone jerk of while insulting everyone around as idiot.

That's my experience of most network admins in big companies. They mostly come across as a bit OpenBSD-mailing-list if you see what I mean. They are defensive as hell, maybe their jobs make them that way.

I got in some nasty arguments with them over innocent jokes that Unix or Windows admins would have just laughed off.

Re:I RTFA (0)

Anonymous Coward | about a year ago | (#44315733)

I got in some nasty arguments with them over innocent jokes that Unix or Windows admins would have just laughed off.

Yeah, the one about being binary curious always seems to piss them off. And never ask a network admin "can you hear my packets now?"

Re:I RTFA (1)

ArsenneLupin (766289) | about a year ago | (#44317137)

can you hear my packets now?

ping -a linux.org

Re:I RTFA (4, Insightful)

evilviper (135110) | about a year ago | (#44316205)

They mostly come across as a bit OpenBSD-mailing-list if you see what I mean. They are defensive as hell, maybe their jobs make them that way.

Imagine running your company's entire IT infrastructure off of a random assortment of Android cell phones... That's basically the job of a Network Administrator.

Despite running the most critical piece of infrastructure, they can't peer into the base system at all, to see how it's going to behave in edge cases. The monitoring and management tools available are shockingly poor, and offer little utility.

They're forced to use a random mix of different network hardware, which all behaves differently, and each has a list of errata 100 pages long. Updates are huge monolithic firmware blobs, guaranteed to have bugs, leading you to choose between dealing with the awful but barely manageable bugs you have now, or updating and dealing with a whole new set of bugs, and potentially some so bad they'll take the system down, and there's relatively little you can do to test this, even if you have a rather large testing network (which most do not).

Software Defined Networking and OpenFlow look like they might finally start changing this, and Network Admins will be able to understand how these complex systems work, and even define the behavior they want. But in the mean time, your Network Admin remains the Prime Minister of the insane asylum.

Re:I RTFA (1)

1s44c (552956) | about a year ago | (#44318291)

Thanks for explaining.

Re:I RTFA (1)

Penguinisto (415985) | about a year ago | (#44317187)

They are defensive as hell, maybe their jobs make them that way.

I got in some nasty arguments with them over innocent jokes that Unix or Windows admins would have just laughed off.

Defensive is a good description. After all, there are only so many commands in IOS, and -- oh, look, the Cisco Nexus switches use Linux now! :)

Re:I RTFA (0)

Anonymous Coward | about a year ago | (#44315905)

(Really, the first 5 bullet points were nothing but "all mere mortal users are just stupid" ranting)

Just about anyone could have posted those five bullet points as a comment to a /. article on yesterday's Netsol outage. I was unimpressed.

Re:I RTFA (0)

Anonymous Coward | about a year ago | (#44316025)

And how is that not true?

Re:I RTFA (1)

Simulant (528590) | about a year ago | (#44317947)

#1 should have been Arrogance.

I'm not sure he even understands the definition of "trait"

This all looks familiar... (1)

r33per (585447) | about a year ago | (#44315553)

This is like a LinkedIn article. 8 ways to do this, 5 systems to improve that, 3 things to avoid when doing the next thing and 9 traits of the successful Whatshisname.

It's maddening.

Maddening? (0)

Anonymous Coward | about a year ago | (#44315695)

Nah, it's just another piece of wr1t1ng f0r f00d by someone who does little else.

If that's what you see on linkedin, then you're spending too much time there. It's the same tripe that fills industry rags and like websites like computerworld.com who even translate this sort of thing into many languages to attract yet more little flies into their web.

This guy is what they call an opinion maker. And if this is the sort of opinion he's making, he can be safely ignored. All the more so because the people who would find what he has to say interesting (brass of various stripes) wouldn't know the good stuff when it bit them in the arse and wouldn't understand it either.

That he keeps getting linked on /. perhaps would be maddening if I could still give a flying fsck.

Trait #10 is certainly not (1)

Edis Krad (1003934) | about a year ago | (#44315565)

Humilty

Re:Trait #10 is certainly not (4, Funny)

cold fjord (826450) | about a year ago | (#44316581)

Good administrators keep humility under control. Too much of it corrodes the equipment. There are sensors for it you know.

This is great! (5, Funny)

Anonymous Coward | about a year ago | (#44315595)

I just bought an MCSE or MSCE or whatever you call it and I was wondering how the fuck I could fool people into believing I actually knew what the fuck I was doing.

Problem solved!

Thanks a lot, InfoWorld!

Captcha: "secured"

Re:This is great! (0)

Anonymous Coward | about a year ago | (#44315677)

You should've bought a CCNA & a CCNP if you want to be an instantly hired Net Admin. But you won't stay hired for long if you just buy them. Maybe 30 days before you are canned. I guess you could bounce from job to job and become a manager for some unfortunate company.

Re:This is great! (1)

SuricouRaven (1897204) | about a year ago | (#44317071)

I had the CCNA. Got my my first job out of uni.

On Helldesk.

Been there ever since.

Re:This is great! (1)

Anonymous Coward | about a year ago | (#44316367)

I just bought an MCSE or MSCE or whatever you call it

Oh come on, that can't be so hard to remember.

It's Minesweeper Consultant & Solitaire Expert - MCSE. MSCE makes no sense at all.

Re:This is great! (1)

Jawnn (445279) | about a year ago | (#44316989)

The veteran network admin never bothered with Microsoft certifications. Networks don't run on Microsoft products.

As someone who has worked in IT for 20 years... (0)

Anonymous Coward | about a year ago | (#44315627)

...I've never met a network admin who will ever, ever admit that the fault was in their domain. This especially applies to dealing with BT, who once denied that a problem was theirs, and when it mysteriously went away said "oh, yes, a switch was down, but we had a change freeze so couldn't work on it for a few days.". Note the lack of apology for the previous lie.

Re:As someone who has worked in IT for 20 years... (2)

AK Marc (707885) | about a year ago | (#44315753)

"Came clear during testing" is what I'd get for the "we fucked up, but we'll never admit to it" response.

Re:As someone who has worked in IT for 20 years... (1)

Mockylock (1087585) | about a year ago | (#44317389)

You'll never prove it. You don't have privileges. :D

Re:As someone who has worked in IT for 20 years... (1)

scubamage (727538) | about a year ago | (#44316795)

I have, but usually it's a low grade network admin responsible for being in the trenches. Network architects? Nope, "the company is going in a different direction..." is about as close as you'll get. Hooray corporate america.

Writers block... (0)

Anonymous Coward | about a year ago | (#44315637)

The guys who write the plot for internet have ran out of ideas.... Just like TV did years ago :(

The Nine Traits (1)

Anonymous Coward | about a year ago | (#44315729)

(as mentioned in the article)
1: We already know it's down
2: If we don't know it's down, it's probably not down
3: We will ping and test several times before digging into the problem
4: Believe it or not, we've tried turning it off and back on again
5: During an outage, we're not just staring at the screen -- we're following a path in our heads
6: We calculate subnet masks and CIDR as easily as breathing
7: We do not tolerate bugs; they are of the devil
8: We can read live packet streams and write highly complex filters in our sleep
9: We take big risks all the time

Re:The Nine Traits (-1)

tlambert (566799) | about a year ago | (#44316013)

(as mentioned in the article)
1: We already know it's down

Then put it on the freaking dashboard for the network service outages so we know not to bother you. If you really don't want to be bothered, put up an "Estimated time to resolution, given past history"/"Estimated time to resolution for [characterized as problem X]", or you're going to get the "we know you know it's down; when will the f---ing thing be back up?" calls anyway.

2: If we don't know it's down, it's probably not down

Something is wrong. Maybe a switch under your control, not mine, is partitioning my part of the network, or maybe the new security you put on MAC address per port and didn't tell us about, or told us about but not in a meaningful way, has made it so we can no longer plug in both our laptops to the same switch port. How about something on the dashboard to let us know how many unresolved issues there outstanding at any given time, or over a period of time as a graph. If there is a spike on the graph, there's something you own that'd not sufficiently monitored, and it's still your problem.

3: We will ping and test several times before digging into the problem

You should be doing short interval periodic monitoring at all times. Even if this part of the dashboard isn't visible to us [users], it should be visible to you, and variances above a certain amount vs. a weighted moving average of expected RTT, for instance, should be flagged.

One of the things the DS developers at Apple hated most, which I implemented, was a kernel weighted moving average recorder on user space DirectoryServices response delay calculator. When making external group resolution requests to user space from kernel space (the kernel credential could only hold 16 groups, and a user could be a member of N, which in practice at Apple, ran to several dozens), the resolution task pops up to user space. If user space takes "too long", the kernel declares it non-authoritative until it re-authorizes itself as authoritative. It flagged extra long latencies in DirectoryServices responding to requests as kernel log messages.

They hated it because it laid the blame squarely where it belonged, at the foot of a very difficult problem to resolve, unless you made automatic local replicas of LDAP subtrees.

If some connection is bouncing around like a handball in a box on a paint shaker, perhaps it's time to disconnect that connection and flag it for field service.

4: Believe it or not, we've tried turning it off and back on again

Yeah, this spackles over a herd of issues that should actually be resolved, like packet storms causing the switch to partition off a port as being "too noisy"; adding more spackle each time it happens ain't going to make it stop happening. Maybe (probably) this one is not your fault, but it'd be nice if the person whose fault it is could get notification of the problem of which you are aware and they are not. Be a f---ing team player, folks.

5: During an outage, we're not just staring at the screen -- we're following a path in our heads

Unexpected dead ends can be found with time domain reflectometers. Try documenting, then automating, the steps you are taking so instead of staring at routing tables, you can be over at the fooseball table, or back home in bed.

6: We calculate subnet masks and CIDR as easily as breathing

So basically, you "grok" radix trees? And?

7: We do not tolerate bugs; they are of the devil

Funny: #4 is explicitly designed to tolerate bugs without fixing them.

Let me give you an explicit corollary, using the terminology from the article: there is no such thing as a white whale. Yes, it might be more important from a business perspective to be back up than it is to track down the specific root cause of the bug, but pretending it isn't there before you even get to the step where you "don't tolerate it" is not doing you any favors. Reverse the order, and you will, over repetition, inch your way toward an eventual understanding of the bug.

8: We can read live packet streams and write highly complex filters in our sleep

Yeah, some of us can read C code of a uniform style at 9600 baud or higher, too. We understand you aren't having a CSI moment or are trying to impress us. Is this intended to target other technical types, because if so it's failed: we all have our seeming "hoodoo", and are happy to let you have yours. Unless this is a message to the suit looking over your shoulder while you are doing it (it was not posted to slashdot originally, after all), save this number for something more useful.

9: We take big risks all the time

Yeah, not so impressed by the lack of virtual testbed tools that I'm willing to just give a pass on this one. Yeah, it happens because it's a fact of life right now, but it's really questionable as to whether or not there should already be software models for all the hardware you are using so you can vet the changes against the software model and avoid the most fatal of errors. Watchdog power cycling and config state reversion after a loss of connectivity seems like it might be a good idea here, as well.

Re:The Nine Traits (1)

pla (258480) | about a year ago | (#44316725)

They hated it because it laid the blame squarely where it belonged, at the foot of a very difficult problem to resolve, unless you made automatic local replicas of LDAP subtrees.

A shortsighted and arbitrary kernel limitation does not really count as the fault of the downstream devs who get to suffer for it. As you point out, your "solution" actively encouraged your devs to make a completely wasteful userspace cache of something the kernel should just provide transparently; since you already know this solution, and apparently had access to the kernel code - Why didn''t you either fix the limitation, or if not possible, just build the damned cache yourself and serve high-latency requests from it, rather than getting into a blame-game with people who (reasonably) just expect LDAP to work as advertised?


Yeah, this spackles over a herd of issues that should actually be resolved

No one except the network admin gives the least damn about "why" it happens. They want email (or more accurately, they want MSN and Fox Sports) back up, and they want it back up NOW. Telling the CIO that you knew about an instant likely "fix" but chose to leave the company crippled for half an hour will generally get you escorted to your car.


Overall, I agree, TFA reads like "we rock because..." self-congratulatory masturbation. But some of your responses (except the last one, interestingly) make it sound like you work for a company with a near-infinite IT budget and a sane CIO. Those count as an extreme rarity in the real world, never forget that.

Re:The Nine Traits (1)

NatasRevol (731260) | about a year ago | (#44316957)

2: If we don't know it's down, it's probably not down

Something is wrong. Maybe a switch under your control, not mine, is partitioning my part of the network, or maybe the new security you put on MAC address per port and didn't tell us about, or told us about but not in a meaningful way, has made it so we can no longer plug in both our laptops to the same switch port. How about something on the dashboard to let us know how many unresolved issues there outstanding at any given time, or over a period of time as a graph. If there is a spike on the graph, there's something you own that'd not sufficiently monitored, and it's still your problem.

9 times out of 10, it's an actual user configuration issue. And when the network guy points out that you screwed up, not him, after you blamed him, you're going to get the attitude you deserve.

Re:The Nine Traits (0)

Anonymous Coward | about a year ago | (#44318227)

"1: We already know it's down"

We're arrogant, and you could never tell us something we don't know because you're a mere user.

"2: If we don't know it's down, it's probably not down"

We're never wrong. It's all your fault.

"3: We will ping and test several times before digging into the problem"

Because we're arrogant we'll take you at your exact words and do no more and no less than investigate exactly that. When you say the server is down we'll ignore the fact you're a user and don't know the difference between the server being down and the service you want to use being down and tell you you're wrong if we can ping it. We wont bother to investigate if it may in fact be the service that's down and will condescendingly tell you it's your fault again for not telling us it was the service not the server because we expect you to know as much about our job as we do and you should know the difference even if it's not your job, and it is ours.

"4: Believe it or not, we've tried turning it off and back on again"

We're a grumpy bunch and we hate jokes.

"5: During an outage, we're not just staring at the screen -- we're following a path in our heads"

Excuse us. We're a bit slow.

"6: We calculate subnet masks and CIDR as easily as breathing"

We talk shit.

"7: We do not tolerate bugs; they are of the devil"

We know nothing about software development and refuse to accept that bugs are an inevitable fact and we gloat about how much more awesome we are than developers even though many developers know more than us because they know how the machine, how networks and so forth work at a much lower level than we do, because they wrote the software we use in the first place. This is why we get paid markedly less than developers, but at least we still get to pretend we're far more awesome if we find a bug.

"8: We can read live packet streams and write highly complex filters in our sleep"

We talk more shit.

"9: We take big risks all the time"

Like, sometimes, every once in a while, we walk to work, and when we're really daring we raid high level dungeons on WoW. We also sometimes reboot the server you're working on and just figure "It'll be okay, no one will be using it at the moment".

Clarifying #4 (4, Insightful)

chill (34294) | about a year ago | (#44315737)

Notice he said "turning an interface on and off", not "rebooting".

Nothing says "I'm a noon and came from a Windows world" like rebooting a switch or router to fix a problem.

Logs on those devices are in memory. Rebooting clears the logs and you then can't troubleshoot. If you can't troubleshoot, you'll never know what really happened. If you don't know the root cause of the failure you can't prevent it from happening again.

Re:Clarifying #4 (4, Insightful)

mysidia (191772) | about a year ago | (#44315811)

Logs on those devices are in memory

Although they should be on the NMS and syslog servers too... and many Juniper M/T series devices, which are very frequently used by large carriers for core and edge routers, have a hard drive mounted on the /var filesystem, that the routers' syslogs get written to.

Rebooting sounds like an act of desperation. It's almost never the cleanest way, and it almost always incurs additional downtime, causes more outages, or further lengthens the network downtime --- since you require 3 to 5 minutes for a reboot, then maybe 3 to 5 more minutes for routing protocols to reconverge.

And god help your soul, if you reboot a Cisco device such as a 72xx or Catalyst 65xx running BGP with 3 or 4 copies of a full table, and with 8 or 9 total peers.

That one weak moment, when reboot was chosen may result in 60 to 90 minutes of trying to coax the network back up gently.

Re:Clarifying #4 (1)

rusty0101 (565565) | about a year ago | (#44315851)

Or that 6509 with a sup720 hosting 25 T3s for some 2000 remote sites using OSPF as your routing protocol. (All in Area 0 of course.)

Re:Clarifying #4 (1)

LordLimecat (1103839) | about a year ago | (#44316573)

Nothing says "I'm a noon and came from a Windows world" like rebooting a switch or router to fix a problem.

Sometimes that IS the answer: Boot up triggers POST, where it may plainly say "your flash is hosed". Or perhaps a voltage spike threw things into a "bad place", and the solution is to do a full power-cycle to bring everything back to where it was. Perhaps the issue only occurs on boot up, and you need to turn it off and on to see the problem happen (such as when someone left the router in 0x2142 mode).

Rebooting is usually not the answer but sometimes it is. You can stubbornly insist that the answer is in the logs all day, have fun troubleshooting a popped capacitor if that is your methodology.

Re:Clarifying #4 (1)

Penguinisto (415985) | about a year ago | (#44317255)

Sometimes that IS the answer

A lot of times, it's the only way to clear up an ill-considered typo + "write mem".

Re:Clarifying #4 (1)

LordLimecat (1103839) | about a year ago | (#44318075)

You mean "copy running startup". "write mem" has been deprecated since before I started working on Cisco 2600s back in 2004 ;)

Re:Clarifying #4 (4, Insightful)

NatasRevol (731260) | about a year ago | (#44316975)

If you don't have a separate syslogd system, you're not doing your job well.

Re:Clarifying #4 (1)

Mockylock (1087585) | about a year ago | (#44317363)

A good admin uses syslog servers.

The BOFH knows the real traits of seasoned admins (5, Funny)

dkleinsc (563838) | about a year ago | (#44315985)

1. A complete disdain or hatred of lusers.
2. A collection of blackmail materials.
3. Homicidal rage.

Re:The BOFH knows the real traits of seasoned admi (1)

Penguinisto (415985) | about a year ago | (#44317265)

Note: #2 requires systems access to the email server (or spam filters) as well as fileshares - something a pure network admin wouldn't have.

Re:The BOFH knows the real traits of seasoned admi (3, Informative)

dkleinsc (563838) | about a year ago | (#44317547)

No it doesn't: packet sniffers.

Veteran network admin trait No. 10 (3, Insightful)

mwvdlee (775178) | about a year ago | (#44315997)

Veteran network admin trait No. 10: We like writing about how awesome we are
Every once in a while we realize we're just glorified factory workers operating machines we know very little about. It is at times like these when we need to point out to ourselves how our brains are amazingly special for being able to navigate a maze of possibilities (completely unlike a delivery boy) and how we can handle the incredibly complex mathematics of subnet masks. It is to remind ourselves that restarting something and waiting for it to fix itself is actually a mental process reserved only for the enlightened few and we must publish such so the world may know of, though not quite comprehend, the extent of our genius.

Re:Veteran network admin trait No. 10 (0)

Anonymous Coward | about a year ago | (#44316087)

Admit it, you hate your admin just because you can.

Re:Veteran network admin trait No. 10 (1)

mwvdlee (775178) | about a year ago | (#44316189)

I work for a small company of a few dozen people where I do the admin work on the side (doesn't take more than a few hours a month), so I don't have an admin, nor do I hate myself ;)

Re:Veteran network admin trait No. 10 (1)

LordLimecat (1103839) | about a year ago | (#44316607)

Thats probably because youre dealing with a single server running Windows in a very simple setup, which is notoriously easy to get away with having a part-time admin for.

The problem is, of course, when things arent working right. Despite what I said above about restarting sometimes being the answer, it USUALLY is not, and the other poster was right-- randomly rebooting DOES mask the issue most of the time. Examples:

  • Print spooler crashes once a day-- just reboot it, or troubleshoot and find that you actually have a corrupt driver which crashes things once a day when it attempts to do its awful auto-update mechanism?
  • Network seems slower than usual: ignore things and assume its in the user's head / reboot the switch, or investigate and find out someone created a switching loop on the network?
  • Same as above: Ignore things, or investigate and discover that users are storing live PST files on your network share, which not only brings the network to its knees but also runs a huge chance of corrupting the entire archive if not addressed?

Ive done consulting for many years, and you're right that a lot of the time you can get away with having an "off-admin"-- in fact, depending on budget it MAY be optimal. Just recognize that youre going to plaster over a LOT of issues, and that things may blow up / require a real admin to fix things one day, and depending on how much plaster it is, it may be an expensive fix.

Re:Veteran network admin trait No. 10 (1)

NatasRevol (731260) | about a year ago | (#44317023)

You expect a part time admin to be able to diagnose a switching loop?

Re:Veteran network admin trait No. 10 (0)

Anonymous Coward | about a year ago | (#44316505)

I has subnet maths, I r awesome. @_@ As a seasoned software developer, I am firstly annoyed that people make a career out of this, I am secondly annoyed if they make even half of what I do.

Re:Veteran network admin trait No. 10 (0)

Anonymous Coward | about a year ago | (#44317049)

Ah yes. Nothing more arrogant than the network admin, except all the coders. Most of whom still google all the snippets of code they work on.

Re:Veteran network admin trait No. 10 (0)

Anonymous Coward | about a year ago | (#44317101)

I have yet to meet a software dev at any place I have worked who doesn't first blame his code not working on there being a mysterious network problem, as if we've activated a little known SPITE protocol in the switches and routers that is specifically singling out only his app from all the other network traffic and deciding to fuck with it just to be a dick.

Re:Veteran network admin trait No. 10 (1)

Tim the Gecko (745081) | about a year ago | (#44319485)

Agreed, the "super powers" aren't that super. On the other hand there are some very simple things that many people screw up. For example, people often get inbound and outbound confused, and forget that "me to X" and "X to me" are often separate problems. Asymmetric routing, where networks hand packets off early to the network with more detailed knowledge of the destination, is a great thing, but many people don't get it. Traceroute is a great tool for getting information, but the return path trips people up all the time. Here are some great notes [nanog.org] on interpreting it.

You are not So Special Snowflake (0)

Anonymous Coward | about a year ago | (#44316031)

"Veteran network admin trait No. 1: We already know it's down Few things are more annoying than having your phone blow up with automated alert messages from your monitoring systems, scrambling to dig into the issue, only to be continually bombarded with humans texting/talking/emailing/calling with the same "Is x down?" question, or even worse, "The network's down!" If the outage is significant, we already know about it, and we are trying to work on it as fast as we possibly can. Continued attempts to deliver elderly information will only impede that effort."

Said as an admin which only looks at its own patch of land. The reality of production, is that you have router, network, firewalls, backends server, front ends server, and so forth. Pray tell us how people know its is your network and not many of the other point of failure ? People asking "is x down" are not a symptom of elderly info, it is a symptom y
"Veteran network admin trait No. 2: If we don't know it's down, it's probably not down Conversely, if we get a message claiming, "The network's down!" yet we have not been notified by any monitoring system, then the problem is almost certainly the complaining user in question." yeah right, because all those system are quasi instant and you are not distracted by something else, as a perfect human. "No, it isn't -- reboot your laptop." I see. You are an idiot.

"If we begin looking into a problem, especially across a WAN or long-haul link with several providers in the middle, we will reserve judgment for the first several minutes. This is because these connections are subject to the vagaries of their path, and connectivity problems can come and go like ghosts in the night. A fiber WAN link that was stable a minute ago but is now exhibiting 30 percent packet loss will more than likely fix itself in short order." I will tell you the same you told the poor user previously : "reboot your router". If not reboot your brain. A network which exhibit such behavior has problem.

"Veteran network admin trait No. 5: During an outage, we're not just staring at the screen -- we're following a path in our heads" nope you jsut said before : first you wait time to wait that it fixes itself in short order. You damn said it yourself.

"Veteran network admin trait No. 9: We take big risks all the time Network admins tend to work on many remote devices. Unlike server admins who can pull up a console if the server is otherwise inaccessible via the network, we usually have no such luxury. This means that changes we make to certain devices carry with them the ever-present threat of the loss of connection. Basically, we're always a missed keystroke or two away from making a problem worse, or causing a big problem where none previously existed. " I have always been one typo away from destroying a whole system. Read : destroy and need very long (8+hours) backup to restore, not make temporary inaccessible. You think you are so special snowflake ? We backend admin can be a keystroke away to *destroy* a whole DB inadvertently. Go cry me a river.


Sorry but the whole reek of "i am so special and important". You are not. You are a cog. In fact, as a "plumber", you might make temporary mistake, but you are nowhere near as dangerous as backend engineeer and administrator.

Re:You are not So Special Snowflake (0)

Anonymous Coward | about a year ago | (#44317775)

Yeah, he's a cog, but don't tell him that. He might want to do non cog things he doesn't have the abilities for, then we're all screwed. Keep telling him he's a special little snow flake, so he stays happy and out of your hair.

just do it (0)

mythix (2589549) | about a year ago | (#44316047)

so, you are just doing your job and would like some free coffee for it?
be happy with your paycheck like everybody else who is doing their job every day, you're not special....

The unspoken trait (1)

GeekWithAKnife (2717871) | about a year ago | (#44316115)

Let's not forget that antisocial trait where we frown on other lesser knowledgeable mortals as we stroke our massive long white beards.

Two traits : Cooperative and uncooperative (4, Interesting)

EmperorOfCanada (1332175) | about a year ago | (#44316263)

Cooperative and uncooperative are the two primary traits of network admins. Seeing that most network admins are ignored when everything is going well and cursed when things are going badly it quickly becomes obvious to most network admins that saying no to most requests becomes a survival trait. But this often means that a network admin starts actively interfering with the actual business (unless the business is network administration)

So take your average non IT company. The sales people want to switch from Blackberries to iPhones and Androids. The network admin has 4 certifications from RIM and recently set up a huge Blackberry server. Plus this particular admin hates all things touch screen. The usual scenario will be that the admin will pull out 9 whitepapers showing just how secure Blackberry is as compared to iOS and Android. Using techno babble he will explain how hackers will be running the company in under a week. The salesman on the otherhand say that they look like tools from the 90s with their blackberries. The network admin wins the battle but then he starts to get nervous as he notices traffic on his BB server is nearly zero. A quick investigation shows that nearly every Blackberry sits in a drawer and the Salesman have gone out and bought their own phones and swapped the SIM cards. The network admin sends out a memo saying this is against corporate policy which is ignored. His attempts to get a salesman fired(to set an example) for violating security fail. He then notices nearly everyone is using gmail instead of his highly secure MailMaster2000. Then sends out a memo indicating that this is against corporate policy. He then implements a 30 day mandatory password rotation. Internal file server traffic nearly drops to zero because everyone switched to dropbox. He then sends out a memo that dropbox is against corporate policy. He then starts blocking sites such as reddit and he notices that network traffic drops to nearly zero. But walking by a sales person's office he notices that they are on reddit. So he investigates and finds out the entire sales team has bought mobile data plans. He then sends out a memo saying that private data plans are against corporate policy.

Then he comes to work only to find a contractor in his office. The contractor is there to "rationalize" IT seeing that after the IT guy insisted that all apps be developed for BB first the sales people have gone out on their own and developed 3 smartphone applications that have increased sales by 80% and that promotions via Reddit have sent corporate website visits through the roof. The company now works with clients via dropbox much more successfully than with the sftp system that merely served to confuse before. With mobile dataplans the salesforce has become much more effective.

Now the IT guy is left filling out a resume where his two best features are many Novell certifications and many Blackberry certifications.

IT people shouldn't cave into every whim of the week but I have seen so many that are stuck in the thinking of whatever year they became head of IT. IT is just one tiny department in so many companies yet I have seen IT somehow be able to treat senior managers of other departments like children. Seeing that they aren't children they often discover the virtues of outsourcing. The key benefit of outsourcing being that if the people they outsource to try pulling any crap they can be dropped in a second.

Re:Two traits : Cooperative and uncooperative (0)

Anonymous Coward | about a year ago | (#44316827)

You obviously do not know what a "network admin" is. In this article and context, network admin is the person/group that designs/implements/maintains the actual network - routers, switches, Internet connectivity, phone connectivity for corp and remote offices, data center network(s), network security, etc.

This article is referring to network admins in the larger, more technically advanced companies. Not some jack-of-all-trades in some podunk, 5-person company wanting to use technology from 2005, thinking it's the latest, greatest thing.

A network admin is not a Windows systems administrator.
He certainly isn't a Novell admin.
He is not the mailman.
He is not the webmaster.
He doesn't build out FTP servers, set password policies, etc.
He doesn't mess with cell phones or pagers.

He certainly doesn't handle Blackberry devices, plans, or apps. Definitely least not in 2013. Let go of your bad memories from 2005 and move on.

Re:Two traits : Cooperative and uncooperative (1)

Anonymous Coward | about a year ago | (#44317143)

Yeah, just try having employees do that in any company that has an alphabet soup of government regulations to comply with like SOX/PCI/HIPAA. Audits are a bitch.

Re:Two traits : Cooperative and uncooperative (0)

Anonymous Coward | about a year ago | (#44317295)

Nice. A lot of managing an IT career after age 35 is about how to avoid being "the right man for the wrong decade".

Re:Two traits : Cooperative and uncooperative (0)

Anonymous Coward | about a year ago | (#44317575)

You lost me at reddit, jerk off.

Re:Two traits : Cooperative and uncooperative (1)

sjames (1099) | about a year ago | (#44318535)

A month later their marketing department is going nuts trying to figure out how their competition manages to anticipate their every move...

Trying to add galmor to regular job. (1, Insightful)

140Mandak262Jamuna (970587) | about a year ago | (#44316279)

The author is trying to make a big deal about how great the network admins are. Can't blame him for feeling like working in a chain gang from some Alabama prison, deployed on contract in a quarry, with the tool chest consisting of just one sledge hammer. Looks like most of the time they ask the user to reboot their machines (trait 2) or reboot network switches (trait 4) or wait for the problem to solve itself (trait 3). Other traits seem to be putting on a brave face, telling themselves how smart they are. I am sure you could find nine such traits for plumbers clearing blocked sewers too, except they can't reboot the sewers nor wait for it to unblock itself.

Regular jobs are the best (0)

Anonymous Coward | about a year ago | (#44318241)

Every job that you make a living can (and perhaps should) be considered glamorous -- even if it just sweeping the streets or emptying the trash bins.

I love hearing different professionals describing their professions in detail; describing the small things that can go wrong and the oddballs cases they have encountered.

Who cares? (0)

irp (260932) | about a year ago | (#44316299)

I did of course phrase the subject to maximize the provocation :-) but my point is, while I really like the plumping to work, I don't really care about "the nine traits of a veteran plumber". For me a network admin is (believe it or not) a LOW TECH job, using of-the-shelf standard components (as any other admin/technician/mechanic). If my mechanic has to stare blankly while mentally figuring out what is preventing my car to start, I will let him do that in peace - because I assume he knows better than me how to do his job.

I have NO doubt that network administration is an intriguing job, with its share of problems to solve. But as long as the down time is counted in hours and not days, I prefer them to work invisible behind the scenes, not bothering me and hindering me to do my job.

Don't get me wrong, I know that a good admin is well worth his pay. But also I love the cleaning personal, especially when they don't clean while I'm working :-)

I guess I was hoping to gain a little insight in how the admin job is done - I dunno, maybe adapt some of it to my little (tiny) home network. But most of the points was fairly obvious - yes I also reboot my router. No of course I can't calculate subnet masks/read packages/write filters in my head. Is it impressive he can? Slightly, but not much, skills like that develop by regular use. I would be more surprised if he had written "we just google it" :-)

Re:Who cares? (2, Interesting)

Anonymous Coward | about a year ago | (#44316643)

Honestly, I can say if someone came in for an interview with this type of an attitude, I would tell them to pound sand.

I'm a network admin; not all of us act like tools like that. If your network admins are getting calls that something is down, then you need to redefine your processes and stop giving out your direct line like a toolbag and instead have users contact the help desk.

I'd have called that article "9 traits of a crusty, worthless network admin". I'm seriously appalled someone even wrote that crap. :(

And FYI, when I interview people, the first thing I do is make sure they're willing to admit they may be at fault and they know how to Google something. I generally will give them a situation of misconfiguration on our part and ask how they communicate that to a user. Be honest. You're human, you screwed up, and apologize for the inconvenience. I generally also ask "how do you solve a problem you've not run into, and how do you answer someone that asks you a question you don't know the answer to right off hand?" I have too many people say they've ran into every problem and they know the answers. There's no shame in admitting you don't know something and you need to Google the answer (or look it up on Cisco's/Juniper's/VendorX's website).

Wow (0)

Anonymous Coward | about a year ago | (#44316351)

Slashdot "articles" have finished scraping the bottom of that barrel and have commenced drilling operations. What a joke.

That article makes me nostalgic (1)

DigitalSorceress (156609) | about a year ago | (#44316423)

I left the world of systems administration and network management in 2006 when I took a position as a software engineer... I've kept my hands wet with managing my personal web servers and those of a few hosting customers, but I do admit to missing those sysadmin/network admin days.

The statement "use it or lose it" really is true - I still have all the troubleshooting skills and nowadays, a LOT more insight into the software side of things, but I'm sure I'm rusty as hell with IOS and such. It's also a significantly more hairy security landscape today than back then, so I guess I can reminisce about "the good old days" from the safety of my bunker - er, home office.

Alternate Trait 1 (2)

bobstreo (1320787) | about a year ago | (#44316899)

If you are good, you know when the network "isn't right" which is much better than UP/Down.

This includes knowing things like a switch port has lost duplex on an ethernet connection,
there is a broadcast storm in a building/site, one of your redundant links is down.

Being right much much more often than being wrong in diagnosis/troubleshooting is also a good trait.

Trait 10: Able to work overnight alot... (1)

Anonymous Coward | about a year ago | (#44317193)

Goes for System Admins too: Used to commonly having to work all night to do routine maintenance then come in the next day. While the rest of IT/Development is sleeping these guys are up all night. There isn't much glamour being on conference calls at 3 AM.

I am a DBA and I generally like to be certain I need to contact an SA or Network Administrator before I do, given the insane hours these guys work. My ultimate fear is that they will quit then Ill be forced to do their job. So I need to be nice to them.

A Net admin's job.. (1)

Mockylock (1087585) | about a year ago | (#44317331)

95% is comprised of defending the network. If it's been up and running, no changes have been made and no equipment has died... chances are, it's NOT the network. Go check group policy.

Re:A Net admin's job.. (1)

JDG1980 (2438906) | about a year ago | (#44318193)

95% is comprised of defending the network. If it's been up and running, no changes have been made and no equipment has died... chances are, it's NOT the network. Go check group policy.

In most small and mid-size companies, group policy is part of the network administrator's job. Your notion that the network admin should only be handling the IT plumbing is something that's only going to work in large corporations. When you can't afford a massive IT department, people have to wear multiple hats.

missing trait (1)

prgrmr (568806) | about a year ago | (#44317763)

insufferable prima donna

You know, when you capitalized "Veteran" (0)

Anonymous Coward | about a year ago | (#44318323)

I perked up because I thought it had something to do with former military..

He's a "Network Admin"? (1)

Kimomaru (2579489) | about a year ago | (#44318475)

The article's author uses the "we" a lot, which strikes me as awkward that he's including himself in the group of "Network Admin" when he writes things like;

Veteran network admin trait No. 1: We already know it's down
Veteran network admin trait No. 3: We will ping and test several times before digging into the problem
Veteran network admin trait No. 5: During an outage, we're not just staring at the screen -- we're following a path in our heads
Veteran network admin trait No. 6: We calculate subnet masks and CIDR as easily as breathing

I guess the term "veteran" is subjective. He should add a new trait;

Veteran network admin trait No. 10: We write articles like we're fulfilling a quota

SNNNOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOORE.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>