Update to The Magic Cauldron 81
Eric S. Raymond wrote
in to tell us that he has updated The Magic Cauldron
(his essay on economics and Open Source) to contain
an appendix on common arguments for keeping device drivers
closed (Pay attention Creative Labs!) He also says "The argument turns on the fact that drivers are small pieces of code, easy
to disassemble if need be. This argument would be considerably strengthened
if I could point readers at a working set of tools for disassembling Windows
drivers into recompileable source (or even just assembly) code. I would
appreciate pointers to any such tools."
Re:Did someone say Sourceror! (Score:1)
harder than it sounds. (Score:3)
Even without using something like this, disassembly of programs cannot be 100% correct. There is a lot of information needed to reconstruct a program that is thrown away by the compiler. For example you can't determine where a jump table (switch statement) or function pointer (virtual function) will lead to without actually executing the program and then it's not possible to cover all of the cases that could occur. IDA does a pretty good job of guessing though. Also IDA has hacks if it is looking at C code like looking for procedure frame headers (push ebp; mov ebp, esp) and termination code (pop ebp; ret). My solution to correct disassembly is to use emulation. This insures you never have an incorrect interpretation, but you will miss code that is never executed, and this can be important for understanding how errors and exceptions are handled.
Just because you can dissemble something doesn't mean you can understand it. It requires a huge amount of time to try to comprehend uses for unnamed memory locations and layout of data structures. In the case of a driver this is frustrated by the fact you are dealing with a lot of hidden code and data (the logic on the hardware). Not all drivers are "small pieces of code." Having looked at several vendor's Direct3d driver source code I can tell you each was extremely large and complex. Writing the driver with full knowledge of everything is hard enough, but disassembling one and understanding it would require super human willpower. There are all sort of commands that will do different things depending on that state of the card, so you have to understand what previously executed code was important to getting the card into that state.
Re:Wrong approach (Score:1)
For the open source community, this is a different story. I occasionally dream that the entire computer buying community suddenly gets wise and starts asking for real change in the industry, but I'm not holding my breath.
Re:categories (Score:1)
Re:An expensive pain in the ass ... (Score:3)
I was thinking in the context of drivers for things like new generation 3d boards, which are not reverse engineer-able in any useful sense.
You don't really "port" a 3d video driver from Windows to Linux, you write a brand new one, basically from scratch. Source code would be nice, but you still need the docs plus actual human beings to ask questions of. (Yeah, glide helps)
3Dfx has done a great job of this, but it took a some amount of liason work (==$$$) on 3dfx's part to make it happen. Every hour the 3dfx engineers spend talking to the linux driver guys is an hour a) they get paid and b) they aren't working on something else. It was probably worth it for them (hey, I bought one) but it was an "allocation of resources" problem just like in sim city.
Another reasons open sourcing drivers might cost a company : The drivers for 3d boards are areas of competition (the drivers can be as important as the chip design for producing good benchmark results)
Again, I _like_ the idea of an open driver development process. I bought and very happily use a 3dfx Voodoo3/2000 because of it. I hear nvidia is doing a great job as well.
-cks
ESR for got another good point (Score:2)
Companies like Matrox and Adaptec have, as far as I can remember, been very good about supporting even their oldest devices with drivers and jumper settings and such. As time goes on they accumulate more and more stuff that their customers expect them to support just as well as they ever did. Opensource fixes their growing legacy support problem. They don't have to worry about weather MS will include their driver in the next OS release, it will get ported to the next version of Linux almost automatically. The only way tht would not happen is if there doesn't exist anyone who is using that card anymore. If that happens, who cares if the driver dies?
Re:Time to market argument (Score:1)
The more basic argument he is making is: Hardware vendors don't want to be in the software business, particular in the boring part of it that's about porting drivers to x, y, z, and w random architecture. If they open the specs and other people can do the porting work for them. There's no reason that has to touch on any wacky proprietary innovation of theirs.
Fundamentally, having proprietary stuff at the interface level is silly. More to the point, it doesn't much happen. That, I think, is the basic argument here.
Gravis Ultrasound (Score:1)
infact its not uncommon to see things like requires pII 400 and 1mg GUS
of couse, sound blaster took over, and everyone knew the low level specs for that...
_
"Subtle mind control? Why do all these HTML buttons say 'Submit' ?"
duh.. (Score:1)
Re:An expensive pain in the ass ... (Score:1)
You say that it cost them resources. How much will it cost them when no one buys their hardware? And you are forgetting the bugfixes and enhancements they will get back. Give and take, you know..
Funny that you mention 3dfx. I own a VooDoo 2 card, and it has given me much joy. But my next card is not going to be from 3dfx. Why you say,
And yes, the other cards are technical superior also. 3dfx is going to lose... bigtime.
Re:Open Source Hardware Movement???? (Score:1)
Sourcerer (Score:1)
Oooh, Sourcerer. I think I remember that one. Was it part of Glen Bredon's Merlin assembler package? (Merlin on C64, Big Mac on the Apple II -- almost the same program.)
fits nicely.. (Score:1)
:)
crackers are not warez doods (Score:1)
Re: (Score:2)
starting point (Score:3)
mammon_ [eccentrica.org]
Disassemblers for the X86 (Score:3)
I did find "the one true" (at least to me) disassembly tool, IDA by http://www.datarescue.com. It is an interactive disassembler that does background disassembly and lets the user identify additional code and data sections (and correct the mistakes
There are only two things about it - it is payware and no source code is available. And it is virtually impossible to find warezed copies on the net. I paid for my copy and I'm still happy with IDA (in fact, I was happy with the trial version and found it well worth the money to upgrade). The copy protection seems also to be relatively hard, at least from what I found (or better, didn't find) by looking at it.
-max
ESR getting defensive? (Score:1)
The fine art of reverse engineering (Score:3)
The second story was with a 68000 monitor and debubber that I had to write a disassembler for in order to move it to a 68010 SBC. The disassembler was hand-made by me.
The third involved the original 8052 BASIC52 that Intel had produced. There were some bugs and lacking features in it, so I wrote a small assembly routine to dump the contents of the chip out its built-in serial port. The 8052 output was captured on an IBM PC, and I wrote a disassembler in MSC 4 to put it back into 8052 assembly language. I then fixed the bugs and added the features I needed, after which I programmed it back into an 8752 for further development. I sent the code back to Ciarcia, and I even got a little note back.
But the biggest tool I used was V Communications Sourcer [v-com.com]. I got started using it to reverse IBM PC ROMS to hack the drive tables, and then started to add little features and fix little bugs. I used it commercially to reverse engineer the IBM RTIC communications card DOS drivers. I did that so we could then port the card to Novell Netware 2.15 as a VAD (IBM had no plans at the time to port it, and we needed it to provide a Bouroughs data link between a Netware server and a Bouroughs mainframe). Sourcer was, and still is, the cleanest disassembler on the market, capable of giving you back highly commented source code from binary files. I haven't updated my license lately since I prefer open systems to Windows, but if I had to do that again I would go back to Sourcer.
The point is that reverse engineering is equal parts art and science, and if you need it bad enough and know assembly well enough, you'll find and/or build the right tools for the job. ESR needs to calm down a bit.
Re:ESR getting defensive? (Score:1)
disassemblers (Score:1)
I don't understand the logic (Score:2)
My argument has been that it's unlikely anyone will reverse engineer your hardware from your software, but perhaps this is not the case?
Metalab (Score:1)
Yeah, since 1994 about 2 drivers a year. (And I know, if you look at the kernel source it's a lot different, but it's so much easier to follow a link.)
Good copy protection...? (Score:1)
I get the funny feeling that that wouldn't work to well...
"HERE? Nope, no keygen, not here, nuh-uh. It's, uhm, important debugging algorithitms that don't disassemble well. Now go away."
he doesn't mention the classic argument (Score:2)
"there once was a big guy named lou
Open Source Books? (Score:2)
1) Character_One Meets Character_Two
2) Character_One Loses Character_Two
3) Character_One Gets Character_Two We should try to flesh the plot out by August 20th, Aiming for a Version 0.1 publication by January 1st. Check The site at http://taz.eng.ua.edu:1138/crutcher/yals [ua.edu]
-Crutcher
When are consumers going to get mad??? (Score:4)
1) Decrease the life of the product, shortening support obligations.
2) Hide any shortcomings to their hardware product, saving on warranty obligations.
3) Forcing the consumer to upgrade by preventing product 'enhancments', or fixing 'problems'
4) Reveal design shortcomings to the competition.
Some wise man said that a there are two reasons someone does something. A good reason, and the _real_ reason. The arguement that it protects proprietarty technology is in most instances completely false, as many chip makers release 'reference' designs to many hardware manufactures, who in turn make only small changes to the basic design. And any valid 'proprietary' enhancements there may be are quickly undone when the next competing product hits the market. So companies are left with the real reason - less hassle and more money.
I do not think that is enough that you argue politely the benifits of open source. Consumers are going have to get tired of being on the endless upgrade mill of expensive hardware and the expensive bloated code to use it. How much power do you need to write a letter or balance a checkbook?
Maybe then we would see real innovation in the computer industry instead of creeping featurism.
The only way that you are going to see open source from a manufacture is if it is legislated by congress answering the demand of angry consumers who catch on. IBM's Aptiva MWAVE is a classic example of the above. Cut and paste the link below into your browser.
http://members.aol.com/mbs1058/mwave/classact.h
Time to market argument (Score:2)
However, even this is not a real excuse - if you have a situation like this, publish as much as you can, and guard your most treasured secrets if you want to be that way.
For example, Matrox released the specs for their graphics G200 cards, except their triangle setup engine (for whatever reason) and they've also released most of the specs for their next gen G400, even before it started shipping!
Re:harder than it sounds. (Score:1)
Check out +fravia/+HCU (Score:2)
is all about reverse engineering (cracking). It isnt just a simple how to crack (software crack, not bad "hacker") site, he really is interested in reverse engineering. There are even pages about Linux cracking. If his site is down just search for +HCU or +ORC along with +fravia. When I was into that type of stuff, before I got into linux, I used w32dsm, which there are shareware version s of it, but just as I stopped doing that IDA became really big. IDA seems to be better, but W32dsm worked well. And for a windows debugger, you can't beat SoftICE, by Numega. No other debugger for windows even comes close.
Re:harder than it sounds. (Score:1)
And EZIP is different than PKLite how? If a program was PKLite'd one would only have to run it in a debugger until it had unpacked itself into memory then dump the memory contents. Eventually, tools such as UnPKLite were written to automate the process. Compressing a file just adds one more level that will take all of 10 minutes to eliminate.
Even without using something like this, disassembly of programs cannot be 100% correct. There is a lot of information needed to reconstruct a program that is thrown away by the compiler. For example you can?t determine where a jump table (switch statement) or function pointer (virtual function) will lead to without actually executing the program and then it?s not possible to cover all of the cases that could occur. IDA does a pretty good job of guessing though.
Also IDA has hacks if it is looking at C code like looking for procedure frame headers (push ebp; mov ebp, esp) and termination code (pop ebp; ret). My solution to correct disassembly is to use emulation. This insures you never have an incorrect interpretation, but you will miss code that is never executed, and this can be important for understanding how errors and exceptions are handled.
You won't miss code. When you come to a branch, take BOTH routes into the code. For ISRs and such, knowing how and when your target processor handles them will allow you to find them easily. It too can be automated.
Just because you can dissemble something doesn?t mean you can understand it. It requires a huge amount of time to try to comprehend uses for unnamed memory locations and layout of data structures.
Here, we agree. However, I've never come across code I was reverse engineering which I didn't eventually figure out. It just takes time, patience, and a good reference library.
Re:disassemblers (Score:1)
and anyone who thinks Finland has no white trash has never been on cruise between Helsinki and Stockholm...
... you were off topic anyway.
Why I didn't write a Mesa driver for RIVA TNT. (Score:1)
Indeed, it always seemed wierd when the program was written in C. However, C is a low level language, and a good programmer can figure it out. As of '99 things have changed a bit. First of all, the complexity of the programs have increased both in size and functionality [ so we can talk about kolmogorov complexity
Now imagine a hardware whose specs aren't available. In the case of RIVA TNT, I was really pissed off and almost going to work on cracking the windows GL drivers open and writing Mesa stuff for it before the free stuff came in. Guess what! I never started it, let's say it didn't turn me on. The real reason, of course, is that it would be almost an impossible task. Although one could argue that the modularity in the hardware design would lead to modularity in the driver code, thus comprehensible assembly stuff. I wouldn't be so jumpy about it, it's still an undecidable problem folks!
Sincerely, I don't think I could accomplish that without knowing card's specs. I recall that it was quite difficult to find out Amiga AGA's specs by disassembling the AmigaOS3.0 ROM. And that was just 2d.
Re:Wrong approach (Score:2)
An awful lot of my friends and acquaintances now list Linux compatibility as not just a contributing factor, but as a deciding one in all of their hardware purchases. This big change, only in recent years, affirms to me your comment that we are indeed a market force now, a fast growing force, and one to be reckoned with.
Re:disassemblers (Score:1)
reverse engineering resources (Score:4)
fravia (east coast) [crazyboy.com]
decompilation page [uq.edu.au]
sandman [cyberspy.com]
greythorne [home.sol.no]
The more interesting question is, given that most of these sites have been around so long, why dont we see more reverse-engineering of software going on? i think the availability of all this information (especially on fravia) weakens esr's argument significantly. regardless of what he says, reverse-engineering is really, really difficult, even for small pieces of code. i dont think he'll convince very many people based on that argument.
As for the appropriate tools, a while ago i found copies of wdasm and softice using ftpsearch (remember, one version of softice was a fully operational time-demo which could be cracked by itself). much of the other stuff you might need that ive seen are freeware or shareware.
unc_
Re:I don't understand the logic (Score:1)
It's better to relase source and win the respect of a community who will then go out of their way to buy your product, than to keep your product closed. Either way, if someone wants to reverse engineer your product, they will.
My argument has been that it's unlikely anyone will reverse engineer your hardware from your software, but perhaps this is not the case?
I can only point to a couple examples where a company wouldn't release specs so people took it upon themselves to reverse engineer the drivers. One recent one would be the GATOS project which reversed engineered the ATI television cards fairly well thus forcing ATI to cooperate in the support of Linux/FreeBSD. The other was the DOS graphic/sound demo group Renassiance who reversed engineered the Advanced Gravis Ultrasound wavetable sound card thus forcing Advanced Gravis to release the low-level card info and a free SDK.
Of course, there's also the arcade emulator scene as an example. They reverse engeneer the game ROMS, usually without access to or much info on the arcade hardware except maybe the make of CPU, and come up with complete working arcade emulators in a week or so. Just take a look at MAME. It's up over 1400 emulated games if I'm not mistaken.
I'm sure if I searched, I could find many more examples.
Just Another AC
Re:disassemblers (Score:1)
W
-------------------
Enterprising hardware manufacturers DV+analog card (Score:1)
Enterprising hardware manufacturers please note: there is currently only
Fast Electronics [fastmultimedia.com] is the only manufacturer of this type of card, viz the Fast DVMaster [fastmultimedia.com] PCI card which has 2 DV inputs, 1 DV output, analog SVHS input and output with both NTSC and PAL support (SECAM extra). The card has no Linux drivers available yet but customer pressure could apparently change this. Requests for Linux drivers are being solicited by Anuschka.Schweizer@fastmultimedia.de [mailto] who is the product manager for the DVMaster. Alternatively, an enterprising competitor might like to enter the market.one DV+Analog video capture card on the market and that product lacks Linux support.
Digital Video capture cards are a limited interim solution because they lack legacy analog video compatibility. A DV and analog capture card, which captures both analog and digital video data, is what many people who buy DV capture cards later realize would be much more useful to them because they'd get backward compatibility with (S)VHS and real-time conversion to Digital Video for their old collections of (S)VHS tapes.
Re:ESR getting defensive? (Score:2)
I gathered that he was saying do this or your potential customers will call you a moron.
As one of those people who influences buying decisions, I can confirm that position. Venders who won't release spec ARE clueless, and I will strongly recommend against them if any reasonable alternative is available.
Wrong approach (Score:4)
Chip vendors don't like this because they don't want their competitors to use their own documentation to produce drop-in replacements for their components. They use trade-secret because it's the cheapest form of protection. However, if they have to open their interfaces to get their chips designed into Linux systems, they will publish their documents and use patents and copyright as protection against their hardware competitors.
Thanks
Bruce
hardware openness database anyone? (Score:3)
their hardware specs and i would rather give it to a company that has open
source drivers. what would be useful in this context would be a web based
hardware database, where find quick info about how open a specific piece of
hardware is before buying one.
like there could be 5 categories:
1 proprietary hardware, company does not give out spec.
2 company allowed people to write binary only drivers under NDA but for free.
3 company releases binary only drivers
4 company published detailed documentation
5 company wrote open source driver or funded/helped in the development
i would only buy hardware in category 4 or 5 but often it is not easy to
find out if the person who wrote drivers got support from the company..
sure, a grep through the linux kernel helps sometimes but that is not
something the marketing departments will consider a threat to the image of
their product.
having such a database where one could directly compare the openness of
various vendors would put some market pressure on those companies..
mond.
categories (Score:1)
4. company releases open source driver (nVidia)
5. company releases detailed specifications (hopefully Matrox, soon)
6. company releases detailed specifications AND has some employees help with development (no one yet that I know of, at least in gfx cards)
Re:hardware openness database anyone? (Score:2)
Bruce
Re:disassemblers (Score:1)
This Sounds Familiar... (Score:1)
Dysprosium
"Frob Your Ooblick!"
Re:When are consumers going to get mad??? (Score:1)
I hope you're not seriously advocating something like this. I can see it now, a computer hardware market as regulated as the Medical Device industry. With clowns like Ralph Nader riding up on top 'directing' hardware manufactuers in how to 'innovate' like good citizens. Costs would skyrocket, the whole industry would devolved into a chickensh*t pack of me-too companies.
It might be good for Linux, because it would cripple everything else.
Re:Disassemblers for the X86 (Score:1)
-
-- Guges --
-
Re:I don't understand the logic (Score:1)
An expensive pain in the ass ... (Score:1)
Re:An expensive pain in the ass ... (Score:2)
Thanks
Bruce
Open Source Hardware Movement???? (Score:1)
--------------------------------------------