What Makes Parallel Programming Difficult? 196
An anonymous reader writes "Intel's Aater Suleman writes about why parallel programming is difficult. ... I was unaware ... that a major challenge in multi-threaded programming lies in optimizing parallel programs not just getting them to run. His analysis is insightful and the case study is very enlightening if you are unfamiliar with parallel code debugging. "
Easy! (Score:4, Funny)
Motherboards rarely have parallel ports these days!
Re:Easy! (Score:5, Funny)
is that things sometimes happen in the wrong order.
Re:Easy! (Score:5, Funny)
The problem with parallel programming
Re: (Score:2)
Difficult Parallel Makes Programming What?
(Prior message was optimized for concurrent throughput).
Re: (Score:2)
That's a different problem, related to the fact that people buy "parallel" but forget to read the part where it says "synchronize". It's kind of like not validating input, or ignoring return values. Not so much a pitfall as a failure to have basic skills.
The problem these folks are having is that they bought parallel but decided it just wasn't optimizing their stuff enough. It'd be really funny if they found they could optimize-away the parallelization, like, by precomputing certain results and using a l
Re: (Score:3)
I ran into this problem myself recently: I wrote a program that interfaced with websites via http, and the problem was that while it was
Re: (Score:2)
Re: (Score:3)
- The nazi grammar parallel
No combination above gramatically words possible of the is correct. You 's drop the should.
Re: (Score:2)
Because its just as difficult as parallel parking?
Re: (Score:2)
Same with serial, PS/2 (mine only had one and using PS/2+USB adapters doesn't work well -- too many pauses and sometimes pause), PCI (not Express), etc. :(
I think they made a movie about this... (Score:2)
"You're just not thinking fourth-dimensionally!"
"Right, right, I have a real problem with that."
Re: (Score:2)
If you've read the article, you'll feel a little smarter today. Even if it was just a good review.
If you haven't heard of Amdahl's Law [wikipedia.org], pay attention. (Since simple.wikipedia.org doesn't have a nice, short explanation, how about a car analogy?)
Amdahl's Law
You're street racing. But there's a school zone in the middle of the course, and you always slow down for the school.
No matter how fast you go on the rest of the course, that school z
Re: (Score:2)
Your understanding of Amdahl's Law isn't quite correct. I guess if you want to use the street race analogy, it's more like you each have your own lane for part of the race, but for another part of the race you have to go single file into a single lane, and if you're parallel to someone as you approach the choke point you have to arbitrarily let them ahead of you or pass them based on some set of rules that may not necessarily be fair. And the goal isn't to win the race but you're all delivering something an
Re: (Score:2)
Re: (Score:2)
I didn't look at the wikipedia page. This is from memory from the parallel programming class I took. Either way the main point is that your analogy was poor.
Re: (Score:2)
Not quite, Amdahl's Law says that no matter how well you optimize your parallel tasks, the one that you cannot parallelize will gobble up your time. I don't think there's a good car analogy for that.
Ask your project manager. He's dealing with that problem on a daily base.
Re: (Score:2)
People are like cars. The faster you drive them, the more hot air you get out their rear end.
Re: (Score:2)
"I don't think there's a good car analogy for that."
Sure there is.
:o) There is ALWAYS a car analogy.
You have yourself a passel of old Dodge Chargers (painted orange for some weird reason), on a REALLY wide road, all tied together through the windows with some good ol' Southern one-inch hemp.
Whichever car has the wrong mix of gasoline to moonshine will drag the whole pack down. To carry the analogy further: if it has real problems it could crash the whole lot.
Re: (Score:2)
While true this isn't due to the problem of not being able to parallelize more tasks. If you want to use the "add more doesn't mean more gets created" metric, use the myth of adding more people to a project to speed it up. Even aside of training and other overhead, you cannot simply parallelize everything in software creation. Let's do a simple example.
There's two tasks that can be ran parallel. An administrative task (like meetings and organizing crap) and the programming task of creating the actual progra
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Importantly, comparisons to factories are generally unwise because manufacturing is 'embarrassingly parallel' -- it doesn't matter if you have to make the whole iPhone in one single step that takes two straight hours and you can't divide it into any constituent part. You can still manufacture as many iPhones a day as you like because you just set up however many manufacturing units which each make one iPhone every two hours, until you have the desired output capacity.
The 'hard' problems to parallelize are t
Re: (Score:2)
1. In manufacturing, the idea is to do the exact same thing a jillion times with the exact same result. Interchangeable parts make different rates of a production easier to deal with. In computing this isn't the case. The assembly line make be identical each of the jillion times, but the data going through it is not.
2. Manufacturing plants are expensive to build. It makes sense for an engineer to spend weeks or even months optimizing the process. We don't have th
Re: (Score:2)
Well of course you'd have a problem if using reals. For four dimensions, you need to typecast to quaternions.
Multithreading some problems is (probably) hard (Score:2)
http://en.wikipedia.org/wiki/P-complete [wikipedia.org]
Re: (Score:2)
I'll reply to you.
You're a bit smarter than me but I think you're saying there are lots of easy tasks that are easy to run on a single linear code thread but are hard to split and recombine with any less loss of time / resources.
Re: (Score:2)
he's saying there are lots of hard problems that are easy to parallelize, and also lots of hard problems that are hard to parallelize
there are probably some easy problems that are hard to parallelize but since they're easy nobody even thinks to bother
parallelization is itself an optimization, so when you start talking about optimizing after you've parallelized you've just gone back to the original problem
maybe what's needed is an architecture that is structured to handle P-complete problems, rather than sin
Re: (Score:2)
(ducks)
unaware? WTF? (Score:2)
I am kinda curious how anyone even tangentially involved in programming could not be aware that the problem with writing parallel programming was doing it for a gain in efficiency. Making a thread or process is generally just a couple lines of code, synchronization with data separation, mutex's and avoiding deadlocks and race conditions has been solved since almost the beginning of parallelism.
Re:unaware? WTF? (Score:5, Insightful)
synchronization with data separation, mutex's and avoiding deadlocks and race conditions has been solved since almost the beginning of parallelism
And yet people constantly get these details wrong in practice.
It's an extra layer of complexity and it introduces extra chances to make mistakes, even around areas where a programmer could know better. There's not much way around that. If people only made coding mistakes around difficult problems software would be dramatically more bulletproof than it actually is.
Re: (Score:2)
I tend to think of it as an extra dimension in code. With non-parallel code, the code you have (it's sequence) is the same as what it's sequence would be when run. With parallel code, the run-time sequence is different than the code as it's laid out in source.
I see people have trouble with just async stuff (eg. AJAX) and have a hard time wrapping their mind around the fact that even though the callback function is in-sequence with the rest of the code, it's not actually called in that sequence - hence the '
Re: (Score:2)
In my experience, the reason people have so much trouble with async stuff is that every single JavaScript API I've seen that does things asynchronously is rather fundamentally designed *wrong*. They don't provide any clean mechanism for passing dat
Re: (Score:2)
I blame that mostly on the languages not being explicit about what operations and methods are or are not thread safe. And for capable programmers those errors generally only occur when you try to make things more efficient by avoiding excess mutex and data duplication in the pursuit of efficiency.
Re: (Score:2)
And yet people constantly get these details wrong in practice.
If they can't get those right innately, then they never even started out to do parallel programming, they just stuck a fork in their execution flow.
Re: (Score:3)
Re: (Score:2)
The key to parallel programming is compartmentalization. With a good enough foundation that compartmentalizes properly, parallel programming would only be a matter of coding the compartments and synchronizing their results.
That having been said, parallel programming these days solves a fairly niche problem. The speed of modern processors make faking parallelism with interrupts viable. The only area where parallelism truly makes sense is when working with extremely large amounts of large chunks of data.
For e
Re: (Score:2)
Actually, interestingly enough, parallelizing code not only results in computational efficiency gains, but it frequ
Re: (Score:2)
Yes they have. It's called Critical Path Analysis.
Moral of the story... (Score:3)
Good tutorial for someone who wants to jump into some parallel programming, but it's mostly Operating Systems 101 (or 601).
Honestly though, if you have not optimized your algorithm or code to for parallelism and you want to do it now, you might probably be better off writing the whole thing from scratch, and the tutorial explains why very nicely/.
Re: (Score:2)
Re:multiple processors (Score:2)
I'll take a stab and reply to you that "8 processors was enough for anyone", in the sense that multiplexing 8 programs is just insane. Better to just run 8 prorams each on their own core, and use some progs that can use 4 cores at a time. That leaves 4 free.
(Overly simpistic) I agree, but 1028 cores is not the answer. We need the next generation in raw core power to move computing forward. 8 killer cores will beat 1024 mediocre cores.
Re:multiple processors (Score:5, Interesting)
If I have a single 8 core machine but it's attached to a netapp disk-array of 100 platters over a network, then the latency means that the round trip of a single-threaded program is almost guaranteed to leave platters idle. If, instead I split a problem up into multiple threads / processes (or use async-IO concepts), then each thread can schedule IO and immediately react to IO-completion, thereby turning around and requesting the next random disk block. While async-IO removes the advantage of multiple CPUs, it's MASSIVELY error-prone programming compared to blocking parallel threads/processes.
A given configuration will have it's own practical maximum and over-saturation point. And for most disk/network sub-systems, 8 cores TODAY is sufficient. But with appropriate NUMA supported motherboards and cache coherence isolation, it's possible that a thousand-thread application-suite could leverage more than 8 cores efficiently. But I've regularly over-committed 8 core machine farms with 3 to 5 thousand threads and never had responsiveness issues (each thread group (client application) were predominantly IO bound). Here, higher numbers of CPUs allows fewer CPU transfers during rare periods of competing hot CPU sections. If I have 6 hot threads on 4 cores, the CPU context switches leach a measureable amount of user-time. But by going hyper-threading (e.g. doubling the number of context registers), we can reduce the overhead slightly.
Now for HPC, where you have a single problem you're trying to solve quickly/cheaply - I'll admit it's hard to scale up. Cache contention KILLS performance - bringing critical region execution to near DRAM speeds. And unless you have MOESI, even non-contentious shared memory regions run at BUS speeds. You really need copy-on-write and message passing. Of course, not every problem is efficient with copy-on-write algorithms (i.e. sorting), so YMMV. But this, too was an advocation for over-committing.. Meaning while YOUR problem doesn't divide. You can take the hardware farm and run two separate problems on it. It'll run somewhat slower, but you get nearly double your money's worth in the hardware - lowering costs, and thus reducing the barrier to entry to TRY and solve hard problems with compute farms.
amazon EC anyone?
A broader problem that also lacks formalization (Score:2)
In my opinion, many problems with software development, are just as applicable in other domains of our life, and parallel programming is definitely one of them. We equally well have problems managing large teams of people working in parallel. These are problems of logistics, management and also (and no I am not joking) - cooking. And we're as bad handling these as we now handle software development. It may be right, however, to start solving this with the computers - no need to throw away rotten food and/or
Re: (Score:2)
Re: (Score:2)
That's a monumental and exceedingly involving task - to create such a novel concept of a language, partly because few have been written, and even fewer are usable to mere mortals, who happen to be using imperative languages just fine, mind you. Articles upon articles full of either terms-within-terms or lack there-of, have been written. Some people once set upon defining this new computing upon which you seem to touch and created www.tunes.org, which has since stagnated.
I would advise you to write down your
make? (Score:2)
Makefiles specify dependencies and recipes quite tidily. A simple implementation of make is provided in the "AWK book" as an example program- http://www.cs.bell-labs.com/cm/cs/awkbook/ [bell-labs.com]
"What Makes Parallel Programming Difficult?" (Score:2, Insightful)
That you have to do everything all at once. How would you tell 50 kids to sort 50 toy cars? How would you tell 50 footballers to line up by height all at once? How would you have 50 editors edit the 50 pages of a screenplay all at once so that it makes sense from a continuity perspective? All these problems are very easy, but become very hard when you have to do it all at once...
Re: (Score:2)
The ultimate solution to most of those appears to be, "have to do it fifty times, and assign one job to a person".
That said, not all problems are so easily dealt with as bulk operations. Perhaps you have some real need to take one item and spit the work up among multiple people. The real driving force for all the scenarios I can think of is a desire or need for an upper bound on job latency.
I do this kind of work as my day job. I've also got some experience in managing groups of coders (at work, not particu
Re: (Score:2)
You can if they swing that way.
Re: (Score:2)
Easy break into n subgroups have them line up by height and then perform pairwise merges.
Lame (Score:3, Interesting)
Chapel (Score:2)
This looks like an advertisement for Chapel but I have no relation to Cray. Having taken a graduate parallel programming course, I cannot agree more with the statement that "Parallel Programming is difficult". I struggled a lot with pthreads and MPI before doing the final assignment in Chapel which was a plea
Parallelisation of serial code is the problem (Score:2)
20 years ago when I was working with transputers we use Occam. It was a very pure parallel programming language and it wasn't too difficult. However, writing parallel code meant starting from scratch (getting rid of the dusty decks of old algorithms as my professor described it). However, this never really happened and we've ended up with primitive parallelisation nailed on to sequential code. The are many parallel architectures, SIMD, MIMD, distributed memory, shared memory and combinations of them all
Re: (Score:2)
Occam still exists. KROC supports the latest version of Occam (Occam-Pi), which supports mobile processes, generics and other concepts acquired from the developments in serial programming and computer clusters.
I consider it to be one of the finest languages out there for learning parallel programming and consider that most of the modern failures in the field are a result of people not knowing it and therefore not knowing the fundamentals. You can't run if you can't walk.
The transputer was the ultimate in co
Re: (Score:2)
Erlang has most of the observable properties of Occam, so it's not all lost.
The problem is that people still insist on using mutable shared state in concurrent programs. If only they stopped doing silly things like that...
Erlang has most of the observable properties of Occam, so it's not all lost.
The problem is that people still insist on using mutable shared state in concurrent programs. If only they stopped doing silly things like that...
Right on, brother
Efficiency is hard (Score:5, Informative)
Using locks and the like make it very easy to do multithreaded and parallel programs.
The big problem comes when you need multiple locks because you find your program is waiting more on locks than anything else which is gumming up the whole works, and that can easily lead to deadlocks and other fun stuff.
Another way is to consider lockless algorithms, which don't have such blocking mechanisms. However, then you get into issues where atomicity isn't quite so atomic thanks to memory queues and re-ordering done in the modern CPU, and thus have to start adding memory barriers before doing your atomic exchanges.
Raymond Chen (of Microsoft) did a nice write up of the lockfree ways to do things and what Windows provides to accomplish them.
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/05/10149783.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/06/10150261.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/06/10150262.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/07/10150728.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/08/10151159.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/08/10151258.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/12/10152296.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/13/10152929.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/14/10153633.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/15/10154245.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/19/10155452.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/20/10156014.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/21/10156539.aspx [msdn.com]
http://blogs.msdn.com/b/oldnewthing/archive/2011/04/22/10156894.aspx [msdn.com]
Re: (Score:3)
Re: (Score:2)
is anything but simple, and moreover cannot be anything but simple!
parse error
Re: (Score:2)
>>Using locks and the like make it very easy to do multithreaded and parallel programs.
Eh, the Dining Philosophers would like to ask you out to eat. Locks can introduce timing issues which can result in a program locking up at random.
The real difficulty of parallel programming comes from two things (speaking as someone who has a Master's in the subject):
1) The development environment isn't as well supported as single-threaded coding.
2) It requires a different mindset to write code solidly. Remember ho
Re: (Score:2)
It also does not help that threads are very easy to mess up. Look at all the traditional programs that used multiple threads for nonparallel concurrency, and how much trouble the developers have with deadlocks, or forgetting to use locks on a shared variable access, or calling code intended only to be used by the other thread, etc.
So even if you have the ideal parallel version of the algorithm all planned out, actually implementing it correctly can still be problematic.
The answer is... (Score:2)
Also... (Score:2)
Functional programming (Score:2)
One more reason why functional programming matters. Many programs become trivial to parallelize when you avoid mutation and side-effects outside of limited, carefully-controlled contexts.
It's truly a joy when you can parallelize your code by changing a single character (from "map" to "pmap"). There's usually a little more to it than that, but you almost never have to deal with explicit locks or synchronization.
Re: (Score:2)
Agreed. Learning to isolate side effects is one of the best things a programmer will get from FP.
Re: (Score:2)
One more reason why functional programming matters. Many programs become trivial to parallelize when you avoid mutation and side-effects outside of limited, carefully-controlled contexts.
More specifically, the problem of parallel programming is the problem of structuring state for concurrent access. Everything else (the mechanics of locks etc) is trivial and mainly a typing exercise.
The reason this is difficult is that it's an optimization; normally when we optimize code we write the naive version first (or near-naive), then optimize it. We might change a hash table to a vector, memcpy() to a few lines of inline assembler for a particular target, hand code CRC/checksumming/RC4 in assemble
Re: (Score:2)
JAVA doesn't produce great programmers IMHO. But Intel is going to need people with lower level skills and C, assembly, machine, forth is just too much to ask for a general curriculum. Assume they are going to have to pick that up on the job, IMHO.
Poor choice of language (Score:2)
What makes parallel programming hard is poor languages. Languages that allow state changes and don't keep them isolated. Isolate changes of state, all changes of state and be careful about what kinds of combinators to use. Google map-reduce works whenever
a) You can organize your data into an array
b) You can do computations on array element in isolation
c) You can do computations on the entire array via. associate operations pairwise
And most programs do meet those properties but they slip in all sorts of
Re: (Score:2)
There's a reason Mapreduce is said to operate on "embarrassingly parallel" problems. There are a lot of them. But there are also a lot of problems which are not embarrassingly parallel; for instance, they have nonlocal data dependencies.
Re: (Score:2)
Nonlocal data dependencies are not a problem. Map reduce is fantastic for distributing to clients. Each client performs a mini map reduce and then the results are collected.
Re: (Score:2)
No I hadn't read your futurechips article. Though I agree with what you wrote. But frankly the parallelism is obvious its only by using C you are making things complex:
a) construct a function that takes an array of char and returns a count hash
b) construct a function that takes two count hashes and adds them to produce a count hash
c) construct a function that splits an array of char evenly into n pieces
array input -> (c) -> each piece goes to (a) -> (b) invoked to combine
Why make it any more c
Re: (Score:2)
I suspect you'll like: http://www.willamette.edu/~fruehr/haskell/evolution.html [willamette.edu]
Parallel programming *NOT* widely needed (Score:4, Interesting)
The dirty secret of parallel programming is that it's *NOT* so widely needed. I think a lot of academics got funding to study automatic parallelization or other parallel techniques, and they latch on to multicore as a justification for it, but it's not.
There is only one GOOD reasons to use multithreading -- because your work is compute-bound. This typically happens on large-data applications like audio/video processing (for which you just call out to libraries that someone else has written), or else on your own large-data problems that have embarrassingly trivial parallelization: e.g.
var results = from c in Customers.AsParallel() where c.OrderStatus="unfilfilled" select new {Name=c.Name, Cost=c.Cost};
Here, using ParallelLINQ, it's as simple as just sticking in "AsParallel()". The commonest sort of large-data problems don't have any complicated scheduling.
There are also BAD reasons why people have used multithreading, particularly to deal with long-latency operations like network requests. But this is a BAD reason, and you shouldn't use multithreading for it. There are better alternatives, as shown by the Async feature in F#/VB/C# which I worked on, which was also copied into Javascript with Google's traceur compiler). e.g.
Task task1 = (new WebClient()).DownloadStringTaskAsync("http://a.com");
Task task2 = (new WebClient()).DownloadStringTaskASync("http://b.com");
Task winner = await Task.WhenAny(task1,task2);
string result = await winner;
Here it kicks off two tasks in parallel. But they are cooperatively multitasked on the same main thread at the "await" points. Therefore there is *NO* issue about race conditions; *NO* need to use semaphores/mutexes/condition-variables. The potential for unwanted interleaving is dramatically reduced.
So in the end, who are the people who still need to develop multithreaded algorithms? There are very few. I think they're just the people who write high-performance multithreaded libraries.
Re: (Score:2)
The dirty secret of parallel programming is that it's *NOT* so widely needed.
That's kind of begging the question, there.
Those who need it know they need it. Those who think it's neato-keen and want to play with it try to come up with ways to use that that are maybe not obvious, and for which it is maybe not even necessary. I've known about this at least since the first Thinking Machines came out and our school got one of the bigger ones, solved the problem they used in the grant proposal that paid for it in about a week, then realized they had a multi-million-dollar computer with
Re: (Score:2)
Thus while YOU may not care about MT, you should strive to make all your code reentrant at the very least. To whatever degree this allows contention-free memory access (e.g. ZERO global variables). This fut
Re: (Score:2)
Graphics is some low hanging fruit that can get a fair bit of benefit from a lot of threads. A lot of operations with graphics involve doing the same thing to a large amount of data that can easily be carved into individual chunks and dealt with in parallel. It's the same with video where the same transformation or filter gets applied independantly to thousands of frames and all you care about is getting it done qui
Re: (Score:3)
The dirty secret of parallel programming is that it's *NOT* so widely needed. I think a lot of academics got funding to study automatic parallelization or other parallel techniques, and they latch on to multicore as a justification for it, but it's not.
It is widely needed, but perhaps not by you.
There is only one GOOD reasons to use multithreading -- because your work is compute-bound.
Ok, now you're confusing the issue. Are you talking about parallel programming or multi-threaded programming? Parallel programming is larger in scope the simple multi-threaded programming.
So in the end, who are the people who still need to develop multithreaded algorithms? There are very few. I think they're just the people who write high-performance multithreaded libraries.
No, there are quite a few and many of them make quite a lot of money to do so. Do you think the programmers at ILM, 3DS, Pixar, and NASA are just sitting around doing nothing? Does your MT algorithm library also know how to optimize for the GPU as well? Are you sure that a one si
Re: (Score:2)
There is only one GOOD reasons to use multithreading -- because your work is compute-bound.
...and therefore, most of the people in the world won't need anything beyond an Intel Atom because their tasks aren't compute-bound.
Seriously, I don't know what kind of code you write for a living, but the code I write is almost always has some portion of compute-bound submodules, even if what I do has nothing to do with video codecs or 3d or whatever field that there are convenient libraries.
Re: (Score:2)
Cooperative multitasking is like going back to Windows 3.1. That operating system totally relied on the trick you mentioned, and as we all know, it sucked.
Why did it suck? Obviously, for many reasons but most of all because once such a "fake thread" of execution starts taking a long time, it blocks all other threads.
Why do you think a modern webserver uses many different threads?
Sigh... I guess you need to gain some more experience in the parallel programming world.
The problem: too much data sharing (Score:2)
The basic problem with parallel programming is that, in most widely used languages, all data is by default shared by all threads. C, C++, and Python all work that way. The usual bug is race conditions.
There have been many languages for parallel programming which don't have default sharing, but they've never taken over outside some narrow niches. Partly because most of them weren't that useful outside their niche.
The other classic problem is that in most shared-data languages with locks, the language d
Scala has added parallel collections and has Akka (Score:2)
The Scala [scala-lang.org] community has tried to move the problem into a more practical realm by adding things like parallel collections [infoq.com], DSL's [scala-lang.org] to abstract out the problem for specific applications and the Akka Project [akka.io] for simpler concurrency.
Most of the parallel programming discussion I've seen is very complicated and not likely to appeal to those who have to do practical day-to-day business projects. By pushing the abstractions up a level, I think the Scala folks have made parallel programming more accessible for the av
Parallelism is very easy,provided you don't do it. (Score:2)
Parallelism is very easy, provided that you don't do it yourself.
Use a pure functional programming language like Haskell that can be automatically parallelized.
Or use a programming language that uses the Active Object Pattern (or the Actor model).
Or do as I do: use C++ to implement message passing, then have one thread per object. Objects don't have public members, they simply communicate by exchanging messages.
In all the above cases, the trick is to avoid state. In Haskell, state is avoided by design; in t
Re: (Score:2)
The barrier for learning to program purely functionally is high. But its a 1x barrier vs. complexity for ever. It makes the system simple.
I agree with him, avoid state.
Linda (Score:2)
Re:30 years (Score:2)
"The news of the problems of parallelization is still news after 30 years"
- paraphrase of Christian saying
Re: (Score:2)
I've seen one example of a threaded serial task. Intel has HyperThread optimization PDF some where showing some interesting tricks using HT. The example they had was on an i7, so fairly recent. They had a serial task that iterated through an array. They loaded an extra thread that synced with the primary thread, ran it on the other virtual thread, and all it did was call the prefetch instruction on the array.
Even though they had a modern architecture with advanced prefetching and linear memory access, using
Re: (Score:2)
uhhggs.. ignore the code.. something happened and some of the code that previewed didn't show up.
Use a practical tool (Score:2)
Erlang (and other functional, single-static-assignment languages) are perfect for parallel programming.
Okay, then the only problem is getting something useful out of Erlang.
Back in 1985 the Japanese government announced a "fifth generation" computing project, with software to be developed in Prolog. So I went and learned Prolog, an intriguing and amusing language. Only problem is, it was totally useless for any actual application, as the Japanese found out.
Sorry, but in order to believe any of the promises of one of those non-vonNeumann languages, I have to see a practical working application first.
Re: (Score:2)
Re: (Score:2)
Erlang (and other functional, single-static-assignment languages) are perfect for parallel programming.
In that case you can restate the original question as "how do I implement an Erlang compiler and runtime?" If, indeed, Erlang embodies the most efficient way to solve all parallell programming problems. Which of course is absurd. Erlang is useless for implementing servers, kernels, runtimes, networking stacks, file systems, device drivers, VM systems, database servers, or anything else that actually makes a computer tick. It's simply an interface to an already-implemented computing system. It's good to
Re: (Score:2)
I Which of course is absurd. Erlang is useless for implementing servers, kernels, runtimes, networking stacks, file systems, device drivers, VM systems, database servers, or anything else that actually makes a computer tick.
When I first read this I assumed the poster was being sarcastic, but reading again they actually believe it. Erlang useless for implementing servers? Erlang is used to implement loads of servers. You look at the back end of the a lot the the top 100 company web services and you will find them using Erlang to implement their server functionality. Erlang is used to provide database systems too.
The idea that Erlang is some sort of toy academic language that is not used for anything practical is a joke. Erlang
Re: (Score:2)
Occam + transputer boards was a brilliant idea. Didn't work out so well in practice but that wasn't due to Occam. It's about the most elegant language I've ever seen.
However, writing an Occam compiler would run into the same sort of issues as the article describes, I fear. Unless CPU-cores take some of the transputer lessons to heart.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
A bit is a mutex if your processor has an atomic test and set instruction. Which these days they all do. A semaphore is an integer protected by a mutex. He oversimplified a bit, but if you know what you're doing it is that simple.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
In the end, what makes it harder is that the things you do on software are usually way more complex than the things you do on hardware. If not for that, that argument would make perfect sense as hardware also have issues with communication latency, non-determinism, and localized data (not just caches), to an even higher degree than software. And you also must make it robust.
Re: (Score:2)
Most problems are not parrallizeable
That is only true in a very theoretical sense and completely wrong in practice. Almost everything that burns CPU on your computer today is easily parallelizable, your video encoder doesn't really care if the other CPU is crunching away on the next scene in the movie, most image filters work just fine when applied to only a section of an image, a game could easily do AI for each unit in parallel and your webbrowser shouldn't really care if each webpage is rendered on a separate thread either.
The part of the
Re: (Score:2)
You are absolutely right on this one. Obviously we've hit limits of CPU performance and parallel is the way to go.
As an aside though let me point one other option which Intel was exploring in the late 80s-early 90s: break the CPU up into a series of processors each with different complementary instruction sets. Intel played around with 486/i960 combo where the 486 offered great task switching, high instruction speed, built in floating point and the i960 offered rapid vector calculation. IBM RS/6000 line
Re: (Score:2)
Because they require several different kinds of completely alien thinking; knowing imperative programming is almost a disadvantage to learning functional programming. Also, most functional languages were too purist to allow actually getting much real work done - the libraries were usually very weak so every problem involved reinventing dozens of wheels. Clojure [clojure.org] may have solved this problem - it isn't too purist and it can use all the Java libraries. It still requires a lot from the programmer, though perhap