×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Python Gets a Big Data Boost From DARPA

Soulskill posted about a year ago | from the from-unclesam-import-money dept.

Python 180

itwbennett writes "DARPA (the U.S. Defense Advanced Research Projects Agency) has awarded $3 million to software provider Continuum Analytics to help fund the development of Python's data processing and visualization capabilities for big data jobs. The money will go toward developing new techniques for data analysis and for visually portraying large, multi-dimensional data sets. The work aims to extend beyond the capabilities offered by the NumPy and SciPy Python libraries, which are widely used by programmers for mathematical and scientific calculations, respectively. The work is part of DARPA's XData research program, a four-year, $100 million effort to give the Defense Department and other U.S. government agencies tools to work with large amounts of sensor data and other forms of big data."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

180 comments

Great. Just Great (1, Insightful)

Anonymous Coward | about a year ago | (#42806011)

The work is part of DARPA's XData research program, a four-year, $100 million effort to give the Defense Department and other U.S. government agencies tools to work with large amounts of sensor data and other forms of big data.

Yeah the govt needs better systems to manage the huge databases and dossiers they are building on everybody with their warrentless wiretaps and reading everybody's emails. Anybody who helps with this project is pretty damn naive if they don't think it will also be used for this.

For that matter anybody who trusts the govt and thinks the govt is your friend is pretty damn naive. Yeah I would like to believe that too. No I won't ignore the mountains of evidence to the contrary. I won't treat all the counterexamples as isolated cases. I see them for what they are: an amazingly consistent pattern. The rule, not the exception. Govt positions are really attractive to sociopath types who just love power and control and a feeling that they are important and they get that feeling by imposing their will on us.

Re:Great. Just Great (5, Insightful)

Kwyj1b0 (2757125) | about a year ago | (#42806151)

Yeah the govt needs better systems to manage the huge databases and dossiers they are building on everybody with their warrentless wiretaps and reading everybody's emails. Anybody who helps with this project is pretty damn naive if they don't think it will also be used for this.

For that matter anybody who trusts the govt and thinks the govt is your friend is pretty damn naive. Yeah I would like to believe that too. No I won't ignore the mountains of evidence to the contrary. I won't treat all the counterexamples as isolated cases. I see them for what they are: an amazingly consistent pattern. The rule, not the exception. Govt positions are really attractive to sociopath types who just love power and control and a feeling that they are important and they get that feeling by imposing their will on us.

So what you are saying is that DARPA funds will be used in a way to further the goals of DARPA/The government? Shocking. I haven't read anything that says which agencies will/won't have access to these tools - so I'd hazard a guess that any department that wants it can have it (including the famous three letter agencies).

FYI, Continuum Analytics is a company that is based on providing high-performance python-based computing to clients. Any packages they might release will either be open source (and can be checked), or closed source (in which case you don't have to use it). They aren't hijacking the Numpy/Scipy libraries. They are developing libraries/tools for a client (who happens to be DARPA). (Frankly, I'd hope that Continuum Analytics open sources their development because it might be useful to the larger community). You do know that DARPA funds also go to improve robotics, they supported ARPANET, and a lot of their space programs later got transferred to NASA?

Basically, I have no idea what you are ranting about. One government organization funded a project - it happens all the time. Do you rant about NSF/NIH/NASA money as well? If so, you'd better live in a cave - a lot of government sponsored research has gone into almost every modern convenience that we take for granted.

Re:Great. Just Great (5, Funny)

Anonymous Coward | about a year ago | (#42806621)

What is this APRANET thing? It sounds like some useless crap loaded acronym to me.

Re:What is this APRANET (sic) thing? (0)

Anonymous Coward | about a year ago | (#42806819)

http://en.wikipedia.org/wiki/ARPANET

Re:Great. Just Great (0)

luis_a_espinal (1810296) | about a year ago | (#42808881)

What is this APRANET thing? It sounds like some useless crap loaded acronym to me.

You gotta be fucking kidding me. Either you are trolling or you are completely clueless about technology. In the case of the later, it begs the question what are you doing in /. If you don't know what ARPANET you should be posting in MySpace instead of posting on a nerd/tech news site. It'd be like me posting opinions on a medicine-related site without knowing the meaning of the word 'penicilin'.

Re:Great. Just Great (5, Informative)

sdaug (681230) | about a year ago | (#42807227)

Frankly, I'd hope that Continuum Analytics open sources their development because it might be useful to the larger community

Open sourcing is a requirement of the XDATA program.

Re:Great. Just Great (0)

Anonymous Coward | about a year ago | (#42808355)

You have no idea what he's talking about? It was pretty clear: factions within the US government wants these tools to datamine all the ISP data they have been snarfing up so they can spy on everyone in the world. Saying that you believe otherwise is a pretty extreme view and, as such, requires a very high standard of proof. Do you have that proof? No, then STFU while us adults try to figure out how to stop this obvious slide into tyranny.

I get the impression that (5, Interesting)

Chrisq (894406) | about a year ago | (#42806023)

I get the impression that in the Engineering and Scientific community Python is the new Fortran. I hope so, because it would be "Fortran done right".

Re:I get the impression that (1)

BlackPignouf (1017012) | about a year ago | (#42806107)

I think you're right.
I love Ruby, it's a very fun and effective language, I could write it in my sleep but there are so many cool projects that are written in Python.
Those languages are *very* similar, and it's a shame that so much effort is being divided between communities.
I might get to learn Python one day but I'm afraid I'd become a so-so programmer in both languages.

Re:I get the impression that (5, Interesting)

jma05 (897351) | about a year ago | (#42806185)

> I might get to learn Python one day but I'm afraid I'd become a so-so programmer in both languages.

I empathize since I conversely only barely use Ruby. Once someone learns one of these languages, there is not that much that the other offers. But happily, one need not learn advanced Python to benefit from these projects.

> it's a shame that so much effort is being divided between communities

AFAIK, all scientific funding from US and Europe is/was always directed to Python, not Ruby. So Python is firmly established as a research language and there is not much effort being divided with Ruby (which seems to have a much more spotted and amateur movement in this direction), at least as far as scientific stuff is concerned (Ruby is more popular on web app side). For me the tension for scientific use is not between Python and Ruby, but between Python and R. Python community is replicating a lot of R functionality these days but R still has a much better lead in science libraries. Happily, it is quite easy to call R from Python.

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42806925)

I think you're right.
I love Ruby, it's a very fun and effective language, I could write it in my sleep but there are so many cool projects that are written in Python.
Those languages are *very* similar, and it's a shame that so much effort is being divided between communities.
I might get to learn Python one day but I'm afraid I'd become a so-so programmer in both languages.

Both languages suffer from the global interpreter lock defect and will require a rewrite in the next 5-10 years if the languages have any chance of surviving in the servers. It will take some very serious, dedicated, low level work and I just don't see it happening. I have this fantasy where Guido and Matsumoto will sit down and write the common code together for a super-interpreter that will handle different syntax in a modular way. I know it's technically possible since GCC is doing something very similar but, again, I just can't see this happening.

In the meantime, Go is looking mighty good...

Re:I get the impression that (3, Interesting)

lattyware (934246) | about a year ago | (#42806995)

The GIL is an overblown issue. Threading is designed to get around issues with accessing slow resources, not for serious parallel computing. Just use multiprocessing if you want to do lots of computing in parallel, problem solved.

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42807619)

This is what Python and Ruby programmers actually believe. It's quite pathetic.

GIL is a non-issue. (1)

luis_a_espinal (1810296) | about a year ago | (#42809099)

I think you're right. I love Ruby, it's a very fun and effective language, I could write it in my sleep but there are so many cool projects that are written in Python. Those languages are *very* similar, and it's a shame that so much effort is being divided between communities. I might get to learn Python one day but I'm afraid I'd become a so-so programmer in both languages.

Both languages suffer from the global interpreter lock defect and will require a rewrite in the next 5-10 years if the languages have any chance of surviving in the servers.

Gee, because there are no distributed enterprise solutions written on Python or Ruby <rolls eyes/>

It will take some very serious, dedicated, low level work and I just don't see it happening.

It already has happened. The solutions aren't just in the mainstream versions, though. Take Jython. On a typical JVM, it is the fastest Python in-the-trenches implementation available. Throw that over specialized Java-focused hardware (like the Azul Vega 3), and you are on fire.

Furthermore, a solution to the GIL problem is not necessary in the general case. In any modern system, the cost of communicating processes vs threads is no longer so much of an issue as it was a decade ago. Depending on the nature of computation, context switching between processes can be as cheap as switching between threads, and the former is typically somewhat (but no completely free) of the locking issues that are experienced with threading paradigms as seen in, say, Java/JEE solutions.

In the back-end server arena where the greatest bottlenecks are those between http servers, app servers and database servers, there are so many, tried and true solutions to the so-called GIL process that it typically renders it as a non-issue. More processes per box, more RAM and SDDs, more boxes collocated on the same subnets running more processes, all communicating with some type of messaging queue. For these typical solutions, the issue of the GIL get blurred into non-existence.

It's only for those applications where you have to squeeze every last drop out of your cores that the GIL becomes an issue, and where Java/JEE shines. But for the typical bizneyty application, a platform with a GIL issue does just fine by simply scaling horizontally.

I have this fantasy where Guido and Matsumoto will sit down and write the common code together for a super-interpreter that will handle different syntax in a modular way. I know it's technically possible since GCC is doing something very similar but, again, I just can't see this happening.

In the meantime, Go is looking mighty good...

Google Go looks mighty good... for systems-level programming. That's what Google intended it to be. For app development, sorry, you need more than a language. You need a tried and true app stack. Until that happens (and it will take some time for that to happen), Java, Python, Ruby and even .NET do more than fine.

You need more than the language (however greatly designed it might be) to make potentially complex domain-specific shit happen.

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42806125)

Fortran done right is fortran that's slow as hell?

Beyond just the speed issue, I've had problems where simulations in python die right in the middle, because it had been developed on a 32 bit machine, and some of the libraries defaulted to using the architecture's precision. The problem was quite hard to debug, because it was way after the numbers had been stored. This is the kind of bullcrap you get when your language doesn't have static types.

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42806163)

Yea... Fortran done right is actually... Fortran done right. There's nothing wrong with the language.

Re:I get the impression that (2, Informative)

Anonymous Coward | about a year ago | (#42806345)

I guess the problem is that people who speak about Fortran actually think about FORTRAN. The last FORTRAN standard was from 1977, and that shows. After that, there had been no new standard and little new development until the Fortran 90 standard (note the different capitalization). Fortran 90 got rid of the old punch card based restrictions by giving it completely new, much more reasonable code parsing rules (it still accepts old form code for backwards compatibility, but you cannot mix both forms in one file because they are too different), gave it a full set of properly nesting flow control statements (actually that was one thing already commonly available as non-standard extension to FORTRAN), and added very powerful array processing, operator overloading, and modules (and probably a few other things I don't remember right now). Later versions even added object orientation (and probably a whole set of other things; I haven't really followed Fortran development beyond Fortran 90).

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42807387)

So the language gets a bad rap for things that weren't standardized until... 23 years ago? And we're comparing against Python? Which didn't hit version 1.0 until 1994?

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42807463)

Next up... comparing the 8086 to the i5....

Re:I get the impression that (5, Informative)

solidraven (1633185) | about a year ago | (#42806195)

You're dead wrong, nothing quite beats Fortran in speed when it comes to number crunching. If you need to go through hundreds of gigabytes of data and performance is important there's only one realistic choice: Fortran. Python isn't fit to run on a large cluster to simulate things, too much overhead. And lets not forget what sort of efficiency you can get if you use a good compiler (Intel Composer). You won't find Fortran on the way out over here, it's here to stay!

Re:I get the impression that (2)

ctid (449118) | about a year ago | (#42806225)

Why would Fortran be any faster than any other compiled language?

Re:I get the impression that (5, Informative)

Anonymous Coward | about a year ago | (#42806261)

Short answer, Fortran has stricter aliasing rules so the compiler has more optimization opportunities. Long answer, see Stack Overflow [stackoverflow.com] .

Re:I get the impression that (1)

martin-boundary (547041) | about a year ago | (#42806267)

Why would Fortran be any faster than any other compiled language?

Because the language is simpler, so the compiler can make assumptions and generate better automatic optimizations. C/C++ are much harder to optimize (=generate optimal assembly instructions).

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42806269)

Fortran compilers have been around for much longer than other compilers, so the optimizations are well-known. Much research has been put into generating Fortran code, so additional research has gone into keeping the Fortran code running fast, since it is hard to redo the algorithms with the amount of testing money spent verifying that the software is correct. That means that old Fortran code persists, with compiler optimization, despite the fact it limits parallelization efforts, because the algorithms can not be altered (automatic parallelization, OpenMP are nice quick-fixes, but will only impress your boss). So, the end result is a push to make sure the compiler optimizations are as fast as possible to keep the old Fortran code decent without a rewrite.

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42806229)

Well.. there's C, of course...

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42806235)

I thought NumPy (which i'm sure you would be using if doing large number crunching) was based on Fortran code (LAPACK) anyway? And with things like IPython clustering, it can run on large clusters of computers easily.

It's probably not as fast as pure fortran, but if it lets scientists build a model by themselves quickly instead of learning fortran or queuing up someone who does than it seems like a good thing to me...

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42809343)

SciPy tries to use LAPACK (or any other such tools) where-ever possible. NumPy is based in C, but does try to utilize specialty math libraries like Intel's MKL wherever possible. So, the core numerical array class (NumPy) is C, while the advanced scientific tools (SciPy) are in C and Fortran. Because of that, they are an *extremely* powerful duo.

Re:I get the impression that (2)

ssam (2723487) | about a year ago | (#42806245)

FORTRAN does arrays in a way thats slightly easier for the compiler to optimise. But some modern techniques and data structures are much harder to do in FORTRAN compared to c++. It is also quite easy to call C, C++ or FORTRAN functions from python.

Writing a loop in python is slow. You express that loop as a numpy array operation you get a substantial way towards c speed. if you use numexpr you will get something faster than a simple C version.

Processing big data is as much about moving the data around, and minimising latency in this movement as the raw processing speed. so a language that lets you express things efficiently will win in the end.

Re:I get the impression that (1)

martin-boundary (547041) | about a year ago | (#42806315)

Processing big data is as much about moving the data around, and minimising latency in this movement as the raw processing speed. so a language that lets you express things efficiently will win in the end.

If by expressing things efficiently you mean easy for the programmer to write, then you're wrong. What matters (doubly so for big data) is full control over the machine's resources, ie how data is laid out in memory, good control over i/o etc. While this has always been the key to fast performance, big data is plagued by big-oh asymptotics. For example, if you can lay out your data structures efficiently enough to keep everything in cache, your running time can easily gain a factor of ten, ie 1 day instead of 10 days. Ask Google or Facebook if they care about that...

Scripting languages have their place where performance doesn't matter _enough_ to optimize, eg your local supermarket chain trying to datamine their customers in time for the end of the month.

Re:I get the impression that (1)

Anonymous Coward | about a year ago | (#42807843)

If by expressing things efficiently you mean easy for the programmer to write, then you're wrong. What matters (doubly so for big data) is full control over the machine's resources, ie how data is laid out in memory, good control over i/o etc. While this has always been the key to fast performance, big data is plagued by big-oh asymptotics. For example, if you can lay out your data structures efficiently enough to keep everything in cache, your running time can easily gain a factor of ten, ie 1 day instead of 10 days. Ask Google or Facebook if they care about that...

Scripting languages have their place where performance doesn't matter _enough_ to optimize

I don't think anyone would dispute that using pure python for a Big Data application would be insanity. But that's not what's happening. Continuum Analytics will be writing its performance-critical code in C (or fortran or another low-level language). They will use Python for non-performance-critical code, including (but not limited to) the API. (This is also how NumPy is written, and SciPy, etc. etc.)

Re:I get the impression that (4, Interesting)

Kwyj1b0 (2757125) | about a year ago | (#42806355)

Compared to plain old Python, yes. But Cython offers a lot of capabilities that improve speed dramatically - just using a type for your data in Cython gives programs a wonderful boost in speed.

As someone who uses Matlab for most of my programming, I have come to detest languages that do not force specifying a variable type and/or declaring variables. Matlab offers neither, but it is a standard in some circles.

llvm (1)

Anonymous Coward | about a year ago | (#42806947)

python linking to llvm is the way to really speed it up and a few groups are seriously working on it

Re:I get the impression that (5, Insightful)

LourensV (856614) | about a year ago | (#42806555)

You're probably right, but you're also missing the point. Most scientists are not programmers who specialise in numerical methods and software optimisation. Just getting something that does what they want is hard enough for them, which is why they use high-level languages like Matlab and R. If things are too slow, they learn to rewrite their computations in matrix form, so that they get deferred to the built-in linear algebra function libraries (which are written in C or Fortran), which usually gets them to within an order of magnitude of these low-level languages.

If that still isn't good enough, they can either 1) choose a smaller data set and limit the scope of their investigations until things fit, 2) buy or rent a (virtual) machine with more CPU and more memory, or 3) hire a programmer to re-implement everything in a low-level language and so that it can run in parallel on a cluster. The third option is rarely chosen, because it's expensive, good programmers are difficult to find, and in the course of research the software will have to be updated often as the research question and hypotheses evolve (scientific programming is like rapid prototyping, not like software engineering), which makes option 3) even more expensive and time-consuming.

So yes, operational weather forecasts and big well-funded projects that can afford to use it will continue to use Fortran and benefit from faster software. But for run-of-the-mill science, in which the data sets are currently growing rapidly, having a freely available "proper" programming language that is capable of relatively efficiently processing gigabytes of data while being easy enough to learn for an ordinary computer user is a godsend. R and Matlab and clones aren't it, but Python is pretty close, and this new library would be a welcome addition for many people.

Re:I get the impression that (4, Insightful)

nadaou (535365) | about a year ago | (#42806929)

You're probably right, but you're also missing the point. Most scientists are not programmers who specialise in numerical methods and software optimisation.

Which is exactly why FORTRAN is an excellent choice for them instead of something else fast (close to assembler) like C/C++, and why so many of the top fluid dynamics models continue to use it. It is simple (perhaps a function of its age) and because of that it is simple to do things like break up the calculation for MPI or tell the compiler to "vectorize this" or "automatically make it multi-threaded" in a way which is still a long from maturity for other languages.

Can you guess which language MATLAB was originally written in? You know that funny row,column order on indexes? Any ideas on the history of that?

R is great an all, and is brilliant in its niche, but how's that RAM limitation thing going? It's not a solution for everything.

MATLAB is pretty good too, as is Octave and SciLab, and it has gotten a whole lot faster recently, but ever try much disk I/O or array resizing for something which couldn't be vectorized? Becomes slow as molasses.

If that still isn't good enough, they can either 1) choose a smaller data set and limit the scope of their investigations until things fit,

heh. I don't think you know these people.

2) buy or rent a (virtual) machine with more CPU and more memory,

Many problems are I/O limited and require real machines with high speed low latency network traffic. VMs just don't cut it for many parallelized tasks which need to pass messages quickly.

Forgive me if I'm wrong, but your post sounds a bit like you think you're pretty good on the old computers, but don't know the first thing about FORTRAN and are feeling a bit defensive about that, and attacking something out of ignorance.

Re:I get the impression that (1)

LourensV (856614) | about a year ago | (#42807641)

You're not picking on me [slashdot.org] , you're arguing your point. That's what this thing here is for, so no hard feelings at all.

I'll readily admit to not knowing Fortran (or much Python! ;-)); I'm a C++ guy myself, having got there through GW-Basic, Turbo Pascal and C. I now teach an introductory programming course using Matlab (and know of its history as an easy-to-use Fortran-alike), and I use R because it's what's commonly used in my field of computational ecology. I greatly dislike R, and I'm not too hot on Matlab either, as the first thing you should do when programming is to decide what the program is about, and to express that you need type definitions, which Matlab nor R have. From a very quick look around, at least recent versions of Fortran do have them, so that's good in my view. As for the RAM limitations in R, it seems to me that that is actually a consequence of the vectorised style of programming and the lack of lazy evaluation: you tend to get either unreadable code with enormous expressions, or a lot of temporaries which eat up lots of RAM.

Replying to your other post [slashdot.org] , I was thinking of the many hundreds of millions that are spent on satellites and the dedicated compute clusters for weather forecasting. I've also heard of budget issues and lack of replacement satellites in that area, but it's still a lot of money compared to most grants. Over here it's big news if someone manages to get a million Euro grant, spread over a couple of years, while NOAA has a 4.7 billion USD yearly budget. Of course they do other things than weather forecasting, I'm comparing an entire government organisation to a single scientific investigation here, but it's a different level for sure.

In the end, I suspect that we're simply in different fields, and therefore seeing different things. Generally speaking, the more physical the field, the more tech-savvy the scientists, and the more computer use. In my institute, Microsoft Excel is by far the number one data processing tool...

Re:I get the impression that (1)

nadaou (535365) | about a year ago | (#42806983)

So yes, operational weather forecasts and big well-funded projects that can afford to use it will continue to use Fortran and benefit from faster software.

I don't mean for this to be pick on LourensV day, but I have another small nit to pick. You're presuming operational weather forecasting is well funded? I don't think funding has anything to do with it. Often it's what the original author knew which chose the language.
And have you seen what's been done to NOAA's budget over they last decade?? Well funded. LOL.

FORTRAN is used because it's easy to get your head around so you can focus on the science not the coding. Much in the same way as Python is meant to be, as a matter of fact.

How's that threading library in Python 2/3 doing? Still not able to actually make more than one thread and has to spawn new processes instead? Python is quite nice, and I welcome the improvements, but it still has a long way to go. Hopefully this a bit of funding will bring that a little closer to reality.

Re:I get the impression that (1)

csirac (574795) | about a year ago | (#42807381)

Perhaps he means it's well funded in the sense that they have dedicated programmers at all. "Run of the mill" science is done by investigating scientists or their jack-of-all-trades research assistants, collaborators or grads/post-docs, etc. most of which are unlikely to have substanital software engineering experience or training in their background.

Nonetheless, they write code - very useful, productive code - but it's in whatever tool or high-level language popular among their peers/discipline (matlab, R, python, perl, fortran... each corner of science has their favourite things and if you want to leverage the work of others you run with whatever everyone else is using unless you have funding and good reasons not to).

Re:I get the impression that (1)

tyrione (134248) | about a year ago | (#42808601)

You lost me at ``Most scientists are not programmers...'' schtick. Whether it was my Mechanical Engineering professors fluent in ADA, C, Fortran, C++ or Pascal or my EE professors in the same, to my Mathematics Professors all in the same, not a single CS Professor could hold a candle to them, unless we started dicking around with LISP, SmallTalk or VisualBasic for shits and giggles. In fact, they became proficient in these languages because they had to write custom software to model nonlinear-dynamic systems. Perhaps in the post 2000 era scientist group we have Matlab/Octave/R/Python lovers but the old school folks are hardcore in their knowledge of those languages.

Rarely does one find an expert in software development who is an expert in any Engineering, Physics or Mathematics field of research.

Re:I get the impression that (1)

solidraven (1633185) | about a year ago | (#42809581)

I disagree partially with what you said based on personal experience. As an EE student I had to learn to use Fortran for my thesis. I needed to run a large EM simulation and not a single affordable commercial program was able to run on a small cluster of computers that was available. So I resorted to using Fortran and MATLAB for visualisation. So I managed to learn basic Fortran over the weekend and then use it to write a working program for a cluster, all within 1 week time. I just don't think I could have done that with Python. Especially considering the time constraints I had in terms of runtime.

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42806745)

You miss the point. The pure CPU speed is important but appropriate language constructs (generators, coroutines) are equally important to deal with memory and processing complexity problems.

While your Fortran code will choke when it hits swap, then Python code might simply nicely fly and finish the task.

Re:I get the impression that (1)

solidraven (1633185) | about a year ago | (#42809667)

Nah, Fortran was designed with number crunching for scientific and engineering applications in mind. It won't choke, it won't stop. Fortran compilers are far smarter than Python when dealing with memory. The language was designed to allow the compiler to make assumptions to speed up computation and make for efficient memory management. But I'll agree that you shouldn't write the entire application in Fortran. For visualisation other languages are better suited (MATLAB/Octave comes to mind). You can have a python script to assign the tasks to the cluster. But for the actual calculations I'd still use Fortran. It's still the tool of the trade for very good reasons.

May I have a word (0)

Anonymous Coward | about a year ago | (#42808585)

Python isn't fit to run on a large cluster to simulate things, too much overhead.

Have you heard of Stackless Python [wikipedia.org] ? Your presumption that Python isn't fit for large clusters to simulate things may be news to the largest single instance human particapatory simulation ever done: New Eden. [eveonline.com]

Re:May I have a word (1)

solidraven (1633185) | about a year ago | (#42809755)

You're comparing two very different tasks. A game and a large simulation are very different things. Lets compare two extremes: EVE online and the FDTD algorithm (EM field solver). EVE Online has a lot of conditionals. It's very unpredictable in memory usage. But the FDTD algorithm has very different properties. It needs a lot of data, but there are no conditional expressions. Additionally what's needed from the memory is known long before it's ever needed. It just goes over the data every pass without analysing it. It simply does calculations. Do you see how this can be done efficiently on a pipelined CPU? You can ensure the data shows up on the right spot at the right time. The Fortran compiler tries to analyse the implemented algorithm and optimize these sort of things, that's where it strength lies. The same sort of compiler would be very difficult to write for Python.

Re:I get the impression that (1)

Dcnjoe60 (682885) | about a year ago | (#42808967)

You're dead wrong, nothing quite beats Fortran in speed when it comes to number crunching. If you need to go through hundreds of gigabytes of data and performance is important there's only one realistic choice: Fortran. Python isn't fit to run on a large cluster to simulate things, too much overhead. And lets not forget what sort of efficiency you can get if you use a good compiler (Intel Composer). You won't find Fortran on the way out over here, it's here to stay!

Isn't that the point of DARPA funding this project - to make it so Python is fit to run on a large cluster to simulate things? I do agree, though, that Fortran is here to stay. However, it is so specialized in what it does and that often a solution then requires multiple languages to get the task accomplished.

Back in the day (1970s) I had a professor who would say that you can write anything in anything. For instance you could write a business app in Fortran and you can use COBOL for plotting trajectories to the moon. But, why would you? Each excel at what they were designed for and create a lot of extra work trying to make them do what they weren't designed for.

Something like Python is good at doing a lot of different things, but not necessarily great at large number crunching/analysis. It seems like DARPA is wanting to change that. That doesn't mean that FORTRAN will be obsolete, but if successful, it does mean that Python can be even more useful in research than it is now.

Re:I get the impression that (1)

solidraven (1633185) | about a year ago | (#42809779)

Sure you can, any language that has a full feature set can do any task that the system is capable off. But efficiency is also important, and Fortran simply has so much advantages over Python. Complex data structures aren't needed for most simulations while they make optimisation so much harder. Additionally interpretation is a serious bottleneck.

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42806219)

You got to be fucking kidding. You either don't know what Python is, or you are completely clueless abou what Fortran is. Here's a hint: Python's main features are:
a) it exists and it is there
b) it is easy to use
c) it is very popular as a scripting language

The rest is just a natural consequence of smart people who don't have a lifetime available to wrap their head around C++ needing to pull some code together, and finding a language which doesn't demand that you sacrifice first child to write a hello world program.

Re:I get the impression that (-1)

Anonymous Coward | about a year ago | (#42806667)

The entire point of Fortran is that it has difficult-to-deal-with aliasing rules that make the compiler more free to produce optimized code. That's why it is suitable for things that require every last bit of performance you can wring out of it. Today probably you can get the same thing with C or C++ provided you are prepared to use things like restrict, but it used to be you couldn't, so Fortran ruled certain topics.

Python is an easy-to-use system with abysmal performance - expect 10-100x slowdown for code that runs in pure Python over a similar C version. If you can get things set up so Python is only gluing other C components together and the data never has to touch native Python data structures or loops, then performance will be fine, but now you aren't really coding in Python any more.

The point is, the purpose of Fortran and the purpose of Python are entirely opposed. They are exactly the opposite of each other. So it boggles the mind how you can think that Python can be Fortran "done right". So much so that now I suspect I got trolled. Well done, sir.

Re:I get the impression that (3, Informative)

Chrisq (894406) | about a year ago | (#42806799)

The entire point of Fortran is that it has difficult-to-deal-with aliasing rules that make the compiler more free to produce optimized code. That's why it is suitable for things that require every last bit of performance you can wring out of it. Today probably you can get the same thing with C or C++ provided you are prepared to use things like restrict, but it used to be you couldn't, so Fortran ruled certain topics.

Python is an easy-to-use system with abysmal performance - expect 10-100x slowdown for code that runs in pure Python over a similar C version. If you can get things set up so Python is only gluing other C components together and the data never has to touch native Python data structures or loops, then performance will be fine, but now you aren't really coding in Python any more.

The point is, the purpose of Fortran and the purpose of Python are entirely opposed. They are exactly the opposite of each other. So it boggles the mind how you can think that Python can be Fortran "done right". So much so that now I suspect I got trolled. Well done, sir.

Yes I understand, and many people made the same point. However Fortran was for a lot of scientists and engineers the hammer to crack any nut. It was used for simple "try outs" where performance wasn't needed, simply because it was the language that Engineers knew. I think the same thing is happening with Python now, it is the first and sometimes only language that many engineers know. Now for the performance issue, it will not give the best performance but packages like SciPy and NumPy do give very good performance (arguably by using these libraries you are just using python to string c functions together, but it is properly integrated). Tests show that you are getting about a third of the performance of Fortran [nasa.gov] , (with the exception of the Fortran DGEMM marix multiply which greatly outperforms Python and other Fortran variants). The typical engineering reaction to performance needs is to throw hardware at the problem, then optimise your algorithm, and only change language if absolutely necessary!

Re:I get the impression that (1)

pjabardo (977600) | about a year ago | (#42807277)

You are actually right but you are missing the point. Python doesn't compete with Fortran, it supplements it. With tools such as f2py, it is very easy to call fortran code from python (and there are tools that make it easy to call C/C++). This combination really potentializes both languages: bottlenecks use Fortran/C/C++ and the rest python. This combination is already popular: numpy/scipy is basically that.

I don't think that being easy is python's main advantage. Using a dynamic environment were you can type code that gets executed immediately and were you can explore the data is a really big help. On the other hand, the same could be done with R, Matlab, Octave or Scilab and it is done. In some ways these languages are better suited than python because they were designed to do math, or more specifically matrices/arrays very well and might have better syntax for that. But then doing anything else increasingly becomes a pain once the problem becomes larger or more complex and that's where, IMHO, python gains an advantage. Better module/OOP environment, better GUI,etc.

By the way, I work on scientific computing, using spectral element methods in computational fluid dynamics and I also work on a wind tunnel and I do lot's of data acquisition and processing. Right now I use C++ for lower level stuff (and bottlenecks) and R. I have been seriously considering switching to Python to have an easier environment to maintain.

Re:I get the impression that (0)

Impy the Impiuos Imp (442658) | about a year ago | (#42806759)

Python, the indent-based, block-structured language? I have about 6 months experience with it, I guess it's not enough to see the advantage of it qua-number crunching syntax.

Oh well, it's just 0.75% of one day's borrowing.

Re:I get the impression that (1)

SpzToid (869795) | about a year ago | (#42806867)

No one seems to be pining away for Fortran programmers. At least not much ayways. A quick 'n dirty search on Dice.com yields 46 results, (and no doubt a few are doubles).

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42806893)

Well, if that's your starting point, you've already failed.

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42807109)

Perhaps true.... with the difference being that FORTRAN is fast as hell and Python is blindingly slow.

Re:I get the impression that (0)

Anonymous Coward | about a year ago | (#42809027)

I know this won't get much love, but julia is Fortran done right. There was slashdot article about it: http://science.slashdot.org/story/12/04/18/1423231/julia-language-seeks-to-be-the-c-for-numerical-computing

Matlab (1)

Anonymous Coward | about a year ago | (#42806065)

Bye-bye Matlab. I liked your plotting capabilities, but that was about it.

Re:Matlab (2)

sophanes (837600) | about a year ago | (#42806637)

matplotlib already does this in conjunction with Numpy and Scipy - its plotting quality and flexibility compares favourably to Matlab.

Its biggest drawback is that it is pretty glacial even by Matlab's standards when rendering large datasets (think millions of points). I'm not sure whether matplotlib or the interactive backend is at fault, but anything DARPA can do to improve the situation would be welcome.

Congratulations are in order (1)

Anonymous Coward | about a year ago | (#42806073)

Seriously- the Continuum folks do great work, and after hanging out with them a bit at the last PyCon I was really impressed with where they seemed to be headed. Hope they make it there.

Python 2 or 3? (3, Interesting)

toQDuj (806112) | about a year ago | (#42806095)

So is this going to focus on Python 2 or 3? Might be a reason to upgrade..

Re:Python 2 or 3? (4, Informative)

SQL Error (16383) | about a year ago | (#42806279)

Both. The prebuilt "Anaconda" distro defaults to Python 2.7, but it also works with 3.3 and 2.6.

Wrong language (4, Funny)

Dishwasha (125561) | about a year ago | (#42806135)

The put the money in the wrong place. They should have put it in to R which very popularly interfaces with Python.

Re:Wrong language (0)

Anonymous Coward | about a year ago | (#42806171)

Maybe that's all the company is going to do... give new interface functions and pocket the money.

Re:Wrong language (1)

toQDuj (806112) | about a year ago | (#42806187)

Perhaps. After all, it is in the nature of companies to ask as much money as possible for as little work as possible.

Re:Wrong language (3, Informative)

SQL Error (16383) | about a year ago | (#42806337)

DARPA runs a lot of these research seed programs, putting a couple of million dollars into a bunch of different but related research projects. In this case the program budget is $100 million in total, and Continuum got $3 million for their Python work (Numba, Blaze, etc). Some of the program money may have gone to R as well; there's a couple of dozen research groups, but I don't have a full list.

Re:Wrong language (1)

csirac (574795) | about a year ago | (#42806473)

Wow, I hope not. As much as I am actually a Ruby fan at heart; and as much as I appreciate the R community and everything R has done, it always seems much easier to write slow and/or memory-intensive R code than in Python. Perhaps I never quite spent enough time with it but there are many corners to the language which seem unnecessarily tedious. And no references - variables are all copied around the place, which is expensive. I know, I know... worrying about pass-by-value and efficiency of assignment statements (well, R doesn't really have statements; everything-is-an-expression) means I'm doing it wrong, but most code I debug is written by someone else who is also doing it wrong..

Then there's pandas [pydata.org] and the rest of the SciPy stack, which is the only reason I used Python over Ruby (I had also considered Perl+Moose) in my last project. pandas is extremely fast, and I was able to write some quite advanced data processing stuff which would normally have needed far more effort in Ruby or Perl.

Re:Wrong language (2)

hyfe (641811) | about a year ago | (#42807291)

http://en.wikipedia.org/wiki/R_(programming_language) [wikipedia.org]

R is a statistical programming language. It has lots of neat methods and functions implemented, and is rules the world of statistical analysis.. which is kinda cool, since it's also open source.

It sits pretty much halfway between Matlab and Python.. It's pretty usuable and convenient because of the huge library, but as a programming language it just, well, sucks ball. Building up the objects some of the methods there need, if you get data from an unexpected source, is just an utter pain in the bottomhole.

Re:Wrong language (0)

Anonymous Coward | about a year ago | (#42808519)

That was a massive long rant about Ruby, the problem is you've got the wrong language. R != Ruby.

Re:Wrong language (1)

drinkypoo (153816) | about a year ago | (#42807445)

Others have complained about limitations of R in this very thread, so it doesn't seem as cut-and-dried as you make it out to be. Python is the popular language of this particular fifteen-minute period, so it's the logical choice to put the effort into. Scientists would like to benefit from language popularity too.

Good news for the Python community (3, Funny)

kauaidiver (779239) | about a year ago | (#42806145)

As a full time Python developer for going on 6 years this is good to hear! Now if we can get a Python-lite to replace Javascript in the browser.

Re:Good news for the Python community (0)

Anonymous Coward | about a year ago | (#42806179)

Wow, I've never thought of this. Imagine modules! :O

Re:Good news for the Python community (1)

lattyware (934246) | about a year ago | (#42807019)

Yeah, the issue is that Python is pretty hard to sandbox, being the hugely dynamic language it is. I imagine it would take a lot to get the browsers to stop working on their JavaScript implementations that they have sunk insane amounts of time and effort into, and start something brand new.

Trust me, I'd love to see it happen, but I don't think it will.

Re:Good news for the Python community (0)

Anonymous Coward | about a year ago | (#42807057)

Yeah, as a full time $LANG developer for going on $RAND years this is good to hear! Now if we can get a $LANG-lite to replace $LANG_I_KNOW_BUT_DONT_MASTER in the $_PLATFORM.

Enthought Python (1)

screff (1201383) | about a year ago | (#42806147)

I wonder how this effort compares to the work being done by Enthought Python. Hopefully it is more open and freely available to all, or better yet, incorporated into the mainline python distro.

Looking forward... (1)

Anonymous Coward | about a year ago | (#42806175)

... to Python operated railguns. That would be awesome :D

Re:Looking forward... (0)

Anonymous Coward | about a year ago | (#42806587)

Looking forward to Python operated railguns. That would be awesome :D

However, when you try to abort the launch command with a CTRL^C

It JUST LAUNCHES EVEN HARDER.

Re:Looking forward... (0)

Anonymous Coward | about a year ago | (#42806641)

However, when you try to abort the launch command with a CTRL^C

It JUST LAUNCHES EVEN HARDER.

That's not a bug, it's a feature!

Re:Looking forward... (0)

Anonymous Coward | about a year ago | (#42807715)

How would it be more awesome than operated railguns?

Hope these guys work with Wes McKinney (Pandas) (1)

bwbadger (706071) | about a year ago | (#42806441)

This DARPA work sound like it's in the same space as the Pandas library. I hope they can work together.

Its all going on making the documentation legible (0)

Anonymous Coward | about a year ago | (#42806767)

Only half a troll, seriously the sphinx/numpy documentation themes are terrible compared to javadoc standard.
Finding epydoc has dropped my swearing to lines of code ratio by heaps.

Re:Its all going on making the documentation legib (1)

lattyware (934246) | about a year ago | (#42807023)

Seriously? Sphinx makes beautiful documentation that is easy to find your way around. Compared to the ugly-ass JavaDocs that are painful to browse through, I wouldn't even give it a second thought.

There's more to XDATA (2)

seekthirst (1457205) | about a year ago | (#42806953)

It's strange that this article focused on Python and Continuum when there is a much bigger story to be had. The XDATA program is being run in a very open source manner, and there will be a multitude of open source tools created and delivered by the end of the contract. The program is focusing on two major tasks: the analytics/algorithmic tools to process big data; and the visualization/interaction tools that go along with them.

Big Data != Analytics (2)

michaelmalak (91262) | about a year ago | (#42808527)

The summary and article seem to conflate Big Data with Analytics. These days the two often go together, but it's quite possible to have either one without the other. Big Data is "more data than can fit on one machine", and analytics means "applying statistics to data". E.g. many Big Data projects start out as "capture now, analyze a year or two from now," and maybe just do simple counts in the interim, which is not "analytics". And of course, many useful analytics take place in the sub-terabyte range.

The irony with this story is that Python is useful for in-memory processing, and not "Big Data" per se. To process "Big Data" typically requires (today, based on available tools, not inherent language advantages) JVM-based tools, namely Hadoop or GridGain, and distributed data processing tasks on those platforms require Java or Scala. Both of those platforms leverage the uniformity of the JVM to launch distributed processes across a heterogeneous set of computers.

The real use case here is one first reduces Big Data using the JVM platform, and only then once it can fit into the RAM of a single workstation, use Python, R, etc. to analyze the reduced data. So typically, yes, these Python libraries will be used in Big Data scenarios, but pedantically, analytics doesn't require Big Data and Python isn't even capable (generally, based on today's tools) of processing raw Big Data.

Imagine the research if we took all lobbying (1)

tyrione (134248) | about a year ago | (#42808615)

cash and put it to advancing applied sciences to better the nation. We piss billions down the drain marketing to morons and yet whine about spending billions on DARPA, DoE and whatnot. This county is truly too stupid for its own well-being.

Pypy (0)

Anonymous Coward | about a year ago | (#42809429)

So how much of that $3M will go towards development of the NumPy port to Pypy? I'm guessing 0%, which is unfortunate, since that is one of the best places to push the state of the art in speed for numerical processing with python. The Pypy community has the modest goal of raising $60k for that work (just 2% of the grant to this company), and they are still only 3/4 of the way to achieving those funds after a year with their shingle out.

http://pypy.org/numpydonate.html

Status: Won'tWork (0)

Anonymous Coward | about a year ago | (#42809863)

Speaking as someone who's been employed by Python nearly a decade, and prior to that was involved in porting scientific Fortran to C & Java (Dear Fortran guys -- I so sorry, it was the job the idiots paid for because management thought Fortran was dying).

It won't work.

It's not that Python can't do it. It's that without a real programmer, python is slow. Even with a real programmer, it's slower -- but that's /often/ recoverable in many ways, particularly in development time.

Scientists that don't code have an easier time learning python. Scientists that do code (well) can learn python, but are often going to want to move into other languages because they *always* want more data and more refined models. I've seen them learn java and c -- but that's a total nightmare, worse than even python.

Contrast to a friend who thinks they can program but can't even fizzbuzz -- they have a dataset that they think is too slow for python. It is too slow the way they do it, but they've copy-pasted a O(log log N) algorithm so badly it's at least O( N log(N)) . Going out of asymptotics, there really is a constant of about "5" before that for all the extra iterations and wholly unecessary subdivisions they do, plus the output is total shit because they don't understand what it means to working in floating point. So a process that I can finish on my desktop in a few hours as long as I have enough RAM takes them three weeks to run on a server.

The thing runs -- except for the 10% of the data they drop, but it's a wholly unreadable mess.

Some of the people that want to do this are "real programmers" -- but many are scientists that just want a visualization and don't give a damn what tool does it as long as the output looks like what they think it should.

They're the same researchers that cut and paste from stackoverflow or expert sexchange, and who just drag and drop code around in notepad trying to get rid of errors.

They'll get an example given a CSV to make a beautiful clustergraph from examples or a friend that knows it, but they'll still develop deeply flawed research and modeling code and never know why or catch it.

Doing this in python may make some of the analysis more accessible as a whole, but it won't fix the 'problem' that most scientists can't actually program.

Maybe they shouldn't have to -- but somebody does.

The problem is really best summed on when describing a bug to a new programmer that wasn't great at math, and clearly used to having a single error mess them up. They figured they could change one thing and fix a totally flawed algorithm...

"Just tell me what line the bug is on"

The answer was : "All but these two".

To the new-non-programmer...this answer was inconceivable. They 'knew' what they told the computer to do, and it was being unreasonable in interpreting their source according to the rules of the language. There had to be one line to fix it -- the notion that the fundamental structure of their logic was wrong was so counterintuitive they didn't believe it even when pointed out.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...