Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Toward a New Unix Shell

Tetsujin (103070) writes | more than 5 years ago

User Journal 2

One thing that's interested me a lot lately is the idea of improving upon, or reinventing the Unix command shell.

One thing that's interested me a lot lately is the idea of improving upon, or reinventing the Unix command shell.

Now, there's a certain class of Unix users who have stuck with the command shell while GUIs have come and gone, been released, overhauled in various releases, and so on until new versions barely resemble old ones. One of the problems I face in this kind of plan is that many of these users are fairly attached to the tradition. They use the current command shells because it's the environment they like, and many of them don't want to use a "reinvented" shell. When in my design I start thinking about giving the shell its own type system to be used over pipelines, or devising a set of rules that tools would have to follow to "play nice" in this environment - I know that a lot of my target audience just flat out isn't going to like it. This concerns me. But still the idea of this environment appeals to me. Somehow I want to make it work, and make it stick.

Now, the first question one has to ask about something like this is, why? Why should the shell change? There are a few reasons I have in mind:

  • If the shell could know things like the type of a piece of data, or the kinds of command-line options a tool supports, it could offer context-relevant help.
  • In present shells, when connecting the output of one tool to the input of another, the user must insert parser/serializer steps. I feel it's appropriate for the shell to be able to offer a greater degree of help than that.
  • I think there's real value in giving the shell a way to deal with "objects" and higher-order programming in general. For instance, an object could be an XML parser or an open network connection or a window into a running application: for the lifetime of that object the shell should be able to issue commands to that object - and when the shell's last reference to that object is destroyed, some action (like closing a connection or killing a process) may be appropriate. There are some command-line tools that implement this sort of behavior themselves, but I think it would be very nice if it were a real feature of the shell.

Also, I believe the current model of the shell has fallen behind how people actually use their computers. For instance:

  • Modern GUIs offer a lot of useful functionality - but the extent to which this functionality is integrated into the command shell is rather limited for various reasons. For instance, why isn't the volume manager or the wi-fi manager that I used inside the GUI also available outside the GUI? The basic answer is that command-line tools aren't well suited to that kind of usage profile in which they are started as a service and then, while running, receive and respond to outside commands. The framework for such a thing simply isn't in place.
  • Scripting languages like Perl and Python offer large and useful libraries that perform all kinds of different features. Why can't shell scripts access these? The basic answer at present is that the programming language provided by the shell lacks the constructs necessary to usefully interface with these utilities. The lack of "object" support (and, specifically, lack of a good mechanism to start something, keep it running, and interact with it, and shut it down when finished), the lack of any sort of namespace support, and the fact that any data going into or coming out of such a library has to be arranged in some ad-hoc format for which the shell provides no specific support - all of this severely hampers the ability to expose these libraries and the practical benefits of doing so.

Microsoft has already come up with their own solution: "Powershell" - a command shell somewhat similar to cmd or a Unix shell, but with support for "commandlets" - commands on the search path which are actually .NET classes which are dynamically loaded and run as part of the shell's process. These "commandlets" exchange .NET objects as their input and output. The shell can then store these .NET objects in environment variables for later recall (and keep track of when the object is no longer referenced, and delete it) - this goes a long way toward usefully exposing Windows API functionality within this shell.

My goal is a bit different. Linux has no standard representation for "objects" (and I'm not in a hurry to embrace Mono, let alone encourage others to do the same) so Microsoft's approach isn't suitable for my goals. Furthermore, without the ability to restrict the behavior of a piece of code within a single process, it becomes more important for the stability of the shell to continue to have tools be separate processes. Therefore, whether these outside processes communicate via shared memory or pipes, either way they need to respect a few common conventions about how data is formatted, and (in the case of "objects" - data in which it's important to know when it's time to destroy it) how to manage object lifetime.

One of the typical complaints is that a plan like this requires all shell tools to agree upon and use the same set of rules for how they format their data. This would be a real problem: people would be slow to move to this format, which in turn would make the shell less useful (since it would lack the tools to run in its "enhanced" environment). No one would want to write a tool that runs only in a new, unproven shell, and no one would want to either shoe-horn their problem into an uncomfortable data format, or waste CPU time by translating their optimal data format into the one the shell wants.

So clearly putting everything into a single data format wouldn't work. And in general, the fewer things I "mandate", the better. So my idea is this: tools can go on communicating with whatever encoding makes sense for them, but there should be some shared means of identifying what that encoding is. In order to do this without requiring, you know, every program in the world to be changed for compliance, there has to be a way to provide this information out-of-band - and, presumably, statically. That way, even if the binary itself has no provisions for working nicely in my shell, the end-user can work around that, without changing source code or recompiling, and without needing the authority to install such a workaround system-wide.

Of course, there's still all kinds of problems with this plan. Among other things, this new shell somehow has to provide a nice new environment, while simultaneously working nicely with things not made for it. For instance, if I provide my own version of "find" - do I have to give it a different name so it won't conflict with the GNU find everybody's used to running? There seems to be a never-ending supply of small problems that need to be solved before this design can really go anywhere. But if all goes well, then maybe someday I'll get it written and you'll give it a try. :)

cancel ×

2 comments

Sorry! There are no comments related to the filter you selected.

What? (1)

Qzukk (229616) | more than 5 years ago | (#27424785)

For instance, why isn't the volume manager or the wi-fi manager that I used inside the GUI also available outside the GUI

It's all there. You're just not using the commandline version of the commands ;) For instance, mdadm for the volume manager.

if I provide my own version of "find" - do I have to give it a different name

Actually, no. Just be sure you obey standard builtin conventions (eg, if someone types /usr/bin/find, they don't want your find), and possibly include an option to disable certain builtins if you're overriding a lot of popular programs. For an example, in bash, cd into a directory that is actually a symlink to another directory and run "pwd". Then run "/bin/pwd".

requires all shell tools to agree upon and use the same set of rules for how they format their data

I've seen something along these lines somewhere, but can't remember if it was a real thing or someone proposing such a thing. Personally, I think the issue is that the vast majority of tool permutations will never occur (your volume manager will never speak to your wifi manager, or at least not say anything meaningful) so having a global data format is not worth the effort. As long as the sets of tools that are used together can agree on a format within that clique, this is sufficient.

What "what"? (1)

Tetsujin (103070) | more than 5 years ago | (#27571939)

For instance, why isn't the volume manager or the wi-fi manager that I used inside the GUI also available outside the GUI

It's all there. You're just not using the commandline version of the commands ;) For instance, mdadm for the volume manager.

True - and one could also configure wi-fi from the command line as well. What gets me, however, is how little integration there is between the two sides. It seems like there's these useful bits of functionality that may or may not be on both the shell and the GUI - and if they are on both, the two don't work together for the most part.

I think there was a slight miscommunication, or else I don't understand what mdadm is for. I was speaking of things like gnome-volume-manager - software that monitors device insertion events and mounts plugged volumes with permissions favorable to the user currently logged in locally to the machine. That's a pretty significant piece of functionality - and if you want it outside of the windowed environment you have to find another piece of software to give you that capability.

if I provide my own version of "find" - do I have to give it a different name

Actually, no. Just be sure you obey standard builtin conventions (eg, if someone types /usr/bin/find, they don't want your find), and possibly include an option to disable certain builtins if you're overriding a lot of popular programs. For an example, in bash, cd into a directory that is actually a symlink to another directory and run "pwd". Then run "/bin/pwd".

That approach has some issues I'm not too happy with. Specifically it requires the user (or script-writer) to know the absolute path of the binary they want to run in order to be 100% sure they're running what they think they are. These paths can vary from system to system (for instance, on an OS with its own versions of find, locate, etc. GNU find may still be present, but it most likely won't be /usr/bin/find. Maybe /usr/local/bin/gfind or something like that...) I'd like to come up with a good solution to this - like a syntax that says "search the path for a command called 'find' - specifically, the GNU version"... 'Course that requires a bit more cooperation from the tools themselves, at some level. There's no current mechanism for a tool to publish information about itself in a way the shell can understand.

When talking about GNU find as opposed to another version, the differences aren't too severe. You do without '-print0' and other GNU extensions, and that's pretty much it. The "find" I would write would have a different problem - the data it outputs would be formatted differently. If I don't have a good solution to that problem then it could cause trouble - for instance if in version 1.0 I don't provide my own version of a tool, but in version 1.1 I do - it could break people's scripts, unless I have a good strategy for dealing with such issues. Of course, that assumes that the adoption rate is greater than zero. XD

requires all shell tools to agree upon and use the same set of rules for how they format their data

I've seen something along these lines somewhere, but can't remember if it was a real thing or someone proposing such a thing. Personally, I think the issue is that the vast majority of tool permutations will never occur (your volume manager will never speak to your wifi manager, or at least not say anything meaningful) so having a global data format is not worth the effort. As long as the sets of tools that are used together can agree on a format within that clique, this is sufficient.

The one example I can think of (on Linux, I mean) of an approach like this was an XML-based shell. Commands were rewritten to output XML and the shell was made to recognize that and format it nicely. I can't remember what the shell was called, and I no longer have it installed.

But I want to emphasize that this isn't the approach I'm advocating. What I advocate is encouraging tools to adopt a common means of identifying the formats they use for reading and writing. Trying to get all tools to use the same format for I/O is just dumb. It ignores the fact that specialized file types exist for good reasons.

It's true most possible tool-to-tool combinations simply don't occur because they're not useful - you're not really looking at N^2 possible two-tool combinations. That said, however, the combinations I'm most interested in are those between the shell itself and the tools running in it. The shell ought to be a good interfacing language - it should be very powerful at extracting data from one tool, manipulating it, and feeding it into another tool. From that perspective, current Unix shells are very weak. They're totally ignorant of the structure of the data coming down the line, so the burden's on the user to extract the meaningful bits of it... and there's only so much you can do with the data when you've got it. You can store it in environment variables, write it out to another file, or you can send it to another command... The provisions for working with the data in the shell itself are generally very limited.

No one would design a programming language like this. In Python there are hundreds of different modules - most of those modules don't interact with each other directly but they all interact nicely with the Python language. One doesn't run a Python module, for instance, to extract a bunch of data from an XML file, and then expect to need to parse the result of that evaluation - rather, you get a Python list. Similarly, if you want to use the parser to read a field from that XML file, and then make a decision based on that result that affects what field you read next - you don't need to build that logic into the XML reader, nor do you have to initialize the XML reader, read the field, shut it down, make your decision, and then (wastefully) restart the XML reader again - you can initialize it once, keep its running state in memory, and repeated queries to it can be handled efficiently.

These are things I want for the shell. The shell, as a programming language, should know about the structure of the data it receives, and it should have an established system for operating a tool while it is running. (Current mechanisms for this are rudimentary at best...)

Anyway, thanks for responding. It's hard to express my points effectively and I know I make mistakes, so I appreciate being able to bounce the ideas off people and see where I go wrong. :)

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>