Just a bugfix release, but this is the first version that can actually run news.arc in full. Performance is still rather dismal though, sorry about that, but I'd like to get it to work right first before getting it to work fast. :)
Hmm, even if the package for doing this comes with Ruby MRI, I'm not sure there's a corresponding out-of-the-box way to take these s-expressions and get running code. But here's a five-year-old article talking about Ruby2Ruby, a library that converts these s-expressions to strings: http://www.igvita.com/2008/12/11/ruby-ast-for-fun-and-profit...
I don't think a third-party tool should count as language support, unless (and inasmuch as) the people maintaining the language regularly go out of their way to avoid breaking that tool.[1][2] This is where I draw the line, because of how it affects backwards compatibility.
Everyday language users tend to think of language updates as backwards-compatible as long as their existing code has the same behavior. But tools like eval(), JavaScript minifiers, and Ruby2Ruby place the bar higher, since their input is arbitrary language code, and it might be written in a newer language version than the tool itself. Incidentally, even a program that calls these tools will continue to work after a language update, as long as the input of these calls is all generated programmatically, rather than directly taken from the program's input.
[1] Well, I guess the term "third-party" might not apply in that case!
[2] I have no idea how much I'm actually talking about Ruby2Ruby, and this Ruby s-expression format in general, when I make this comment.
I assume by "s-expressions" we mean nested lists whose first elements are often symbols which represent operators. Apparently, Ruby already has built-in support for that code representation: http://stackoverflow.com/questions/6894763/extract-the-ast-f...
"[...] though how it could do that without also transforming into Lisp is another matter."
My experiments with Nulan have convinced me that it's easier (and better) to start with a Lisp and then add Ruby syntax which desugars to S-expressions, rather than trying to turn Ruby into a Lisp.
I should point out there are a few languages I missed, and possibly others:
| Nulan seems like an that also compiles to Javascript
| Rainbow is a Java implementation of Arc
| RainbowJS seems like a port of Rainbow to Javascript
First of all I don't like the fact that the present reference Arc depends on MzScheme/Racket. This makes setting it up and using it a chore, and I think this is a not insignificant factor in what hobbled wider acceptance of the language. I wanted to be able to just do a ./configure ; make ; make install and start using it, the way I can presently do with, say, Ruby. As it is you have to install a rather large Scheme interpreter (Racket 5.1.1 has over 500,000 lines of code; it might have been a little different if it were a smaller Scheme like scheme48) just to get started.
I've also wanted to have a language like Tcl or Lua that one could easily embed as a scripting language in a larger project. This is one of the things that Arcueid is designed to do: just link a single library and have an interpreter available with a relatively simple API. Arcueid is also be designed to be easily extensible via a simple foreign function interface because I don't want to use it for just programming web applications, although that is something that I would doubtless use it for a lot as well.
Writing in C (or something that compiles to C) is also probably the only way to get an Arc faster than the implementations we have today. Of course my present implementation is still somewhat slower than Arc on Racket, but then again the goal for now is to build a foundation upon which further optimisation can later be done. As always, it's better to get it to work right first, and then make it run quickly later... Eventually Arcueid will have a JIT compiler of sorts but that's a little further down the roadmap.
As for Arc, it seems like it's much better designed than any other Lisp dialect out there. The parentheses are still there, but they are used more sparingly than Scheme, which seems out to irritate you by putting extra parentheses for something so basic as a let binding when you could easily do with less and have more comprehensible code. This fact was illustrated to me about a year ago when rocketnia gave me versions of the Hanson-Lamping algorithm for dynamic-bind in Scheme and in Arc here:
The contrast was rather startling to me. I understood the algorithm immediately on seeing the Arc code.
Another thing about other Lisps that is really irritating to me are the overly long function names like multiple-value-bind and invoke-restart-interactively and stuff like that. They make a program look rather cluttered. Arc has for the most part avoided excessively long identifiers in the core and that has been good.
Macros are something that I've also been really interested in, and something that looks like is only really possible with a Lisp. Ruby is my programming language for most of my day to day work and the simple metaprogramming it makes possible makes me wish that it could do macros too, though how it could do that without also transforming into Lisp is another matter.
It might be simplest to just install postfix (I've tried it several times, and the three-screen configuration wizard Just Works) and then follow the steps in http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix... to have postfix route through SES.
I'll update anarki later today with a sample for sending email through postfix.
If I had to give a reason for making Arc/Nu... well, that's really complicated and involves a lot of history and backstory.
Basically, before Arc/Nu, there was ar, which was created by awwx. It was a rewrite of Arc in Racket, where the compiler was intentionally exposed to Arc. This meant you could not only access the compiler from Arc, but could actually change it!
So rather than writing the compiler in Racket, you'd write the minimum necessary to get things going, then write the rest of the compiler in Arc. This naturally made the compiler very easy to hack for Arc users.
I really liked that, but I also disliked certain things in ar. One of those is that all the compiler stuff is prefixed with "racket-", which is really verbose.
Another dissatisfaction is that although awwx did add in a module system for Arc, it was very... uh... basically you could create multiple Racket namespaces, which are completely separate from each other, and load Arc into each one.
The problem is that because they're separate, it's really difficult for them to communicate with each other. Which kinda defeats the point.
Another problem is that you had to reload Arc every time you created a new namespace, which meant ~1-2 second pauses every single time. Which is pretty ridiculous.
I felt I could do better, so I made a fork of ar and hacked away on it. But having to use the "racket-" prefix really bothered me, so I experimented with ways to get rid of it.
Eventually I decided to start over with pg's vanilla Arc 3.1, rather than ar. I fixed it so it would work in Racket rather than mzscheme, and then I did significant refactoring, making things faster, using new techniques I had discovered, etc.
I also added in a module system which was exactly the same as ar's module system, except that you could make a module that inherits from another module. This solved the "have to reload Arc all the time" problem, and also made it a bit easier to communicate between modules.
The end result was Arc/Nu. It had all the benefits of ar, all the benefits of Arc 3.1, and none of the downsides of either one.
In the end, though, Arc/Nu's module system was terrible. Yes, you could now make modules that inherit from other modules, but I kept running into all kinds of problems, both theoretical and practical. But it was the best module system I knew of.
I wrote some programs in Arc/Nu, and after many months passed, I discovered hyper-static scope.
Before I continue with this story, I'll need to explain Nulan. Nulan is a Lisp I created from scratch. It has hyper-static scope, hygienic macros, and it compiles to JavaScript and can thus be used in web browsers.
I decided to make Nulan because I wanted to experiment with cool new things like hyper-static scope, but I couldn't do that in Arc/Nu because that would break backwards compatibility with Arc 3.1. I also wanted a language for web browsers that would be better than JavaScript.
While implementing hyper-static scope in Nulan, I discovered how amazing boxes are. Seriously, they're great. Using them, you can create a module system which is easily thousands of times better than any other module system I had ever seen.
After that, I realized that I had to change Arc/Nu to use boxes. And so I did. The module system in Arc/Nu is now amazing, and no longer based upon Racket namespaces.
As for whether I'm still using Arc/Nu, the answer is yes. I like using it for writing shell scripts, since I find Arc vastly better at that task than Python or Bash.
Great question; I'm looking forward to other people's answers.
"When one writes a new tool it's usually (hopefully) to satisfy a pretty frustrating need."
Building a programming language is a pretty natural itch. For a certain kind of programmer, at least. I'll give you some reasoning and rationalizations for wart, without any certainty that I can separate the two.
I've been vaguely aware of lisp almost since I was exposed to computers, back in 1996. But in India algol-style languages reigned. Basic, fortran, cobol, pascal, C/C++. A little behind the times. I took a lisp course when I moved to the US back in '99 and I could program little things[1] in common lisp after that. But I'd still stare at things like the recursive definition for reverse and wonder precisely what it was that made it so different from what I'd write in C. I periodically go back to that question, but more on that further down.
After programming mostly in C for 10 years, burning out, recovering, the usual coming-of-age stuff, I found myself in the bay area doing rails and the startup scene for a couple of years ('07-'10). I'd been interested in social software and recommendation systems since Reddit got started. I even signed an NDA with the Reddit guys to get at their code and try to build a recommendation system for them. But the lawyers took too damn long and I'd lost interest by the time they sent it to me (sometime '07). But around '08 I started thinking about recommendation systems again. Arc was out by this point[2], and it acted as a great gateway drug. Here, finally, was a lisp that helped filter out all the cruft and let me focus on how lisp was different from C, and that was ergonomic enough that I had to work to stop programming in it. At some point I looked at the logs generated by news.arc, and noticed that they were just s-exprs that could be read back in if I so desired. Hmm, crazy idea: a recommendation system that can see the recent actions in a session to inform recommendation decisions. The easiest way to do that would be to stick with lisp.
So I left rails behind and started a fresh (and my final, so far) startup in arc, to build a feedreader that didn't require understanding RSS. I never did do much with the idea of adapting to the current session, because I could never get to a point where I knew enough about a user to make any intelligent decisions. Providing a bootstrap experience that kept them coming back long enough turned out to be an eternally unsolved problem. I found out the importance of UX and how much I sucked at it. A year later the project was dead and I was working at Google. But the interest in lisp stuck, and I continued doing stuff with it, noodling on a series of little questions about its design, why it did things this way and not that. Sometimes I learned the hard way something people knew back in the '60s. Occasionally I discovered something new[3].
But I was still dissatisfied. One of the things that had bothered me all through this period was webservers and memory leaks. No matter what platform I used -- rails, python, racket, common lisp -- it seemed any non-trivial website would periodically run out of memory and die. Everywhere I looked, people dealt with this in a brutal way: by periodically restarting their servers and clearing the slate. Ugly, and it seemed to point at an endemic problem: languages spent tons of effort tweaking their garbage collectors, but programmers would mess up in some subtle way and accidentally prevent stale objects from being collected. Often this was because of interactions between multiple projects. Often there were no tools for the low-level debugging required. Or if there was, you needed to learn too much about the high-level internals of the stack you were relying on.
There were other dissatisfactions. Common lisp was a lisp-2, and its APIs were ancient and baroque[4]. I could sense that I could get used to them, but the mind rebelled at polluting my brain with all those warts. Racket was more modern and a lisp-1, but it gave up a lot of the dynamism of lisp with its phase-separation rules, hygienic macros and module system. Both had oodles of documentation, but common lisp's docs were poor and examples often wouldn't run. It had a 'standard' constraining creativity and unionizing bad APIs with lots of detailed rules, but the standard hadn't been updated in years so you were often doing stuff out of its ambit anyway (I still don't understand its condition system). It was the worst of both worlds. Racket has a much nicer documentation system, and yet it was too overwhelming for reasons I still struggle to pinpoint[5]. At some point, it seemed, the documentation was complex enough to ask, why can't I just deal with the codebase directly rather than some interface to it? How can I fork racket and throw out hygienic macros until I gain the wisdom of their need?
So eventually, after leaving arc, after trying to build arc out of common lisp macros[6], after some time agonizing over just switching to Factor[7], I bit the bullet, threw out all the dependencies, and returned to my roots in C. My goals were to create a dynamic lisp-1 with non-hygienic macros where I could define code in any order[8], and to do so with as little code as possible, in such a way that others could come after me and query the design space I had explored, asking what-if questions: what if I want to turn off this feature? Why do we implement this like so? The goal was never to build a 'real' language the way the world thinks about it. I wanted to see how far I could push a design without any black-box interfaces, where programmers had to understand the entire stack, and therefore where the entire stack had to be simple enough to fit in a single head[9], one that encouraged the combination of high- and low-level fluency necessary to track down problems like memory leaks, one where the documentation was intertwined inextricably with the implementation, so it couldn't go out of date and it could be queried interactively[10]. One that didn't spend thousands of lines of code trying to 'optimize' arbitrary code, but provided programmers the tools to measure their applications and attack bottlenecks for themselves without compromising readability, the way 'macros' provide a la carte mechanisms for reducing our own boilerplate. That's The Lisp Way.
What I have so far is a far cry from this goal. Sometimes I worry that I'm just playing in the shallows where it's fun[11] and avoiding the scary abyss. Programming languages are fun, but if I can stay disciplined wart will be more than a language. And nobody will have to ever rediscover why something is built just like so.
[5] Many of my initial annoyances were bugs in arc at the time (http://arclanguage.org/item?id=13616), or a result of the legacy mzscheme language (since fixed by Arc/Nu). Would I have embarked on this if the timing had been just a little different?
[10] I don't want to require new programmers to understand every last line of code before they can use a program. I want the program to reward curiosity, so that newcomers are able to drill down into the right bits when they run into problems, to answer questions for themselves without needing to find an expert to talk to (http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn...), and to iteratively try tweaking the code with instant feedback about regressions they introduce.
"I don't know if it's worthwhile to complicate the notion of "list" just to accommodate a special case of argument lists."
Yeah I see your point. My aim was to extend the correspondence between the syntax for vararg params, rest params and lists of regular params. But those features merely match a (proper) list of args to some template. I'm entangling the template with the notion of a list. Hmm..
Sounds like a good topic, and I'd be interested in hearing to hear your thoughts on this. I've been thinking about the same kind of thing, but I'm looking to use it in a weakly typed subset of a statically typed language. My motivation is mostly to see if we can make it convenient to externally manage the memory needs of otherwise encapsulated programs.
This is probably a discussion for another thread. ^_^;
I think the membership check you're using is like this:
(The final cdr is an element iff it isn't nil.)
If the list is...
A cons cell:
If the car is the value we're looking for, succeed. Otherwise,
continue by searching the cdr.
Nil:
Fail.
A non-cons, non-nil value:
Continue by comparing the list to the value we're looking for.
I feel we could simplify this specification by rewriting the last two cases in one of these ways, ordered from my least to most favorite:
(The final cdr is an element.)
(This breaks (pushnew nil ...) in existing Arc code.)
...
A non-cons value:
Continue by comparing the list to the value we're looking for.
(The final cdr is not an element.)
...
A non-cons value:
Fail.
(The final cdr is always nil and is not an element.)
(Arc 3.1 already uses this.)
...
Nil:
Fail.
A non-cons, non-nil value:
Raise an exception.
By using the "element iff it isn't nil" approach, you're able to use 'pushnew to traverse the simple argument lists you build as intermediate results of that 'make-br-fn implementation. But I don't know if it's worthwhile to complicate the notion of "list" just to accommodate a special case of argument lists.
I wonder if the compiler can't see through a 'procedure-rename call to realize it can still apply optimizations to the code. Like, maybe it can optimize ((lambda (x) ...) 2) but not ((procedure-rename (lambda (x) ...) 'foo) 2).
Well! I just learned something new! Racket's "procedure-rename" is ridiculously slow. In case you don't know what I'm talking about, it's just a function that lets you rename functions:
(procedure-rename (lambda () 1) 'foo)
Anyways, here's the timing tests I did for Arc/Nu using "procedure-rename":
What a ginormous difference. Removing it doubled the speed of Arc/Nu! Given that the sole purpose of "procedure-rename" is to, well, rename functions, I wouldn't expect it to have such a huge runtime performance penalty, but apparently it does...
Now multiple languages are fully supported. The way it works is that, within a file, you can use "w/lang" to temporarily change the language. For instance, suppose you had the following "arc/3.1" program:
(w/lang arc/nu
(var foo 1))
(= bar (+ foo 2))
Within the "w/lang" block, it's using "arc/nu", but outside, it's using "arc/3.1"! Using this, it's easy to import libraries written in "arc/nu":
(w/lang arc/nu
(import foo))
And vice versa, if you're writing a program in "arc/nu", you can use "w/lang" to import things from "arc/3.1":
(w/lang arc/3.1
(import foo))
By the way, because all of this is using boxes at compile-time, there's zero runtime cost. The only downside is that if you use two languages at once, it has to load both of them, which increases the startup time.
I use the server portion of Arc to write web-based applications, and would like to upload files.
Anarki hadn't worked very well or very fast when uploading. I might just bite the bullet and use web.py alongside Arc, and have Python calling Arc after a file has been saved. I might also resort to using Anarki, although breaking backward compatibility in a few ways can become a bit of a concern.
I wish there were a language that offered what I need.
After some major refactoring, I reverted it back to use vanilla Arc 3.1.
Now, Arc/Nu has a rudimentary "multiple language" feature. When using the "arc" executable, you can now specify the "--lang" property.
For instance, when using "arc --lang foo", it will look for a folder called "foo" in the "lang" subdirectory. This folder should contain a file called "main" which should be written in Racket. Using this file, you can implement custom languages using the Arc/Nu compiler.
Currently, only two languages are supported: "arc/nu" and "arc/3.1"
As you can see, "nu.nu" is quite sparse at the moment, but it does load correctly if you use "arc --lang arc/nu". With some more refactoring, I'll be able to make it so that arc/nu and arc/3.1 can use libraries written in the other language.
