(mac foo -> {bar qux corge})

That works just fantastic in Racket, where you can splice arbitrary things into the AST, but that won't work in JS, where things generally have to be given a name.

I could, of course, change it so that it expands to the symbols bar, qux, and corge, and then it would work in JS... but... ew, lack of hygiene. It just wouldn't be the same language anymore.

But then I realized something else... to explain this, I'll need to explain how the Nulan compiler works. Firstly, Nulan doesn't have much of a concept of "symbol": macros expand to values (not symbols), quote is discouraged, etc.

But Nulan takes it a step further. The Nulan compiler will actually remove all symbols from the AST before evaling it in Racket.

How this works is, at compile-time, the compiler has this big dictionary which represents the environment. This environment maps symbols to mutable/immutable boxes.

If the symbol refers to an immutable box, the compiler will unbox it and splice it in directly. If it's a mutable box, it'll put the box into the AST, which will be unboxed at runtime.

This is faaaaaaaast. There's no symbol lookup, not even for globals, thanks to Nulan's hyper-static scope.

Because of this, I'm pretty sure Nulan will end up being significantly faster than Arc. I don't think it'll be faster than Racket though, because Racket's modules are lexical, so I'm sure Racket can do the same sort of optimization. I do think it'll allow Nulan to be just as fast as Racket even though Nulan doesn't use Racket's modules.

Anyways, the point is, symbols are bound to boxes, not to values. And then these boxes are inserted into the AST. That way you can have the same symbol with different meaning:

  (var foo 1)
  (def bar -> foo + 5)

  (var foo 3)
  (bar)

Then it creates a new box and binds it to "foo", but the compiler has already replaced the "foo" inside the function "bar" with a box, so there's no problem, and thus the call to (bar) returns 6, rather than 8.

---

Okay, that works faaaantastic in Racket, but JS doesn't have boxes, and you can't inline values into the AST anyways. So what do I do? Well, there's a little trick I'm using...

I think I'll be able to explain it easiest just by showing what the above compiles into in JS:

  var foo = 1;
  var bar = function () { return foo + 5 };

  var foo2 = 2;
  bar();

Effectively, it's saving all the different "versions" of the foo variable, so that each variable is unique. And so, that's how I simulate Racket's boxes in JS.

This has another nifty side effect: on the JavaScript side, you can choose which version of the variable to call. So this gives the benefits of hyper-static scope not only to Nulan code, but also to JavaScript code that calls Nulan code.

This reminds me a lot of SSA:

http://en.wikipedia.org/wiki/Static_single_assignment_form

---

So, going back to the macro example way at the top. Consider this:

  (let bar 5 (foo))

  (do (var bar 5)
      (bar qux corge))

  var bar2 = 5;
  bar(qux, corge);

Anyways, with that out of the way, I actually think I'm going to go for strategy #2, because I only lose a tiny bit of flexibility, but it should be much much faster.

---

Just use my Function( "x", ... )( x ) pattern. Keep in mind that you should only need to translate to JavaScript once per eval or top-level command, rather than once per macro call. The "AST" {bar qux corge} will still exist as an intermediate representation, since we still need to process 'bar if it's a macro and compile 'qux and 'corge if it isn't.

---

"If the symbol refers to an immutable box, the compiler will unbox it and splice it in directly. If it's a mutable box, it'll put the box into the AST, which will be unboxed at runtime."

I did this in Penknife too, but Penknife only had mutable boxes.

Well, I did it for global scope anyway. You say you're doing this for local scope too?! The boxes for a function's parameters don't even exist at compile time, right?

---

"JS doesn't have boxes"

Sure it does! I usually use { val: _ } when I want a box.

You should probably keep using separate JavaScript variables, but here are your examples using boxes:

  // (var foo 1)
  (function () {
      env.foo = { val: 1 };
  })();
  
  // (def bar -> foo + 5)
  (function () {
      var g1 = env.foo;
      env.bar = { val: function () {
          return nulan.plus_( g1.val, 5 );
      } };
  })();
  
  // (var foo 3)
  (function () {
      env.foo = { val: 3 };
  })();
  
  // (bar)
  (function () {
      return bar();
  })();
  
  // (mac foo -> {bar qux corge})
  (function () {
      var g1 = env.bar;
      var g2 = env.qux;
      var g3 = env.corge;
      env.foo = { val: nulan.macro_( { bar: g1 }, function () {
          return [ g1.val, g2.val, g3.val ];
      } ) };
  })();
  
  // (let bar 5 (foo))
  // ==>
  // (do (var bar 5)
  //     (bar qux corge))
  (function () {
      var g1 = env.foo.val.captures.bar;
      var g2 = env.qux;
      var g3 = env.corge;
      return (function () {
          var bar = 5;
          return (0, g1.val)( g2.val, g3.val );
      })();
  })();

Since I'm not using separate JavaScript variables for separate definitions, the original bar is totally out of scope in (let bar 5 (foo)), and I end up having to smuggle it in as part of foo (env.foo.val.captures.bar). I did this in Penknife, and I called the foo->bar path a "lexid." If I had to do it again, I might consider generating unique variables outside every environment, something like what you're doing.

For an easier solution all around, we can call the 'foo macro at run time, rather than relying on the compiler to expand it beforehand.

  // (let bar 5 (foo))
  // ==>
  // (do (var bar 5)
  //     (bar qux corge))
  nulan.eval_( env, [ "let", "bar", 5, [ "foo" ] ] );

As a bonus, this approach allows side effects during macroexpansion to work properly, rather than being forgotten by the compilation process.

The reason I didn't do this in Penknife is because Penknife macros suppressed parsing, the most expensive step, so this kind of compilation wouldn't have done any good.

---

"Anyways, with that out of the way, I actually think I'm going to go for strategy #2, because I only lose a tiny bit of flexibility, but it should be much much faster."

Just wait until someone comes along and thinks your language is too static. ;)

That approach is also good for debugging. Compiling to so-called idiomatic JS is a sacrifice many JS-targeting languages make right now.

---

"I'd still like to hear rocketnia's advice on bytecode, though, for future reference."

I've only done a few things with bytecode, most of which involve reverse engineering GB/NES/SNES machine code and game-specific binary data. I've taken a look at bytecode generation for the JVM a few times, but I know better than to trouble myself with that. :-p

Bytecode is a term commonly used to describe binary formats for programs that run on virtual machines. It's designed to be computationally cheap to execute (e.g. JIT-compile) on multiple CPU machine code dialects.

Ewwww. Slow. And yes I know you only need to eval once per top-level form, but the problem is that means you need to actually keep the top level forms around and the compiler/runtime needs to always be present.

Keep in mind that I want to use Nulan to write Chrome Extensions. Now imagine a popup that occurs when the user presses a button. This popup is coded in Nulan. Now, every time the user presses the button, it has to re-parse, re-compile, and re-eval all of the Nulan code that is inside the popup.

I consider that unacceptable for my desires, where the code should be about as fast as handwritten JS. So, Nulan compiles things ahead of time to a JS file, to achieve those speeds.

---

"Well, I did it for global scope anyway. You say you're doing this for local scope too?! The boxes for a function's parameters don't even exist at compile time, right?"

Well, in Racket, I just use a "let" for local scope, so no, locals aren't boxes. But I could make locals use boxes too. It would just be a lot harder to implement. At that point I might as well just write an interpreter.

And in fact, the interpreted vau version of Nulan does use boxes for all variables, global and local.

---

"Sure it does! I usually use { val: _ } when I want a box."

Yes, but then it has to do the boxing/unboxing at runtime in addition to the variable lookup. Whereas with Nulan's scheme, you get almost all the same benefits, but with just the variable lookup. The only downside is that the boxes are no longer available at runtime (but you can access the fake "boxes" at compile-time).

---

"Just wait until someone comes along and thinks your language is too static. ;)

That approach is also good for debugging. Compiling to so-called idiomatic JS is a sacrifice many JS-targeting languages make right now."

Yes it is a tradeoff. Yes I know it's more static than Arc or the Racket version of Nulan. But with my scheme, I can actually make things feel fairly fluid. As an example, this works:

  (var + (fn ...))

  (+ 1 2)

So, the only real downside is that you can't do this:

  (def bar -> ...)

  (mac foo ->
    (bar ...))

  (mac foo ->
    {bar ...})

  (&eval (def bar -> ...))

  (mac foo ->
    (bar ...))

  env.foo = { val: nulan.macro_( { bar: g1 }, function () {
      return [ g1.val, g2.val, g3.val ];
  } ) };

  return (function () {
      var bar = 5;
      return (0, g1.val)( g2.val, g3.val );
  })();

That object becomes the "captures" property in env.foo.val.captures.bar later. In Penknife, that's how I let code have run time access to the lexical scopes of compile-time-expanded macros.

---

"Why do you do (0, g1.val)? That's the same as just using g1.val."

  var o = {
    foo: function () {
      return this
    }
  }

  o.foo()       -> Object
  (0, o.foo)()  -> Window

  o.foo()       -> Object

After this encounter, I reasoned that a.b( ... ) must be its own syntax, only superficially related to a.b and b( ... ). My choice to use this syntax is explicit. This intuition worked well for a while, but then this behavior surprised me:

  (o.foo)()     -> Object
  ((o.foo))()   -> Object

Turns out the spec actually specifies this behavior in terms of "References." An expression doesn't just return a value; it returns a Reference that can be either resolved to a value, assigned to, deleted, passed to typeof, or called as a method call. (Maybe there are some other options too.)

In the spec's own words:

"The Reference type is used to explain the behaviour of such operators as delete, typeof, and the assignment operators. For example, the left-hand operand of an assignment is expected to produce a reference. The behaviour of assignment could, instead, be explained entirely in terms of a case analysis on the syntactic form of the left-hand operand of an assignment operator, but for one difficulty: function calls are permitted to return references. This possibility is admitted purely for the sake of host objects. No built-in ECMAScript function defined by this specification returns a reference and there is no provision for a user-defined function to return a reference. (Another reason not to use a syntactic case analysis is that it would be lengthy and awkward, affecting many parts of the specification.)"

An ECMAScript Reference is similar to what I independently called a "fork" in Penknife. Forks were my approach to unifying Arc-style metafns, setforms, and macros into a single syntactic mechanism. A Penknife syntactic abstraction would return a fork value, and that value would have behaviors for parsing a macro call body, for setting, for getting, and for acting as a top-level command. The result of the parsing behavior would be another fork, so this branching process could handle multiple layers of brackets, metafn style. (Now I understand this as a monadic, coinductive approach. Yay big words.)

I've compared JavaScript's method invocation to metafns before: http://arclanguage.org/item?id=12094. I was somewhat supportive of the combination in that post, but I think Arc, JavaScript, and Penknife just use this technology for name punning, rather than resolving any essential conundrums of language design, so I'm not very enthusiastic about the technique in general.