As an aside: maybe it's better to make the primitives really, really primitive?

Basically we define separate %n+ for adding two fixnums, %f+ for adding two flonums, %fn+ for adding a flonum and a fixnum (in that order) and, %nf+ for adding a fixnum and a flonum (in that order).

Then we can write + in lib-ac.scm.arc as:

  (set +
    (fn rest
      ($+ rest)))
  (set $+
    (fn (args)
      (if args
          (if (%cdr args)
              ($sub+ (%car args) (%cdr args))
              (%car args))
          0)))
  (set $sub+
    (fn (accum rest)
      (if rest
          ((fn (val rest)
             (if (%is (%type accum) 'int)
                 (if (%is (%type val) 'int)
                     ($sub+ (%n+ accum val) rest)
                     (if (%is (%type val) 'num)
                         ($sub+ (%nf+ accum val) rest)
                         ; raise exception
                         ()))
                 (if (%is (%type accum) 'num)
                     (if (%is (%type val) 'int)
                         ($sub+ (%fn+ accum val) rest)
                         (if (%is (%type val) 'num)
                             ($sub+ (%f+ accum val) rest)
                             ; raise exception
                             ()))
                     ; raise exception
                     ())))
            (%car rest) (%cdr rest))
          accum)))

Now, macros. I tried to think about it yesterday, and I don't really understand why you need this protected mode. I mean, macros have to be defined (globally) before they are called and treated as such. Since you can already identify all the globals (and since a macro is a global thing annotated by 'mac), it should be easy to

- make a list of all the symbols holding a macro at a given moment in the code

- explore all the sexprs to see if one of those symbols is the first element of any sublist

- if it is to, make the transformation

- do it again until nothing changes anymore

  (do
    (= xe 0)
    (mac somethingstupid ()
      (= xe (+ xe 1))
      `(prn "something stupid" ,xe)))

If I understand well, we have to interpret the code as we compile it (as a macro's body is executed when a function is defined/compiled, not when executed) ? To me, macros were something translating a piece of code into another one, period. But that's not always the case as your example shows. If someone writes

  (mac foo
    (prn "blah"))

  (def bar (n)
    (foo) (* n 2))

The global environment is shadowed by a table; when this table is accessed, if a table entry does not exist, we look it up in the real environment using 'eval (!!) and retain a copy. This allows the macro to use stuff like 'map.

Edit: Also, it might be useful to organize the primitives as "functions", like what I did with cons_fun().

So:

  #define CONS() {obj d = POP(); TOS() = cons_fun(TOS(), d);}

  (%cons foo (%car bar))
  ==>
  PUSH(cons_fun(foo, car_fun(bar)));

  QUOTE_CONSTANT[0] = cons_fun(FIX2OBJ(1),cons_fun(FIX2OBJ(2),NILOBJ))

Called the "Arc stack", maybe? ^^

Currently this is what is done:

  QUOTE_CONSTANTS[0] = cons_fun(FIX2OBJ(1),cons_fun(FIX2OBJ(2),NILOBJ))

  PUSH(FIX2OBJ(1));
  PUSH(FIX2OBJ(2));
  PUSH(NILOBJ);
  CONS();
  CONS();
  QUOTE_CONSTANTS[0] = POP();

  (car (cdr foo))

  PUSH(car_fun(cdr_fun(LOCAL(3)/*foo@42*/)));

  #define car_fun(o) (((pair *)(o))->car)

The implementation consists in two arrays : one for the keys, the other one for values. When you look for an object, it goes through the keys array and performs 'is comparison until it finds it. It is as scalable as alists, I know, but I'll work on the actual implementation later...

There are a few other new things. Most notably, the functional notation for accessing parts of lists/strings/hash tables is now implemented : ('(foo bar baz) 2) returns 'baz.

I thought it would bring horrible performance : since something at a functional position can be not only a closure, but also a string, a table or a pair, you have to do multiple dispatch every time you call a closure. Ugh... Well, obviously, gcc (with -O3) is very clever, or I'm missing something, because the actual slowdown isn't really significative.

> There are a few other new things. Most notably, the functional notation for accessing parts of lists/strings/hash tables is now implemented : ('(foo bar baz) 2) returns 'baz.

Cool! How'd you implement it?

call* is not implemented yet.

As for keys in hash tables, I admit I didn't really bother yet. It is quite broken in canonical Arc anyway (and in Anarki too for that matter) : it doesn't work correctly with lists and does not work correctly with strings too (if you update a string key, you're dead). If the hash itself is its own key, it breaks, but I don't know if this is supposed to be feasible or not.

  %string-ref
  %list-ref
  %table-ref

  (= call* (%table)) ;or whatever the table-creating primitive is
  (%sref-table call* 'string
    (fn (s i) (%string-ref s i)))
  (%sref-table call* 'list
    (fn (s i) (%list-ref s i)))
  (%sref-table call* 'table
    (fn (s i) (%table-ref s i)))
  ; for compatibility with existing Anarki
  (set ref
    (fn (c i)
      (if (%is (%type c) 'string)
          (%string-ref c i)
          (if (%is (%type c) 'list)
              (%list-ref c i)
              (if (%is (%type c) 'table)
                  (%table-ref c i)
                  ; dummy stub for error message, when
                  ; errors are implemented
                  ())))))

  obj fn = SYM2OBJ("fn");
  obj typ;
  goto initjump;
  jump:
  // most common case, so check it first
  if((typ = type(LOCAL(0))) == fn){
    goto realjump;
  } else {
    memmove(&LOCAL(3),&LOCAL(2), (num_args - 2)*sizeof(obj));
    ++num_args;
    LOCAL(2) = LOCAL(0);
    LOCAL(0) = table_lookup(GLOBAL(CALL_TABLE), typ);
    //possibly add a check here to see if it's a function
  }
  realjump:
  pc = LOCAL(0)[0]; //or whichever it should be
  initjump:
  switch(pc){
    ...

  -#define CAR() { if (TOS() != NILOBJ) {pair * p = (pair *) POP(); PUSH((obj)(p->car)); }}
  -#define CDR() { if (TOS() != NILOBJ) {pair * p = (pair *) POP(); PUSH((obj)(p->cdr)); }}
  +#define CAR() { pair * p = (pair *) POP(); PUSH((obj)(p->car)); }
  +#define CDR() { pair * p = (pair *) POP(); PUSH((obj)(p->cdr)); }

To be specific:

  $ ls
  arc2c/ arc-wiki/
  $ cd arc-wiki/
  $ cp -r ../arc2c/* .
  $ ./arc.sh
  Use (quit) to quit, (tl) to return here after an interrupt.
  arc> (load "arc2c.arc")
  nil
  arc> (compile-file "t.arc")
   <lots of stuff>

re: primitives: I think it's better to split the primitives: %+ for numeric addition, %string+ for string addition. Then define in lib-ac.scm.arc:

  (set +
    (fn (a b)
      (if (and (is (type a) 'int) (is (type b) 'int))
          (%+ a b)
          (if (and (is (type a) 'string) (is (type b) 'string))
              (%string+ a b)
              (%err "+: type mismatch")))))

Further, what if we need to pass '+ as a function to another function?

  (def hmm (op)
    (op 1 2 3 4 5))

Personally I think memory should be managed by refcounts, and GC only when the cyclic garbage adds up. However adding refcounts is somewhat harder since every state-mutating 'set, 'sref, 'scar, 'scdr, and 'cons needs to decrement the current obj refcount and increment the new obj refcount.

Which brings to mind the Lisp koan (http://www.lisp.org/humor/ai-koans.html):

One day a student came to Moon and said, "I understand how to make a better garbage collector. We must keep a reference count of the pointers to each cons." Moon patiently told the student the following story-

    "One day a student came to Moon and said, "I understand how to make a better garbage collector...

On the opposite, palying with mzscheme's or Lua's FFI is a real pleasure : you don't have to bother with GC. You even have (sometimes) your malloced object collected for you.

But if we can cetnralize the refcount operations in a single (or a very small number) of places, I'm OK... Their talking about stack_push / stack_pop is rather inspiring...

For information : On a GC-relativey-intensive program (mainly calculating (fib 40), which generates a lot of garbage), and with a heap of 50 million possible references, for a total time of 228000 ms, i got the following GC info :

  total time : 177ms, nb cycles : 17, max time : 42ms, avg time : 10 ms

Really? You've tried it? Because docstrings are supposed to be removed by the unused global removal step.

Edit: I just did. This is a bug.

Is there any hope of getting arc2c able to compile itself anytime soon? This is exactly what you would need for a self-hosting arc environment. -JH

Unfortunately macros require us to embed an Arc interpreter in the compiler itself, and we can't use the builtin 'eval because we don't want the macros under compilation to accidentally bash global functions/variables used by the compiler itself.

- PUSH the closure to be called

- PUSH the list of args

- call the APPLY() function, which just POPs the two elements, PUSHes all the elements of the arg list one after one, then call the closure (maybe after changing the continuation argument of that closure by hand, at runtime) ?

That's a runtime behavior, for sure, and probably a few cases should be hard-coded in the generated C file. E.g., (apply + '(1 2)) should be translated to (+ 1 2), then to 3, if nothing bad (redefinition of '+ or 'apply by the user) happened. But in the general case, you can't know.

> - PUSH the closure to be called

> - PUSH the list of args

> - call the APPLY() function, which just POPs the two elements, PUSHes all the elements of the arg list one after one, then call the closure (maybe after changing the continuation argument of that closure by hand, at runtime) ?

Which continuation argument do you pass?

Suppose it's like this:

  (%car (%apply foo bar))

  (set foo
    (fn (a b)
      (ccc
        (fn (k)
          (a k b)))))

Remember, calling a function consists of the following steps:

1. Push the closure

2. Push the continuation

3. Push the arguments

The problem is step 2: continuation.

Possibly we need to insert the default 'apply during the same step as inserting the default 'ccc ? Then we could define '%apply as accepting the function, the continuation, and a plain list of arguments.

> Anyway, we will eventually have to implement an interpreter in the generated code, to deal with dynamic stuff. Maybe closures should be made available through a hashtable mapping their name(s) to actual code, and not only through a hard-coded array as it is now ?

s/closure/global variable/, maybe?