Arc Forumnew | comments | leaders | submit | almkglor's commentslogin
4 points by almkglor 6438 days ago | link | parent | on: help deploying arc to a web server

Do you have apache up?

The port for HTTP should really be port 80. However, I think most of the people here who have deployed the Arc server generally put Apache (serving on port 80) between the real world and the Arc server (serving on port 8080). This is generally done because nobody trusts the Arc server to be hackproof yet, especially not the Anarki version, which has been touched by quite a few people and may not be perfectly secure (but at least has a few more abilities than the Arc2 version).

If you're willing to risk it, then try (nsv 80), which forces the Arc server to listen on 80; you might need to run as root though.

-----

1 point by jsomers 6438 days ago | link

Thanks. This is helpful, but not quite there yet.

I don't have root access, so naturally (nsv 80) gives "listen on 80 failed (Permission denied..." error.

Apache is up, but I'm not sure what I can do with it. Is there a simple redirect I need to run?

To be precise, I've loaded arc2 into a directory [mydomain]/news corresponding to news.[mydomain].com, with the news.arc specifying "news.[mydomain].com" as the site url; I launch the Arc REPL from there ("mzscheme -m -f as.scm") and do "(nsv)"... to no avail.

Thanks for the help so far, really appreciate it.

-----

2 points by almkglor 6438 days ago | link

Try accessing news.[mydomain].com:8080. If you have some sort of terminal access on your remote computer, try lynx http://news.mydomain.com:8080/

As for the Apache <-> Arc talking thing, you'll have to wait for someone else to answer that question; I honestly don't know, because I haven't deployed one yet.

-----

1 point by jsomers 6438 days ago | link

This was my first thought too, although unfortunately I get a network timeout error.

-----

3 points by almkglor 6438 days ago | link | parent | on: arc2c: a proposed threading model

The problem with transforming each Arc function to a C function is the stack space consumed at each call. Sure, gcc does tail call optimization, but not all compilers are gcc.

Chicken fixes this by keeping track of stack and GC'ing the stack when it's full; I'm not sure about Bigloo but I hear it's got a pretty good Scheme function == C function equivalency, so it might be useful to see how they fixed the tail call optimization problem there.

That said, the current execute() function accepts a pc argument, which specifies which Arc function it begins with. It may be possible to pass the pc to go to together with a new stack, but I don't know pthreads.

-----

2 points by stefano 6438 days ago | link

I think you could trust every decent C compiler to do tail recursion optimization, but if you want full compatibility then the mapping Arc fun -> C fun doesn't work. pthread requires you to pass a pointer to a C function i.e. an adress where to jump. We could wrap every thread within a C function and leave all the other functions as they are currently implemented. The C function would just call execute() with the right parameters. If execute() doesn't use global vars, there will not be race condition problems.

-----

2 points by almkglor 6438 days ago | link

Well, I've started to import bits of execute from globals to locals. However I do have access to a few bits of global variables, specifically the quoted-constants array (those created by 'foo and '(a b c d), etc.); this table is initialized at startup (before the first call to execute). I would suppose this read-only array would be okay to access?

As for wrapping them in C functions: the problem is that the most basic Arc threading function isn't 'thread, it's 'new-thread, which accepts a function as input. 'thread is defined as:

  (mac thread body
    `(new-thread (fn () ,@body)))
In theory, new-thread could be called with any function:

  (new-thread
   (if
    (something)
      (fn () (ha-ha-ha))
    (something-else)
      (fn () (he-he-he))
      (fn () (je-je-je))))
Sure, it won't happen most of the time, since most people will sensibly use the simpler 'thread, but exploratory, exploratory...

It would be possible to implement if pthreads or whatnot can pass even just a single pointer to the newly-threaded function, but if it can't (why not?) then our alternative is to create a bunch of C functions for each Arc function, which just calls the execute() function with the correct number.

---------

Edit: okay, I did a little eensy-weensy bit of research on pthreads, and it seems that pthreads can pass a pointer to the called C function.

So I suppose we can modify execute's signature:

  obj execute(long pc, obj stack[], obj *sp);
  int main(void){
    /*init...*/
    obj *stack = GC_MALLOC(sizeof(obj)*MAX_STACK);
    obj *sp = stack;
    execute(0, stack, sp);
  }
Then the NEWTHREAD() primitive would be:

  typedef struct{
    long pc;
    obj *stack;
    obj *sp;
  } newthreadpasser;
  #define T_THREAD 9999 //some random number
  typedef struct{
    long type; /*T_THREAD*/
    pthread_t val;
  } thread;
  #define NEWTHREAD() \
   {obj *newstack = GC_MALLOC(sizeof(obj) * MAX_STACK);\
    obj *newsp = &newstack[2]; \
    newstack[0] = POP(); \
    newstack[1] = THREAD_EXIT_CONTINUATION;\
    newthreadpasser * np = GC_MALLOC(sizeof(newthreadpasser));\
    np->pc = ((obj*)newstack[0])[0];\
    np->stack = newstack;\
    np->sp = newsp;\
    thread * threadobj = GC_MALLOC(sizeof(thread));\
    threadobj->type = T_THREAD;\
    pthread_create(&(threadobj->val), NULL, threadcall, (void*) np)\
    PUSH((obj)threadobj);}
Finally, threadcall:

  void *threadcall(void * args){
    newthreadpasser * np = (newthreadpasser *) args;
    execute(np->pc, np->stack, np->sp);
  }
Hmm.

-----

2 points by stefano 6438 days ago | link

This could work. As for global variables, if they are read-only, then there won't be any problem. What happens if you load in parallel two different modules (with their constants)? Maybe loading a file should be made atomic, or at least a part of it, such as constant values initialization.

-----

1 point by almkglor 6438 days ago | link

Well, a "load" at top level simply inserts the code for the loaded file directly into the source for now. Any other load statement is not supported.

-----

3 points by almkglor 6438 days ago | link | parent | on: arc2c: a proposed threading model

Yes, true. It would also be a pretty simple solution IMO, although potentially fraught with some dangers, especially when checking dead threads.

In any case I'm currently thinking of adding a library, where if a global variable is not read (only assigned to), it is simply removed.

Basically: extract the set of global variables that are read, and the set that are written to. If both sets are not equal, eliminate all set operations on members of the written set that are not members of the read set (replace them with their (car ast!subx)). Then eliminate any hanging constants: naked fn definitions and other constants that are in sequences not in the tail position. Repeat until set of global variables that are written is a full subset of the set of global variables that are read, and if they're not equal, raise an unbound variable error (i.e. a global is read but never assigned to, which as you mentioned would segfault. Not perfect of course, because reading the global before it is assigned to will still crash, but better than nothing).

-----

4 points by almkglor 6438 days ago | link | parent | on: arc in java

> a PHP implementation ...

Oh.... my..... deux ex machina....

-----


Hmm, interesting. This could actually be of use in the arc2c compiler ^^

-----

4 points by almkglor 6439 days ago | link | parent | on: Asynchronus I/O in arc

This appears to be correct. However, what asynchronous I/O do you need?

It may be possible to write some sort of semaphore-like object which would synchronize across threads, alternatively I think you can probably query the state of threads.

Also, how do you want async I/O? Do you want a callback style where a callback function is called once the I/O completes or fails? Or an async I/O which aborts if it doesn't complete within the necessary time?

Either of the above solutions is implementable using threads.

-----

1 point by zhtw 6438 days ago | link

Actually I can live without asynchronous I/O (smth like posix aio_* functions). Just nonblocking read and select/poll would be enough. I have some experience with threads and believe that they just are not worth the throuble in most cases. I find the twisted python's approach (with reactor and deferreds) much more elegant. So I wanted to implement smth similar in arc but it seems I can't at least in the obvious way.

-----

2 points by almkglor 6438 days ago | link

Well, threads are really pretty easy in Arc.

For example I think this ought to work:

  (def nonblocking-read (port)
    (with (done nil
           rcvd nil
           rv (table))
      (thread (= rcvd (read port))
              (= done t))
      (= rv!isdone (fn () done))
      (= rv!received (fn () rcvd))))

  (= foo (nonblocking-read port))
  (while (~foo!isdone)
    (do-what-you-will))
  (now-process (foo!received))
Not sure though; it really depends on how well mzscheme handles blocking I/O.

-----

1 point by zhtw 6438 days ago | link

Yes, thanks, it must work. But it means that you need to have exactly N suspended threads when you're waiting for data from N sockets. Thread pool helps a lot but in worst case you will still need exactly N threads.

-----

2 points by almkglor 6437 days ago | link

Thread management is done automatically for you by the underlying mzscheme. A bit more of research seems to suggest that mzscheme can actually handle blocking I/O even if it uses green threads, and it can also have OS threads if compiled with support for it.

From what little I could grok from http://download.plt-scheme.org/doc/insidemz/insidemz-Z-H-8.h... it seems that there's not much overhead per thread anyway.

What are your concerns?

-----

3 points by soegaard 6437 days ago | link

MzScheme does not use OS threads. A long time ago it did, but it is pretty hard to make it work on Windows, OS X and Linux at the same time. Also context switching becomes cheaper without OS threads.

If you have found anything on OS threads in "Inside MzScheme" it must be in the section on how to embed MzScheme into a C program (with its own thread).

-----

1 point by zhtw 6437 days ago | link

Oh, is that rather usual that scheme implementation uses green threads? If so, maybe I really shouldn't worry about it. I was concerned only because I thought it was expensive to have one system thread per connection.

Thanks for the link.

-----

1 point by soegaard 6437 days ago | link

Most Scheme implementations doesn't use OS threads due to the cost of context switches. It is standard practice to use one thread for each connection.

See for example this very well written introduction to Systems Programming with PLT Scheme:

http://pre.plt-scheme.org/docs/html/more/index.html

-----

1 point by almkglor 6437 days ago | link

Maybe, I'm not sure. I couldn't grok the doc much. AFAIK mzscheme uses green threads only if it isn't compiled with "better" thread support.

As for cost per thread - no, I don't worry about it, it seems that mzscheme threads are reasonably light.

-----


Variadic functions now work:

  (set list
    (fn rest
      rest))

  (prn (list 9 8 7 6 5 4 3 2 1))

-----

2 points by sacado 6439 days ago | link

wow, congratulations, you are much more productive than I am ! I won't work a lot on the compiler this week as I am far from my own computer, so I guess many things will have changed next time I'll explore the code :)

-----

2 points by almkglor 6439 days ago | link

Thanks ^^. I'm not that productive, really - I haven't figured out yet how to properly handle environments like so:

  (let private-variable nil
    (set set-private
      (fn (x) (set private-variable x)))
    (set get-private
      (fn () private-variable)))
I've been circling this problem for some time now, getting nowhere quick.

The problem is that the current style of implementing closures - where local and closure variables are copied to the closure structure - strikes me as a premature optimization. A useful one - if closures are not nested, then all variable references are O(1). However for shared mutable variables we need to use a different strategy. I'm trying to figure out some way of retaining the current closure style for cases where variables are not mutated, versus for cases that closure variables are.

BTW how come no one else seems to want to work on the compiler? It seems that aside from you I'm the only commiter?

-----

1 point by absz 6439 days ago | link

Well, I'm watching with fascination, but it's over my head. I'm working through Essentials of Programming Languages (Friedman, Wand, and Haynes), though, so maybe soon ;)

-----

2 points by almkglor 6440 days ago | link | parent | on: hosting for arc web app

From a very cursory review of the code: 'newscache looks like it's related to releasing data.

-----

1 point by chriszf 6440 days ago | link

I'm not convinced of that. It appears to be a macro for building a cache of various things with a timeout. It doesn't appear to touch the stories* table.

On an unrelated note, can anyone explain to me the memory model/threading mechanism interaction? It seemed at first that it was a single-threaded server, but after looking at srv.arc, it's clear that multiple threads are being spawned for the server process, yet stories, profs, and votes* aren't don't seem to be rebuilt from permanent storage, which implies the tables can be shared between processes. Which, I guess, goes back to my original question. So this isn't unrelated at all.

-----

1 point by almkglor 6440 days ago | link

Hmm. After another cursory glance, yes, this appears about correct.

As for memory model: global variables are always shared between threads. The underlying implementation automatically locks tables when mutating them AFAIK.

As for suck up memory: I suppose the thinking is that a single post may very well take less than half a kilobyte, and modern computers have gigabytes of memory, so...

-----

1 point by chriszf 6439 days ago | link

That was my thought too. I suppose I may have been overestimating the traffic to HN, or the size of the server or both. I guess I wanted a definitive answer about having to restart the server periodically. That seems to be the case, but it doesn't seem to be too bad.

This seems to be a pretty good model for a moderate-traffic app with data that gets modified infrequently. I don't think I'm quite ready to arc just for that, but I was wondering if this is a sensible approach in any other language, say, python? Any thoughts?

-----

1 point by almkglor 6439 days ago | link

Well, Arc is still starting. In more-established languages, such approaches are unnecessary: there are libraries which will handle caching of data etc. properly, releasing unused memory for garbage collection once they are no longer accessed for some time, and rebuilding objects from permanent store if they have been disposed. In Arc it's still not yet implemented, so that approach works fine.

That said you might be interested in the so-called "Anarki" repository, which contains some of the elements I and others have built so that the server works a little better. For example: being able to serve files in subdirectories of your Arc installation, instead of the Arc installation; table-like data structures for caching data, or for persistent disk-based data; a slightly more extensible language, with some of the more common methods of extension already prepackaged in macros; etc.

-----

1 point by chriszf 6439 days ago | link

Can you give an example of such libraries, off the top of your head?

I am basically interested in small multi-user applications that don't sit on top of relational databases. There really isn't too much information out there on the matter. Everyone seem to want to use a database, even for the simplest things. I suppose that given the pedigree of arc, this flat file storage business seems sensible enough.

Anyway, thanks for the pointer to anarki. I'll take a look at it.

-----

2 points by almkglor 6439 days ago | link

LOL, no. ^^ I don't program web apps much.

For that matter, most languages prefer db's because of the fact that file storage operations don't have, umm, structure.

In fact the canonical Arc web app, news.arc, has a list structure to store in the "flat" file. Thus for simple apps where entities only have a few not very complex fields, textual representations of lists seem to be enough.

In other languages however their "array" syntax (which is approximately what lists are in Arc) is usually not readable by a built-in function a la lisp 'read. Also, their array syntax is usually not the center of attention, unlike in Lisp where the code syntax is itself the "array" syntax.

-----


Closures:

It seems that closures aren't nested (i.e. do not refer to other frames). In short, when we have a closure within a closure:

  (fn (x)
    (fn (y)
      (fn (z)
        (prn x)
        (prn y)
        (prn z))))
This compiles to (pseudo-Arc):

  (fn (self x)
    (make-closure
      (fn (self y)
        (make-closure
          (fn (self z)
            (prn self.0)
            (prn self.1)
            (prn z))
          (list self.0 y)))
      (list x)))
(here, self means the closure of the function, not 'afn self).

Because the frames aren't shared, this means the current closure-conversion model can't be easily(?) modified made to work on the following code:

  (fn (x)
    (list
      (fn () x)
      (fn (v) (set x v))))
I'm not sure how to resolve this unfortunately. And again I'm not 100% sure of my analysis.

-----

4 points by stefano 6439 days ago | link

You could try transforming

(fn (x) (list (fn () x) (fn (v) (set x v))))

to

  (fn (x)
    (let x (cons x nil)
      (list
        (fn () (car x))
        (fn (v) (set-car x v)))))
This way the adress of the cons cell x isn't shared, but the cell itself is shared. The compiler would do this transformation automatically.

-----

3 points by almkglor 6439 days ago | link

Done and on the git.

Some notes:

1) I use a new structure, the sharedvar, which doesn't correspond to any Arc structure. See the other post which had me blinking stupidly at stefano's transformation before I realized how cool it was: http://arclanguage.com/item?id=5784

2) This new structure is untyped (i.e doesn't have type tags, unlike the pair and symbol structures). I intend to make shared variables "seamless" to the upper Arc though, so this should be fine...

3) My original name for this structure was closure-settable, which I shortened to sharedvar instead.

PS f34r my new gravatar

-----

1 point by stefano 6438 days ago | link

Wow. You implemented it very quickly.

Making the new structure untyped shouldn't create problems, but it could make development difficult: if you have a bug in the transformer or in something related to it, you'll probably get a Seg. fault error instead of a clean "Not a sharedvar" error. The check could then be removed when the code is released.

-----

1 point by almkglor 6438 days ago | link

Hmm. So far the transformation seems correct anyway; and really, the transform is quite simple and appears mathematically correct. Also, because of the way the compile-file driver is structured, it would be possible to remove/replace each individual transform.

Which reminds me, we need to have a proper error continuation too.

-----

2 points by almkglor 6439 days ago | link

Huh.

blinks stupidly

Cool.

Why didn't I....

Cool.

Blink

Cool.

Edit: Okay, I managed to snap out of shock.

The base idea is pretty good, although the 'cons cell (which is composed of a type id, a car pointer, and a cdr pointer) can be replaced with a smaller "closure-settable" cell, which would be an untyped object composed of a single pointer:

  (fn (x)
    ; % means it's a primitive
    (let x (%closure-settable x)
      (list
        (fn () (%closure-settable-read x))
        (fn (v) (%closure-settable-write x v)))))
Because the closure variable abstraction is not accessible from the high-level Arc language, the closure-settable doesn't need to be typed tagged

-----

1 point by almkglor 6441 days ago | link | parent | on: Anyone here going to Startup School?

Anarkist?

oops ^^

-----

More