shader already said the first thing I thought of:

Ideally, the 'results' in the documentation would still be computed dynamically, so you can always see whatever the current output would be, but also have the expected results available for separate regression testing.

All the talk about stdin/stdout/randomness got me realizing that, really, any state is going to be an issue (as state is wont to be). Not that it's often a problem in a mostly-functional language like Arc.

On the one (liberating) hand, Arc's philosophy clearly guides us to not lose sleep over the potential problems of state---just be careful. I see your destructive examples are careful to use ret, for instance, rather than global assignment. On the other hand, we are losing the ability to express certain kinds of tests here. Ideally, we could eval the examples in their own "sandbox" and capture their output, just as if it had been at a REPL.

This is an interesting problem, because it feels so familiar. After some slow, tired thoughts (been a long day for me), it hit me: "Oh! We're doing doctests, except in the other direction."

In Python (https://docs.python.org/2/library/doctest.html):

1. Copy/paste a REPL interaction into a docstring.

2. Looking up the documentation shows the static string.

3. A test function (doctest.testmod()) combs through docstrings, parses them apart, runs them in isolated environments, and checks their results.

Here:

1. You give some literal code as examples (yeah, Lisp!).

2. Looking up the documentation evaluates the example in the current environment.

3. Running the example generates some output that we print out as though it had been pasted from a REPL interaction.

For point 1, using sexps seems more befitting to Arc than parsing out docstrings. Yet we do lose something here: writing literal strings of expected output is cumbersome, as you mentioned. Much better would be to just copy & paste an entire REPL interaction (inputs & outputs) into a multiline string and call it a day.

Point 3 is a small detail: to run the tests, you "have" to look at the docs. Except writing a doctest.testmod() equivalent isn't hard---just evaluate all the examples without printing their docs.

But Point 2 is an interesting one. Arc doesn't have much support for environment-mangling: eval doesn't like lexical variables, lack of modules/namespaces, the difficulties of environment introspection, the weirdnesses that macros introduce, etc. About the only way I see to have a guaranteed clean slate (without requiring us to just "be careful" with the examples) is if we could fork off another Arc interpreter, run the example in that process, and capture the output (including stderr!) for comparison with the docstring. I reckon it would be slow as hell (one interpreter for each block of examples), but is that otherwise feasible?

I like the idea of just having to compare output strings universally (the repl-output comment I made), so that we don't really have to decide which expecteds to selectively evaluate. Instead, just look at the strings: (is "#hash(...)" "#hash(...)"). But there are probably ways that just comparing the output strings could break. Obvious things like random number generation, race conditions, user input. But even tables: are two tables with the same contents guaranteed to serialize the same way? I'm too tired to effectively think through or look up an answer.

Ideally, the 'results' in the documentation would still be computed dynamically, so you can always see whatever the current output would be, but also have the expected results available for separate regression testing.

Here's a couple more issues:

a) Expected results are currently unevaluated, so there's no way to represent tables, and so on.

  arc> (help counts)
  [fn]  (counts seq)
  Returns a table with counts of each unique element in 'seq'.

  Examples:
    arc> (counts '(b a n a n a))
    (obj b 1 a 3 n 2) <-- ERROR

c) We can't show stdout and stderr.

---

In particular, I would like a way to convert an interactive console session (my normal method of testing code) into a set of unit tests. Not sure exactly what the right way to do that would be though.

I'm less concerned with handling stdin, but the obvious example for do is:

  (do prn.1 prn.2 prn.3)

  (examples do
    (do (prn "line 1")
	(prn "line 2")
	(prn "line 3"))
    _)

  arc> (help do)
  [mac] (do . args)
  Evaluates each expression in sequence and returns the result of the
  last expression.

  Examples:
    arc> (do (prn "line 1")
	     (prn "line 2")
	     (prn "line 3"))

In fact, if you're just testing REPL interactions...

  (mac repl-output (expr)
    `(tostring
       (write (eval ',expr))
       (prn)))

  arc> (repl-output (prn 'hi))
  "hi\nhi\n"
  arc> (repl-output (prn "hi"))
  "hi\n\"hi\"\n"
  arc> (repl-output 'hi)
  "hi\n"
  arc> (repl-output (do prn.1 prn.2 prn.3))
  "1\n2\n3\n3\n" 

I actually had an arc version for a brief period[1], but it didn't seem like there was as much need for it here. But perhaps the email notifications would be useful. I'll see if I can bring it back.

This gets you two things.

1) You can now provide grouped details pages, like the 'list operations' and 'macros' pages from the arc3.1 docs.

2) You can provide connective narrative associated with a particular tag, in a separate document.

a) All examples are now checked when you load tests.arc.

b) You can pass in an unprintable expression using valueof. For example:

  (examples sref
    (ret x '(1 2 3)
      (sref x 4 1))
    (1 4 3)
    (ret x "abc"
      (sref x #\d 0))
    "dbc"
    (ret x (obj a 1 b 2)
      (sref x 3 'd))
    (valueof (obj a 1 b 2 d 3)))

  arc> (help sref)
  [fn]  (sref tem v k)
  Sets position 'indices' in 'aggregate' (which might be a list, string, hash
  table, or other user-defined type) to 'value'.

  Examples:
    arc> (ret x '(1 2 3)
	   (sref x 4 1))
    (1 4 3)

    arc> (ret x "abc"
	   (sref x #\d 0))
    "dbc"

    arc> (ret x (obj a 1 b 2)
	   (sref x 3 'd))
    #hash((b . 2) (a . 1) (d . 3))

i) If it's _, checking is skipped and help doesn't print the result.

ii) If it's of the form (valueof x) then we evaluate x when printing and comparing.

iii) Otherwise we compare against the raw value.

I could eliminate b) by always evaluating x. This would make tests look like:

  (examples list
    (list 1 2 3)
    '(1 2 3)                <-- note the quote
    (list "a" '(1 2) 3)
    '("a" (1 2) 3))         <-- note the quote

  (examples list
    (list 1 2 3)
    (1 2 3)
    (list "a" '(1 2) 3)
    ("a" (1 2) 3))

Merging the concept can be helpful, but requires more of the people making the examples in the first place. Also, just because a code snippet makes a good example does not mean it makes a good test, and vice versa.

I would prefer a solution where 'examples didn't include any predefined results at all, and they were all just evaluated during help. If desired, someone working with the unit test suite could write code that leveraged the examples, but it wouldn't be necessary. That way we could use good illustrative examples that may not make good tests, and good thorough tests that may not make good examples.

I certainly agree that the vocabulary of matchers is incomplete. And if we can't find a small basis set of them, this whole second alternative to tests start to seem inelegant.

Perhaps we should just tag some tests to be shown in help and call it a day. Though I like seeing the tests right next to each function. That seems useful even if we rip out the online help altogether. In fact, I started out writing examples when the macro was a noop. We could just inline all the tests, but then it would be overwhelming to have 20 tests for each function.. Ok, I'll stop rambling now. Summary of use cases:

a) Getting feedback that something broke when we make a change. (Unit tests currently do this.)

b) Seeing some examples as we read the code.

c) Online help at the repl.

I hadn't thought of the use case for wanting examples while perusing the code itself, but I must admit that I find that somewhat uncommon. I don't often know which file a function is defined in, and rarely need it given the help support we have in the repl. If I do want to look at the code, the 'src utility shows the source of a function. If I want to edit it, I often use the help utilities to find the file anyway. So having the results only available via the repl wouldn't bother me any.

Actually, I don't know if the examples should be automatically displayed with 'help, or queried separately.