From 002d042e670745cd7745804d6c29f82934e83e41 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 29 Apr 2021 03:43:07 +0300 Subject: [PATCH 001/832] no more remaining failures --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index f3b24930..5cd1c514 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -89,8 +89,6 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: Fix remaining failures and errors detected by test suite. - # TODO: `make_isxpred` is now obsolete because `mcpyrate` does not rename hygienic captures of run-time values. Make it explicit at the use sites what they want, and remove `make_isxpred`. (E.g. `curry` wants to match both `curryf` and `currycall`, exactly. Some use sites want to match only a single thing.) # TODO: locref could be an ASTMarker anywhere that needs a source location reference; extract `.body` if so. From eadd2b05a6606b7f3847fa9434797a2e053e8584 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 29 Apr 2021 03:52:55 +0300 Subject: [PATCH 002/832] add "new version soon" announcement --- README.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7d6b2c34..4e6f9926 100644 --- a/README.md +++ b/README.md @@ -9,16 +9,32 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f *Some hypertext features of this README, such as local links to detailed documentation, and expandable example highlights, are not supported when viewed on PyPI; [view on GitHub](https://github.com/Technologicat/unpythonic) to have those work properly.* +### New version soon! + +**As of April 2021, `unpythonic` 0.15 is Coming Soon™.** + +As of [7bb1198](https://github.com/Technologicat/unpythonic/commit/7bb1198605087f1dd7ca292e33afd53e5aa9721d), the initial porting effort of `unpythonic` to Python 3.8 and the new [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander is complete. In fact, if you want to play around with 0.15-pre, the code is already in `master`. + +The codebase already fully works on 3.8 and `mcpyrate`, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. + +Beside internal changes related to the macro system, the major goal is full compatibility with Python 3.9, including the `unpythonic.typecheck` module, which is currently the last one failing tests on 3.9. + +For details, see [the 0.15 milestone](https://github.com/Technologicat/unpythonic/milestone/1). + +I'm also considering renaming 0.15 to 1.0, since the codebase is mostly stable at this point, and we have already adhered to [semantic versioning](https://semver.org/) since 2019, anyway (albeit with a leading zero). + + ### Dependencies None required. - [mcpyrate](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, and an interactive macro REPL. -The officially supported language versions are **CPython 3.8** and **PyPy3 3.7**. +The officially supported language versions are **CPython 3.8** and **PyPy3 3.7**. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 7.3.4 (language version 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. + ### Documentation [Pure-Python feature set](doc/features.md) From 4a6000e4cbb46ce6edafaf33b81363849143e300 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 29 Apr 2021 04:02:08 +0300 Subject: [PATCH 003/832] mention equip_with_traceback requires Python 3.7+ --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 45c9d079..c864e6d6 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3413,7 +3413,7 @@ Functions can also be specified for the `else` and `finally` behavior; see the d **Added in v0.14.3**. -Equip a manually created exception instance with a traceback. This is useful mainly in special cases, where `raise` cannot be used for some reason. (The `signal` function in the conditions-and-restarts system uses this.) +In Python 3.7 and later, equip a manually created exception instance with a traceback. This is useful mainly in special cases, where `raise` cannot be used for some reason. (The `signal` function in the conditions-and-restarts system uses this.) ```python e = SomeException(...) From 0e6eca501bc7dd465a87e0eab0c033b27580c0c7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 12:41:53 +0300 Subject: [PATCH 004/832] readings: add some material on holy traits --- doc/readings.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/readings.md b/doc/readings.md index 76a146ba..d41a1866 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -141,6 +141,13 @@ The common denominator is programming. Some relate to language design, some to c - [Walid Taha 2003: A Gentle Introduction to Multi-stage Programming](https://www.researchgate.net/publication/221024597_A_Gentle_Introduction_to_Multi-stage_Programming) +- *Holy traits*: + - [Tom Kwong 2020: Holy Traits Pattern](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) (book excerpt) + - [Lyndon White 2019: Emergent features of Julialang: Part II - Traits](https://www.juliabloggers.com/the-emergent-features-of-julialang-part-ii-traits/) + - [Harrison Grodin 2019: Multiple inheritance, sans inheritance](https://github.com/HarrisonGrodin/radical-julia/tree/master/traits) + - [Types vs. traits for dispatch](https://discourse.julialang.org/t/types-vs-traits-for-dispatch/46296) (discussion) + - We have a demonstration in [unpythonic.tests.test_dispatch](../unpythonic/tests/test_dispatch.py). + ## Python-related FP resources From 934b54b6c7a78958e4b64c5830eff1bf1f719885 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 15:01:31 +0300 Subject: [PATCH 005/832] clarification: devs working in Python, not the Python language devs --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0517d9c5..0952bdba 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -67,7 +67,7 @@ - *Having no docstring is better than having a placeholder docstring.* - If a function is not documented, make that fact explicit, to help [static analyzers](https://pypi.org/project/pyflakes/) flag it as needing documentation. - To help discoverability, the full documentation `doc/features.md` (or `doc/macros.md`, as appropriate) should contain at least a mention of each public feature. Examples are nice, too. - - Features that have non-obvious uses (e.g. `@call`), as well as those that cannot be assumed to be familiar to Python developers (e.g. Common Lisp style *conditions and restarts*) should get a more detailed explanation. + - Features that have non-obvious uses (e.g. `@call`), as well as those that cannot be assumed to be familiar to developers mostly working in Python (e.g. Common Lisp style *conditions and restarts*) should get a more detailed explanation. ## Technical overview From dd52cd45118bad4d9faa839140eefd717b642f4c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 15:06:33 +0300 Subject: [PATCH 006/832] `mcpyrate` uses only one top-level definition for the macro entry point --- CONTRIBUTING.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0952bdba..ef758319 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -207,7 +207,6 @@ As of the first half of 2021, the main target platforms are **CPython 3.8** and - The docstring must be placed on the macro entry point, so that the REPL will find it. This forces all macro docstrings into that one module. (That's less magic than injecting them dynamically when `unpythonic` boots up.) - A macro entry point can be just a thin wrapper around the relevant [*syntax transformer*](http://www.greghendershott.com/fear-of-macros/): a regular function, which takes and returns an AST. - You can have an expr, block and decorator macro with the same name, in the same module, by making the macro interface into a dispatcher. See the `syntax` kw in `mcpyrate`. - - If you do this, the docstring should be placed in whichever of those is defined last, because that one will be the definition left standing at run time (hence used for docstring lookup by the REPL). - Syntax transformers can and should be sensibly organized into modules, just like any other regular (non-macro) code. - But they don't need docstrings, since the macro entry point already has the docstring. - If your syntax transformer (or another one it internally uses) needs `mcpyrate` `**kw` arguments: From 3207d5f9bb7c7fa2fc5755b74cfb84051a1f9db3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 15:09:44 +0300 Subject: [PATCH 007/832] more examples of thin macro layer --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ef758319..c406b47e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -200,7 +200,7 @@ As of the first half of 2021, the main target platforms are **CPython 3.8** and - **Macros.** - *Macros are the nuclear option of software engineering.* - Only make a macro when a regular function can't do what is needed. - - Sometimes a regular code core with a thin macro layer on top, to improve the user experience, is the appropriate solution for [minimizing magic](https://macropy3.readthedocs.io/en/latest/discussion.html#minimize-macro-magic). See `do`, `let` for examples. + - Sometimes a regular code core with a thin macro layer on top, to improve the user experience, is the appropriate solution for [minimizing magic](https://macropy3.readthedocs.io/en/latest/discussion.html#minimize-macro-magic). See `do`, `let`, `autocurry`, `forall` for examples. - `unpythonic/syntax/__init__.py` is very long (> 2000 lines), because: - For technical reasons, as of MacroPy 1.1.0b2, it's not possible to re-export macros defined in another module. (As of `unpythonic` 0.15, this is no longer relevant, since we use `mcpyrate`, which **can** re-export macros. So `unpythonic.syntax` may be revised in a future version of `unpythonic`.) - Therefore, all macro entry points must reside in `unpythonic/syntax/__init__.py`, so that user code can `from unpythonic.syntax import macros, something`, without caring about how the `unpythonic.syntax` package is internally organized. From 32bb0fcdb4889f2060a5a3b63a35ef3e8db6a4e2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 15:39:32 +0300 Subject: [PATCH 008/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 5cd1c514..7e3d7f48 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -93,6 +93,8 @@ # TODO: locref could be an ASTMarker anywhere that needs a source location reference; extract `.body` if so. +# TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) + # TODO: Brackets: use "with test[...]" instead of "with test(...)" in the test modules # TODO: Remove any unused `expander` kwargs from the macro interface From 14301263f974bcda8806743c79fd909e8ee2eee1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 15:44:34 +0300 Subject: [PATCH 009/832] porting: locref no longer needed --- unpythonic/syntax/__init__.py | 2 -- unpythonic/syntax/autoref.py | 3 +-- unpythonic/syntax/lazify.py | 3 +-- unpythonic/syntax/tailtools.py | 11 +++-------- unpythonic/syntax/tests/test_util.py | 2 -- unpythonic/syntax/util.py | 14 +++++--------- 6 files changed, 10 insertions(+), 25 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7e3d7f48..0c486a94 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -91,8 +91,6 @@ # TODO: `make_isxpred` is now obsolete because `mcpyrate` does not rename hygienic captures of run-time values. Make it explicit at the use sites what they want, and remove `make_isxpred`. (E.g. `curry` wants to match both `curryf` and `currycall`, exactly. Some use sites want to match only a single thing.) -# TODO: locref could be an ASTMarker anywhere that needs a source location reference; extract `.body` if so. - # TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) # TODO: Brackets: use "with test[...]" instead of "with test(...)" in the test modules diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 33e55e82..dc43f910 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -219,5 +219,4 @@ def transform(self, tree): newbody.append(AutorefTransformer(referents=always_skip + [o]).visit(stmt)) return wrapwith(item=q[h[AutorefMarker](u[o])], - body=newbody, - locref=block_body[0]) + body=newbody) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 50cf9be1..ba1f801d 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -419,7 +419,6 @@ def transform_starred(tree, dstarred=False): # strict trampoline does not have the maybe_force_args (that usually forces the args # when lazy code calls into strict code). return wrapwith(item=q[h[dyn.let](_build_lazy_trampoline=True)], - body=newbody, - locref=body[0]) + body=newbody) # ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 4c1286de..6974ea77 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -397,13 +397,10 @@ def prepare_call(tree): # TODO: needs to be modified, too. # FDef = type(owner) if owner else FunctionDef # use same type (regular/async) as parent function - locref = callcc # bad but no better source location reference node available non = q[None] - non = copy_location(non, locref) maybe_capture = IfExp(test=q[n["cc"] is not h[identity]], body=q[n["cc"]], - orelse=non, - lineno=locref.lineno, col_offset=locref.col_offset) + orelse=non) contarguments = arguments(args=[arg(arg=x) for x in targets], kwonlyargs=[arg(arg="cc"), arg(arg="_pcc")], vararg=(arg(arg=starget) if starget else None), @@ -416,8 +413,7 @@ def prepare_call(tree): args=contarguments, body=contbody, decorator_list=[], # patched later by transform_def - returns=None, # return annotation not used here - lineno=locref.lineno, col_offset=locref.col_offset) + returns=None) # return annotation not used here # in the output stmts, define the continuation function... newstmts = [funcdef] @@ -525,8 +521,7 @@ def transform(self, tree): # Leave a marker so "with tco", if applied, can ignore the expanded "with continuations" block # (needed to support continuations in the Lispython dialect, since it applies tco globally.) return wrapwith(item=q[h[ContinuationsMarker]], - body=new_block_body, - locref=block_body[0]) + body=new_block_body) # ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tests/test_util.py b/unpythonic/syntax/tests/test_util.py index 48277c51..876907d4 100644 --- a/unpythonic/syntax/tests/test_util.py +++ b/unpythonic/syntax/tests/test_util.py @@ -266,8 +266,6 @@ def ishello(tree): test[type(wrapped) is list] thewith = wrapped[0] test[type(thewith) is With] - test[thewith.lineno == 9001] - test[thewith.col_offset == 9] test[type(thewith.items[0]) is withitem] ctxmanager = thewith.items[0].context_expr test[type(ctxmanager) is Name] diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 4abe2ed1..9a6167eb 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -391,24 +391,20 @@ def transform(self, tree): return tree return StatementTransformer().visit(body) -def wrapwith(item, body, locref=None): +def wrapwith(item, body): """Wrap ``body`` with a single-item ``with`` block, using ``item``. ``item`` must be an expr, used as ``context_expr`` of the ``withitem`` node. ``body`` must be a ``list`` of AST nodes. - ``locref`` is an optional AST node to copy source location info from. - If not supplied, ``body[0]`` is used. - Syntax transformer. Returns the wrapped body. + + This function is intended to be called from macro implementations. We leave out + the source location information, so that the macro expander can auto-fill it. """ - if isinstance(locref, ASTMarker): # unwrap contents of Done() et al. - locref = locref.body - locref = locref or body[0] wrapped = With(items=[withitem(context_expr=item, optional_vars=None)], - body=body, - lineno=locref.lineno, col_offset=locref.col_offset) + body=body) return [wrapped] def isexpandedmacromarker(typename, tree): From c2a8008b9d6eaa42a5374864ece46bd1850e2e5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 16:04:26 +0300 Subject: [PATCH 010/832] porting: get rid of now-unnecessary AST line number population code --- unpythonic/syntax/dbg.py | 19 ++++++++++--------- unpythonic/syntax/lazify.py | 6 ++---- unpythonic/syntax/letdo.py | 7 +++---- unpythonic/syntax/letsyntax.py | 12 +++++++----- unpythonic/syntax/tailtools.py | 13 ++++--------- 5 files changed, 26 insertions(+), 31 deletions(-) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index e93a1e37..8f9416e2 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -6,9 +6,9 @@ The printing can be customized; see ``dbgprint_block`` and ``dbgprint_expr``. """ -from ast import Call, Name, Tuple, keyword +from ast import Call, Name, keyword -from mcpyrate.quotes import macros, q, u, a, h # noqa: F401 +from mcpyrate.quotes import macros, q, u, a, t, h # noqa: F401 from mcpyrate import unparse from mcpyrate.quotes import is_captured_value @@ -74,10 +74,10 @@ def dbg_block(body, args): # (the problem is we must syntactically find matches in the AST, and AST nodes don't support comparison) if type(args[0]) is not Name: # pragma: no cover, let's not test the macro expansion errors. raise SyntaxError("Custom debug print function must be specified by a bare name") - p = args[0] - pname = p.id # name of the print function as it appears in the user code + pfunc = args[0] + pname = pfunc.id # name of the print function as it appears in the user code else: - p = q[h[dbgprint_block]] + pfunc = q[h[dbgprint_block]] pname = "print" # override standard print function within this block class DbgBlockTransformer(ASTTransformer): @@ -86,13 +86,14 @@ def transform(self, tree): return tree # don't recurse! if type(tree) is Call and type(tree.func) is Name and tree.func.id == pname: names = [q[u[unparse(node)]] for node in tree.args] # x --> "x"; (1 + 2) --> "(1 + 2)"; ... - names = Tuple(elts=names, lineno=tree.lineno, col_offset=tree.col_offset) - values = Tuple(elts=tree.args, lineno=tree.lineno, col_offset=tree.col_offset) + names = q[t[names]] + values = q[t[tree.args]] tree.args = [names, values] # can't use inspect.stack in the printer itself because we want the line number *before macro expansion*. + lineno = tree.lineno if hasattr(tree, "lineno") else None tree.keywords += [keyword(arg="filename", value=q[h[callsite_filename]()]), - keyword(arg="lineno", value=(q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None]))] - tree.func = q[a[p]] + keyword(arg="lineno", value=q[u[lineno]])] + tree.func = pfunc return self.generic_visit(tree) return DbgBlockTransformer().visit(body) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index ba1f801d..d8db704e 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -308,14 +308,13 @@ def transform_starred(tree, dstarred=False): return tree else: - ln, co = tree.lineno, tree.col_offset thefunc = self.visit(tree.func) adata = [] for x in tree.args: if type(x) is Starred: # *args in Python 3.5+ v = transform_starred(x.value) - v = Starred(value=q[a[v]], lineno=ln, col_offset=co) + v = Starred(value=q[a[v]]) else: v = transform_arg(x) adata.append(v) @@ -331,8 +330,7 @@ def transform_starred(tree, dstarred=False): # Construct the call mycall = Call(func=q[h[maybe_force_args]], args=[q[a[thefunc]]] + [q[a[x]] for x in adata], - keywords=[keyword(arg=k, value=q[a[x]]) for k, x in kwdata], - lineno=ln, col_offset=co) + keywords=[keyword(arg=k, value=q[a[x]]) for k, x in kwdata]) tree = mycall return tree diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 59f8e9a2..9621b26b 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -23,7 +23,7 @@ Load) import sys -from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 +from mcpyrate.quotes import macros, q, u, n, a, t, h # noqa: F401 from mcpyrate import gensym from mcpyrate.markers import ASTMarker @@ -110,7 +110,7 @@ def _letimpl(bindings, body, mode): # - the exact AST structure, for the views letter = letf bindings = [q[(u[k], a[v])] for k, v in zip(names, values)] - newtree = q[h[letter](a[Tuple(elts=bindings)], a[body], mode=u[mode])] + newtree = q[h[letter](t[bindings], a[body], mode=u[mode])] return newtree def letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): @@ -434,8 +434,7 @@ def do0(tree): newelts.append(q[a[local(thelocalexpr)]]) newelts.extend(elts[1:]) newelts.append(q[_do0_result]) # noqa: F821 -# newtree = q[t[newelts]] # TODO: doesn't work, missing lineno TODO: test with mcpyrate - newtree = Tuple(elts=newelts, lineno=tree.lineno, col_offset=tree.col_offset) + newtree = q[t[newelts]] # TODO: Would be cleaner to use `do[]` as a hygienically captured macro. return do(newtree) # do0[] is also just a do[] diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 9d74dd6d..84bf56ae 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -4,7 +4,9 @@ # at macro expansion time. If you're looking for regular run-time let et al. macros, # see letdo.py. -from ast import (Name, Call, Starred, If, Constant, Expr, With, +from mcpyrate.quotes import macros, q, a # noqa: F401 + +from ast import (Name, Call, Starred, Expr, With, FunctionDef, AsyncFunctionDef, ClassDef, Attribute) from copy import deepcopy @@ -97,10 +99,10 @@ def register_binding(withstmt, mode, kind): args = [] if mode == "block": - value = If(test=Constant(value=1), - body=withstmt.body, - orelse=[], - lineno=stmt.lineno, col_offset=stmt.col_offset) + with q as value: + if 1: + with a: + withstmt.body else: # mode == "expr": if len(withstmt.body) != 1: raise SyntaxError("'with expr:' expected a one-item body (use a do[] if need more)") # pragma: no cover diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 6974ea77..1551a60b 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -487,8 +487,7 @@ def transform(self, tree): return tree # don't recurse! if type(tree) in (FunctionDef, AsyncFunctionDef): if type(tree.body[-1]) is not Return: - tree.body.append(Return(value=None, # bare "return" - lineno=tree.lineno, col_offset=tree.col_offset)) + tree.body.append(Return(value=None)) # bare "return" return self.generic_visit(tree) block_body = ImplicitBareReturnInjector().visit(block_body) @@ -670,18 +669,14 @@ def transform(tree): if type(tree.values[-1]) in (Call, IfExp, BoolOp): # must match above handlers # other items: not in tail position, compute normally if len(tree.values) > 2: - op_of_others = BoolOp(op=tree.op, values=tree.values[:-1], - lineno=tree.lineno, col_offset=tree.col_offset) + op_of_others = BoolOp(op=tree.op, values=tree.values[:-1]) else: op_of_others = tree.values[0] if type(tree.op) is Or: # or(data1, ..., datan, tail) --> it if any(others) else tail tree = aif(Tuple(elts=[op_of_others, - transform_data(Name(id="it", - lineno=tree.lineno, - col_offset=tree.col_offset)), - transform(tree.values[-1])], - lineno=tree.lineno, col_offset=tree.col_offset)) # tail-call item + transform_data(Name(id="it")), + transform(tree.values[-1])])) # tail-call item elif type(tree.op) is And: # and(data1, ..., datan, tail) --> tail if all(others) else False fal = q[False] From 5f25a42bdc069c65745b14ee180b6dcce2abcf48 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 16:04:46 +0300 Subject: [PATCH 011/832] oops, fix test --- unpythonic/syntax/tests/test_util.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/unpythonic/syntax/tests/test_util.py b/unpythonic/syntax/tests/test_util.py index 876907d4..a80c46fe 100644 --- a/unpythonic/syntax/tests/test_util.py +++ b/unpythonic/syntax/tests/test_util.py @@ -259,9 +259,6 @@ def ishello(tree): with testset("wrapwith"): with q as wrapwith_testdata: 42 # pragma: no cover - # known fake location information so we can check it copies correctly - wrapwith_testdata[0].lineno = 9001 - wrapwith_testdata[0].col_offset = 9 wrapped = wrapwith(q[n["ExampleContextManager"]], wrapwith_testdata) test[type(wrapped) is list] thewith = wrapped[0] From 74a4b67c5fadafa93af2ef4ee7f4276835f9fd4d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 16:04:51 +0300 Subject: [PATCH 012/832] remove unused import --- unpythonic/syntax/util.py | 1 - 1 file changed, 1 deletion(-) diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 9a6167eb..8515dc10 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -6,7 +6,6 @@ from ast import (Call, Lambda, FunctionDef, AsyncFunctionDef, If, With, withitem, stmt) -from mcpyrate.markers import ASTMarker from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer, ASTVisitor From 7db4b9611c91480535fb644dca3c02c2836b8d90 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 17:27:37 +0300 Subject: [PATCH 013/832] Python 3.9: if a generic has no args, it has no ".__args__". --- unpythonic/typecheck.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 567527dc..089cfd63 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -23,7 +23,7 @@ class _MyGenericAlias: # unused, but must be a class to support isinstance() ch pass try: - _MySupportsIndex = typing.SupportsIndex + _MySupportsIndex = typing.SupportsIndex # Python 3.8+ except AttributeError: # Python 3.7 and earlier # pragma: no cover class _MySupportsIndex: # unused, but must be a class to support isinstance() check. pass @@ -230,7 +230,8 @@ def get_origin(tp): if not isinstance(value, tuple): return False # bare `typing.Tuple`, no restrictions on length or element type. - if not T.__args__: + # Python 3.9: if a generic has no args, it has no `__args__` attribute. + if not hasattr(T, "__args__") or not T.__args__: return True # homogeneous element type, arbitrary length if len(T.__args__) == 2 and T.__args__[1] is Ellipsis: @@ -249,7 +250,8 @@ def get_origin(tp): def ismapping(statictype, runtimetype): if not isinstance(value, runtimetype): return False - if T.__args__ is None: # Python 3.6: consistent behavior with 3.7+, which use unconstrained TypeVar KT, VT. + # Python 3.9: if a generic has no args, it has no `__args__` attribute. + if not hasattr(T, "__args__") or T.__args__ is None: # Python 3.6: consistent behavior with 3.7+, which use unconstrained TypeVar KT, VT. args = (typing.TypeVar("KT"), typing.TypeVar("VT")) else: args = T.__args__ @@ -269,7 +271,8 @@ def ismapping(statictype, runtimetype): if safeissubclass(T, typing.ItemsView) or get_origin(T) is collections.abc.ItemsView: if not isinstance(value, collections.abc.ItemsView): return False - if T.__args__ is None: # Python 3.6: consistent behavior with 3.7+, which use unconstrained TypeVar KT, VT. + # Python 3.9: if a generic has no args, it has no `__args__` attribute. + if not hasattr(T, "__args__") or T.__args__ is None: # Python 3.6: consistent behavior with 3.7+, which use unconstrained TypeVar KT, VT. args = (typing.TypeVar("KT"), typing.TypeVar("VT")) else: args = T.__args__ @@ -293,8 +296,11 @@ def iscollection(statictype, runtimetype): # At run time, the `__args__` are actually empty - it looks # like a bare Sequence, which is invalid. HACK the special case. typeargs = (int,) - else: + # Python 3.9: if a generic has no args, it has no `__args__` attribute. + elif hasattr(T, "__args__"): typeargs = T.__args__ + else: + typeargs = None # Python 3.6: consistent behavior with 3.7+, which use an unconstrained TypeVar T. if typeargs is None: typeargs = (typing.TypeVar("T"),) From 75ffe1ff264639192fbce821bbba029a8cde7861 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 17:28:20 +0300 Subject: [PATCH 014/832] fully works on Python 3.9 now --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 4e6f9926..beaf7138 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,6 @@ As of [7bb1198](https://github.com/Technologicat/unpythonic/commit/7bb1198605087 The codebase already fully works on 3.8 and `mcpyrate`, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. -Beside internal changes related to the macro system, the major goal is full compatibility with Python 3.9, including the `unpythonic.typecheck` module, which is currently the last one failing tests on 3.9. - For details, see [the 0.15 milestone](https://github.com/Technologicat/unpythonic/milestone/1). I'm also considering renaming 0.15 to 1.0, since the codebase is mostly stable at this point, and we have already adhered to [semantic versioning](https://semver.org/) since 2019, anyway (albeit with a leading zero). From c962f52f448a81fab5842c45e228eb78f9db2490 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 17:31:29 +0300 Subject: [PATCH 015/832] upgrade coverage CI workflow to use Python 3.8 --- .github/workflows/coverage.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index b7e1e3cf..fc40d2ce 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.6] + python-version: [3.8] steps: - uses: actions/checkout@v2 From e05a199aa4a06c37895cc164404d07fd8d1751f0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 17:37:31 +0300 Subject: [PATCH 016/832] namedlambda: add NamedExpr test (currently disabled due to compatibility) --- unpythonic/syntax/tests/test_lambdatools.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index 3345f57c..c884c5c7 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -57,6 +57,11 @@ def runtests(): foo = let[(f7, lambda x: x) in f7] # let-binding: name as "f7" # noqa: F821 test[foo.__name__ == "f7"] + warn["NamedExpr test currently disabled for syntactic compatibility with Python 3.6 and 3.7."] + # if foo2 := (lambda x: x): # NamedExpr a.k.a. walrus operator (Python 3.8+) + # pass + # test[foo2.__name__ == "foo2"] + # function call with named arg def foo(func1, func2): test[func1.__name__ == "func1"] From 2fc20fb70aade6d2f1d24bb9dcc27ce6f24dd6d2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 30 Apr 2021 17:39:48 +0300 Subject: [PATCH 017/832] update compatibility note --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index beaf7138..2d191286 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f As of [7bb1198](https://github.com/Technologicat/unpythonic/commit/7bb1198605087f1dd7ca292e33afd53e5aa9721d), the initial porting effort of `unpythonic` to Python 3.8 and the new [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander is complete. In fact, if you want to play around with 0.15-pre, the code is already in `master`. -The codebase already fully works on 3.8 and `mcpyrate`, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. +The codebase already fully works, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. For details, see [the 0.15 milestone](https://github.com/Technologicat/unpythonic/milestone/1). From 5d070b5f1a7e1d3db6b21b4c09238ac780f6da8d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 00:11:25 +0300 Subject: [PATCH 018/832] use both pypy-3.6 and pypy-3.7 in CI --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 2b32bebf..7ffd1add 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.6, 3.7, 3.8, 3.9, "pypy3"] + python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7] steps: - uses: actions/checkout@v2 From 0273c4b5ba1528a051f705b73ddf1e494b81502c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 00:23:59 +0300 Subject: [PATCH 019/832] porting: remove make_isxpred, now obsolete `mcpyrate` does not need to rename hygienically captured values, so the name will always match exactly. If the use site needs something more complex, `isx` already supports that: just pass a predicate instead of a string as the name to be detected. --- unpythonic/syntax/__init__.py | 1 - unpythonic/syntax/lambdatools.py | 15 ++++++--------- unpythonic/syntax/lazify.py | 12 ++++++------ unpythonic/syntax/letdoutil.py | 16 ++++++++-------- unpythonic/syntax/nameutil.py | 12 ------------ unpythonic/syntax/tailtools.py | 4 ++-- unpythonic/syntax/tests/test_nameutil.py | 7 +------ unpythonic/syntax/util.py | 4 ++-- 8 files changed, 25 insertions(+), 46 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 0c486a94..30a9afa6 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -89,7 +89,6 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: `make_isxpred` is now obsolete because `mcpyrate` does not rename hygienic captures of run-time values. Make it explicit at the use sites what they want, and remove `make_isxpred`. (E.g. `curry` wants to match both `curryf` and `currycall`, exactly. Some use sites want to match only a single thing.) # TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 93525f1b..b1ccb120 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -16,13 +16,12 @@ from ..dynassign import dyn from ..misc import namelambda -from ..fun import orf from ..env import env from .astcompat import getconstant, Str, NamedExpr from .letdo import do from .letdoutil import islet, isenvassign, UnexpandedLetView, UnexpandedEnvAssignView, ExpandedDoView -from .util import (is_decorated_lambda, isx, make_isxpred, has_deco, +from .util import (is_decorated_lambda, isx, has_deco, destructure_decorated_lambda, detect_lambda) def multilambda(block_body): @@ -53,18 +52,17 @@ def issingleassign(tree): return type(tree) is Assign and len(tree.targets) == 1 and type(tree.targets[0]) is Name # detect a manual curry - iscurry = make_isxpred("curry") def iscurrywithfinallambda(tree): - if not (type(tree) is Call and isx(tree.func, iscurry) and tree.args): + if not (type(tree) is Call and isx(tree.func, "curry") and tree.args): return False return type(tree.args[-1]) is Lambda # Detect an autocurry from an already expanded "with autocurry". # CAUTION: These must match what unpythonic.syntax.curry.autocurry uses in its output. - iscurrycall = make_isxpred("currycall") - iscurryf = orf(make_isxpred("curryf"), make_isxpred("curry")) # auto or manual curry in a "with autocurry" + currycall_name = "currycall" + iscurryf = lambda name: name in ("curryf", "curry") # auto or manual curry in a "with autocurry" def isautocurrywithfinallambda(tree): - if not (type(tree) is Call and isx(tree.func, iscurrycall) and tree.args and + if not (type(tree) is Call and isx(tree.func, currycall_name) and tree.args and type(tree.args[-1]) is Call and isx(tree.args[-1].func, iscurryf)): return False return type(tree.args[-1].args[-1]) is Lambda @@ -235,7 +233,6 @@ def isfunctionoruserlambda(tree): # Create a renamed reference to the env() constructor to be sure the Call # nodes added by us have a unique .func (not used by other macros or user code) - _ismakeenv = make_isxpred("_envify") _envify = env class EnvifyTransformer(ASTTransformer): @@ -284,7 +281,7 @@ def isourupdate(thecall): self.generic_withstate(tree, enames=(enames + [ename]), bindings=newbindings) else: # leave alone the _envify() added by us - if type(tree) is Call and (isx(tree.func, _ismakeenv) or isourupdate(tree)): + if type(tree) is Call and (isx(tree.func, "_envify") or isourupdate(tree)): # don't recurse return tree # transform env-assignments into our envs diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index d8db704e..7b8d760c 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -10,7 +10,7 @@ from mcpyrate.walkers import ASTTransformer from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, - isx, make_isxpred, getname, is_decorator, wrapwith) + isx, getname, is_decorator, wrapwith) from .letdoutil import islet, isdo, ExpandedLetView from ..lazyutil import Lazy, passthrough_lazy_args, force, force1, maybe_force_args from ..dynassign import dyn @@ -87,8 +87,8 @@ def lazy(tree): # variant `frozendict(mapping1, mapping2, ...)`. _ctorcalls_that_take_exactly_one_positional_arg = {"tuple", "list", "set", "dict", "frozenset", "llist"} -islazy = make_isxpred("lazy") # unexpanded -isLazy = make_isxpred("Lazy") # expanded +unexpanded_lazy_name = "lazy" +expanded_lazy_name = "Lazy" def lazyrec(tree): # This helper doesn't need to recurse, so we don't need `ASTTransformer` here. def transform(tree): @@ -99,9 +99,9 @@ def transform(tree): elif type(tree) is Call and any(isx(tree.func, ctor) for ctor in _ctorcalls_all): p, k = _ctor_handling_modes[getname(tree.func)] lazify_ctorcall(tree, p, k) - elif type(tree) is Subscript and isx(tree.value, islazy): # unexpanded + elif type(tree) is Subscript and isx(tree.value, unexpanded_lazy_name): pass - elif type(tree) is Call and isx(tree.func, isLazy): # expanded + elif type(tree) is Call and isx(tree.func, expanded_lazy_name): pass else: # mcpyrate supports hygienic macro capture, so we can just splice unexpanded @@ -293,7 +293,7 @@ def transform_starred(tree, dstarred=False): # Lazy() is a strict function, takes a lambda, constructs a Lazy object # _autoref_resolve doesn't need any special handling elif (isdo(tree) or is_decorator(tree.func, "namelambda") or - any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, isLazy) or + any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, expanded_lazy_name) or any(isx(tree.func, s) for s in ("_autoref_resolve", "AutorefMarker"))): # here we know the operator (.func) to be one of specific names; # don't transform it to avoid confusing lazyrec[] (important if this diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 6581e092..3210df50 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -10,15 +10,15 @@ import sys from .astcompat import getconstant, Str -from .nameutil import isx, make_isxpred +from .nameutil import isx def where(*bindings): """[syntax] Only meaningful in a let[body, where((k0, v0), ...)].""" raise RuntimeError("where() is only meaningful in a let[body, where((k0, v0), ...)]") # pragma: no cover -_isletf = make_isxpred("letter") # name must match what ``unpythonic.syntax.letdo._letimpl`` uses in its output. -_isdof = make_isxpred("dof") # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. -_iscurrycall = make_isxpred("currycall") # output of ``unpythonic.syntax.curry`` +letf_name = "letter" # must match what ``unpythonic.syntax.letdo._letimpl`` uses in its output. +dof_name = "dof" # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. +currycall_name = "currycall" # output of ``unpythonic.syntax.curry`` # TODO: switch from call to subscript in name position for let_syntax templates. def canonize_bindings(elts, allow_call_in_name_position=False): # public as of v0.14.3+ @@ -98,9 +98,9 @@ def islet(tree, expanded=True): if type(tree) is not Call: return False kind = "expanded" - if isx(tree.func, _iscurrycall) and isx(tree.args[0], _isletf): + if isx(tree.func, currycall_name) and isx(tree.args[0], letf_name): kind = "curried" - elif not isx(tree.func, _isletf): + elif not isx(tree.func, letf_name): return False mode = [kw.value for kw in tree.keywords if kw.arg == "mode"] assert len(mode) == 1 and type(mode[0]) in (Constant, Str) @@ -203,9 +203,9 @@ def isdo(tree, expanded=True): if type(tree) is not Call: return False kind = "expanded" - if isx(tree.func, _iscurrycall) and isx(tree.args[0], _isdof): + if isx(tree.func, currycall_name) and isx(tree.args[0], dof_name): kind = "curried" - elif not isx(tree.func, _isdof): + elif not isx(tree.func, dof_name): return False return kind diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index abce0099..fd04857e 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -66,18 +66,6 @@ def isx(tree, x, accept_attr=True): (key and ismatch(name)) or (accept_attr and type(tree) is Attribute and ismatch(tree.attr))) -# TODO: obsolete function, remove -def make_isxpred(x): - """Make a predicate for isx. - - Here ``x`` is an ``str``; the resulting function will match also - hygienically captured identifiers. - """ - # `mcpyrate` only renames captured macros; the names of captured - # run-time values live in the keys in the `lookup_value` calls - # (where the original name is preserved, with no renaming needed). - return lambda name: name == x - def getname(tree, accept_attr=True): """The cousin of ``isx``. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 1551a60b..a398a60c 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -23,7 +23,7 @@ from mcpyrate.walkers import ASTTransformer, ASTVisitor from .astcompat import getconstant, NameConstant -from .util import (isx, make_isxpred, isec, +from .util import (isx, isec, detect_callec, detect_lambda, has_tco, sort_lambda_decorators, suggest_decorator_index, ContinuationsMarker, wrapwith, isexpandedmacromarker) @@ -601,7 +601,7 @@ def transform(self, tree): # Tail-position analysis for a return-value expression (also the body of a lambda). # Here we need to be very, very selective about where to recurse so this would not # benefit much from being made into an ASTTransformer. Just a function is fine. -_isjump = orf(make_isxpred("jump"), make_isxpred("loop")) +_isjump = lambda name: name in ("jump", "loop") def _transform_retexpr(tree, known_ecs, call_cb=None, data_cb=None): """Analyze and TCO a return-value expression or a lambda body. diff --git a/unpythonic/syntax/tests/test_nameutil.py b/unpythonic/syntax/tests/test_nameutil.py index 791921df..a7c1fcf8 100644 --- a/unpythonic/syntax/tests/test_nameutil.py +++ b/unpythonic/syntax/tests/test_nameutil.py @@ -6,7 +6,7 @@ from mcpyrate.quotes import macros, q, h # noqa: F401, F811 -from ...syntax.nameutil import isx, make_isxpred, getname +from ...syntax.nameutil import isx, getname from ast import Call @@ -26,11 +26,6 @@ def runtests(): test[isx(attribute, "ok")] test[not isx(attribute, "ok", accept_attr=False)] - with testset("make_isxpred"): - isfab = make_isxpred("fab") - test[isx(q[fab], isfab)] # noqa: F821 - test[isx(q[someobj.fab], isfab)] # noqa: F821 - with testset("getname"): test[getname(barename) == "ok"] test[getname(captured.func) == "capture_this"] diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 8515dc10..cb240777 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -11,7 +11,7 @@ from .astcompat import getconstant from .letdoutil import isdo, ExpandedDoView -from .nameutil import isx, make_isxpred, getname +from .nameutil import isx, getname from ..regutil import all_decorators, tco_decorators, decorator_registry @@ -52,7 +52,7 @@ def g(ec): # <-- should grab from here and `throw` covers the use of `unpythonic.ec.throw`.) """ fallbacks = ["ec", "brk", "throw"] - iscallec = partial(isx, x=make_isxpred("call_ec")) + iscallec = partial(isx, x="call_ec") def detect(tree): class Detector(ASTVisitor): def examine(self, tree): From 93aa8bd33d2769df5aa10b9b2424153e841c739f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:08:31 +0300 Subject: [PATCH 020/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 30a9afa6..1b242bd1 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -89,9 +89,12 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. +# TODO: have a common base class for all `unpythonic` `ASTMarker`s? # TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) +# TODO: `make_dynvar` needs to be better advertised in the docs. A workflow example would also be nice. + # TODO: Brackets: use "with test[...]" instead of "with test(...)" in the test modules # TODO: Remove any unused `expander` kwargs from the macro interface @@ -146,6 +149,8 @@ # TODO: `mcpyrate` now provides the necessary infrastructure, while `unpythonic` has the macros # TODO: needed to make interesting things happen. Update docs accordingly for both projects. +# TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... + # Syntax transformers and internal utilities from .autoref import autoref as _autoref from .autocurry import autocurry as _autocurry From 72e2a00b9f721b2739ba5f111e06c6b15366b110 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:10:01 +0300 Subject: [PATCH 021/832] add explicit __all__ exports for unpythonic.syntax modules --- unpythonic/syntax/astcompat.py | 6 ++++-- unpythonic/syntax/autocurry.py | 2 ++ unpythonic/syntax/autoref.py | 2 ++ unpythonic/syntax/dbg.py | 3 +++ unpythonic/syntax/forall.py | 5 ++++- unpythonic/syntax/ifexprs.py | 17 ++++++++++------- unpythonic/syntax/lambdatools.py | 5 +++++ unpythonic/syntax/lazify.py | 2 ++ unpythonic/syntax/letdo.py | 6 ++++++ unpythonic/syntax/letdoutil.py | 6 ++++++ unpythonic/syntax/letsyntax.py | 2 ++ unpythonic/syntax/nameutil.py | 8 +++++++- unpythonic/syntax/nb.py | 2 ++ unpythonic/syntax/prefix.py | 2 ++ unpythonic/syntax/scopeanalyzer.py | 6 ++++++ unpythonic/syntax/simplelet.py | 2 ++ unpythonic/syntax/tailtools.py | 4 ++++ unpythonic/syntax/testingtools.py | 6 ++++++ unpythonic/syntax/util.py | 16 +++++++++++++++- 19 files changed, 90 insertions(+), 12 deletions(-) diff --git a/unpythonic/syntax/astcompat.py b/unpythonic/syntax/astcompat.py index 25a05c5a..c1eb4653 100644 --- a/unpythonic/syntax/astcompat.py +++ b/unpythonic/syntax/astcompat.py @@ -1,8 +1,10 @@ # -*- coding: utf-8 -*- """Conditionally import AST node types only supported by recent enough Python versions (3.7+).""" -# This is an internal module and does not have an officially defined `__all__`. -# Any names defined here are fair game to use anywhere inside `unpythonic.syntax`. +__all__ = ["NamedExpr", + "Num", "Str", "Bytes", "NameConstant", "Ellipsis", + "Index", "ExtSlice", + "getconstant"] import ast diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index d68bf880..082748c7 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -1,6 +1,8 @@ # -*- coding: utf-8 -*- """Automatic currying. Transforms both function definitions and calls.""" +__all__ = ["autocurry"] + from ast import Call, Lambda, FunctionDef, AsyncFunctionDef from mcpyrate.quotes import macros, q, a, h # noqa: F401 diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index dc43f910..217af5d2 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -1,6 +1,8 @@ # -*- coding: utf-8 -*- """Implicitly reference attributes of an object.""" +__all__ = ["autoref"] + from ast import (Name, Assign, Load, Call, Lambda, With, Constant, arg, Attribute, Subscript, Store, Del) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 8f9416e2..8fb7bf7a 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -6,6 +6,9 @@ The printing can be customized; see ``dbgprint_block`` and ``dbgprint_expr``. """ +__all__ = ["dbgprint_block", "dbg_block", + "dbgprint_expr", "dbg_expr"] + from ast import Call, Name, keyword from mcpyrate.quotes import macros, q, u, a, t, h # noqa: F401 diff --git a/unpythonic/syntax/forall.py b/unpythonic/syntax/forall.py index a22feb1a..e497174a 100644 --- a/unpythonic/syntax/forall.py +++ b/unpythonic/syntax/forall.py @@ -1,6 +1,8 @@ # -*- coding: utf-8 -*- """Nondeterministic evaluation (a tuple comprehension with a multi-expr body).""" +__all__ = ["forall", "insist", "deny"] + from ast import Tuple, arg from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 @@ -9,9 +11,10 @@ from .letdoutil import isenvassign, UnexpandedEnvAssignView from ..amb import monadify -from ..amb import insist, deny # for re-export only # noqa: F401 from ..misc import namelambda +from ..amb import insist, deny # for re-export only # noqa: F401 + def forall(exprs): """[syntax, expr] Nondeterministic evaluation. diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index 4633228c..866d16bc 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -1,12 +1,22 @@ # -*- coding: utf-8 -*- """Anaphoric if.""" +__all__ = ["aif", "it", + "cond"] + from ast import Tuple from mcpyrate.quotes import macros, q, a # noqa: F811, F401 from .letdo import implicit_do, let +def aif(tree): + test, then, otherwise = [implicit_do(x) for x in tree.elts] + bindings = [q[(it, a[test])]] + body = q[a[then] if it else a[otherwise]] + # TODO: we should use a hygienically captured macro here. + return let(bindings, body) + # TODO: `mcpyrate` has a rudimentary capability like Racket's "syntax-parameterize". # TODO: Make `it` a name macro that errors out unless it appears inside an `aif`. # @@ -21,13 +31,6 @@ def __repr__(self): # pragma: no cover, we have a repr just in case one of thes return "" it = it() -def aif(tree): - test, then, otherwise = [implicit_do(x) for x in tree.elts] - bindings = [q[(it, a[test])]] - body = q[a[then] if it else a[otherwise]] - # TODO: we should use a hygienically captured macro here. - return let(bindings, body) - def cond(tree): if type(tree) is not Tuple: raise SyntaxError("Expected cond[test1, then1, test2, then2, ..., otherwise]") # pragma: no cover diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index b1ccb120..19383633 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -1,6 +1,11 @@ # -*- coding: utf-8 -*- """Lambdas with multiple expressions, local variables, and a name.""" +__all__ = ["multilambda", + "namedlambda", + "f", # for quicklambda + "envify"] + from ast import (Lambda, List, Name, Assign, Subscript, Call, FunctionDef, AsyncFunctionDef, Attribute, keyword, Dict, Constant, arg, copy_location) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 7b8d760c..7655a050 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -1,6 +1,8 @@ # -*- coding: utf-8 -*- """Automatic lazy evaluation of function arguments.""" +__all__ = ["lazy", "lazyrec", "lazify"] + from ast import (Lambda, FunctionDef, AsyncFunctionDef, Call, Name, Attribute, Starred, keyword, List, Tuple, Dict, Set, Subscript, Load) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 9621b26b..12e84eb3 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -1,6 +1,12 @@ # -*- coding: utf-8 -*- """Local bindings (let), imperative code in expression position (do).""" +__all__ = ["let", "letseq", "letrec", + "dlet", "dletseq", "dletrec", + "blet", "bletseq", "bletrec", + "local", "delete", "do", "do0", + "implicit_do"] # used by some other unpythonic.syntax constructs + # Let constructs are implemented as sugar around unpythonic.lispylet. # # We take this approach because letrec needs assignment (must create diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 3210df50..101d3934 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -5,6 +5,12 @@ Separate from util.py due to the length. """ +__all__ = ["where", + "canonize_bindings", # used by the macro interface layer + "isenvassign", "islet", "isdo", + "UnexpandedEnvAssignView", "UnexpandedLetView", "UnexpandedDoView", + "ExpandedLetView", "ExpandedDoView"] + from ast import (Call, Name, Subscript, Index, Compare, In, Tuple, List, Constant, BinOp, LShift, Lambda) import sys diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 84bf56ae..cabc848b 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -4,6 +4,8 @@ # at macro expansion time. If you're looking for regular run-time let et al. macros, # see letdo.py. +__all__ = ["let_syntax_expr", "let_syntax_block"] + from mcpyrate.quotes import macros, q, a # noqa: F401 from ast import (Name, Call, Starred, Expr, With, diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index fd04857e..ec4f7fc4 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -1,5 +1,11 @@ # -*- coding: utf-8 -*- -"""Utilities for working with identifiers in macros.""" +"""Utilities for working with identifiers in macros. + +Main purpose is to be able to query both direct and hygienically captured names +with a unified API. +""" + +__all__ = ["isx", "getname"] from ast import Name, Attribute diff --git a/unpythonic/syntax/nb.py b/unpythonic/syntax/nb.py index cbd59c3d..8c53d4d2 100644 --- a/unpythonic/syntax/nb.py +++ b/unpythonic/syntax/nb.py @@ -4,6 +4,8 @@ Auto-print top-level expressions, auto-assign last result as _. """ +__all__ = ["nb"] + # This is the kind of thing thinking with macros does to your program. ;) from ast import Expr diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index d5c828c0..d0aed4a3 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -4,6 +4,8 @@ Experimental, not for use in production code. """ +__all__ = ["prefix", "q", "u", "kw"] + from ast import Name, Call, Starred, Tuple, Load, Subscript import sys diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index ef26b254..8efe0a8c 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -67,6 +67,12 @@ OOPSLA '13. http://dx.doi.org/10.1145/2509136.2509536 """ +__all__ = ["isnewscope", + "scoped_transform", + "get_lexical_variables", + "get_names_in_store_context", + "get_names_in_del_context"] + from ast import (Name, Tuple, Lambda, FunctionDef, AsyncFunctionDef, ClassDef, Import, ImportFrom, Try, ListComp, SetComp, GeneratorExp, DictComp, Store, Del, Global, Nonlocal) diff --git a/unpythonic/syntax/simplelet.py b/unpythonic/syntax/simplelet.py index d5833a28..5b7a07f2 100644 --- a/unpythonic/syntax/simplelet.py +++ b/unpythonic/syntax/simplelet.py @@ -15,6 +15,8 @@ # Unlike the other submodules, this module contains the macro interface; # these macros are not part of the top-level ``unpythonic.syntax`` interface. +__all__ = ["let", "letseq"] + from mcpyrate.quotes import macros, q, a, t # noqa: F811, F401 from ast import arg diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index a398a60c..807a2685 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -3,6 +3,10 @@ The common factor is tail-position analysis.""" +__all__ = ["autoreturn", + "tco", + "call_cc", "continuations"] + from functools import partial from ast import (Lambda, FunctionDef, AsyncFunctionDef, diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 118cb175..721f261e 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -4,6 +4,12 @@ See also `unpythonic.test.fixtures` for the high-level machinery. """ +__all__ = ["isunexpandedtestmacro", "isexpandedtestmacro", "istestmacro", + "fail_expr", "error_expr", "warn_expr", + "the", + "test_expr", "test_expr_signals", "test_expr_raises", + "test_block", "test_block_signals", "test_block_raises"] + from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym, unparse diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index cb240777..b8ee9c17 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -1,5 +1,19 @@ # -*- coding: utf-8 -*- -"""Utilities for working with syntax.""" +"""Utilities for working with syntax. + +This module also contains the definitions for working with "decorated lambdas". +""" + +__all__ = ["isec", "detect_callec", + "detect_lambda", + "is_decorator", + "is_lambda_decorator", "is_decorated_lambda", "destructure_decorated_lambda", + "has_tco", "has_curry", "has_deco", + "sort_lambda_decorators", "suggest_decorator_index", + "eliminate_ifones", "transform_statements", + "wrapwith", + "isexpandedmacromarker", "UnpythonicExpandedMacroMarker", + "ExpandedContinuationsMarker", "ExpandedAutorefMarker"] from functools import partial From a81b0ba51c130842dd31d1c6aa38fb279bdd55f5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:10:20 +0300 Subject: [PATCH 022/832] comment wording --- unpythonic/syntax/util.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index b8ee9c17..873a1abe 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -280,7 +280,7 @@ def transform(self, tree): return FixIt().visit(tree) # TODO: should we just sort the decorators here, like we do for lambdas? -# (The current solution is less magic, but less uniform.) +# (The current solution is less magic, but also less uniform.) def suggest_decorator_index(deco_name, decorator_list): """Suggest insertion index for decorator deco_name in given decorator_list. From 8e647249a03ef102edbed3c47940e39cec229d7d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:11:34 +0300 Subject: [PATCH 023/832] make explicit that these markers are for expanded macros --- unpythonic/syntax/autocurry.py | 2 +- unpythonic/syntax/autoref.py | 22 +++++++++++----------- unpythonic/syntax/lazify.py | 2 +- unpythonic/syntax/tailtools.py | 6 +++--- unpythonic/syntax/util.py | 10 +++++----- 5 files changed, 21 insertions(+), 21 deletions(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 082748c7..f88d9a80 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -28,7 +28,7 @@ def transform(self, tree): return tree hascurry = self.state.hascurry - if type(tree) is Call and not isx(tree.func, "AutorefMarker"): + if type(tree) is Call and not isx(tree.func, "ExpandedAutorefMarker"): if has_curry(tree): # detect decorated lambda with manual curry # the lambda inside the curry(...) is the next Lambda node we will descend into. hascurry = True diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 217af5d2..8b6481ff 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -14,7 +14,7 @@ from .astcompat import getconstant, Str from .nameutil import isx -from .util import wrapwith, AutorefMarker +from .util import wrapwith, ExpandedAutorefMarker from .letdoutil import isdo, islet, ExpandedDoView, ExpandedLetView from ..lazyutil import force1, passthrough_lazy_args @@ -40,12 +40,12 @@ # # One possible clean-ish implementation is:: # -# with AutorefMarker("o"): # no-op at runtime +# with ExpandedAutorefMarker("o"): # no-op at runtime # x # --> (lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))) # x.a # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x")))).a # x[s] # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))))[s] # o # --> o (can only occur if an asname is supplied) -# with AutorefMarker("p"): +# with ExpandedAutorefMarker("p"): # x # --> (lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))) # x.a # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))).a # x[s] # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x")))[s] @@ -94,13 +94,13 @@ def autoref(block_body, args, asname): # TODO: We can't use `unpythonic.syntax.util.isexpandedmacromarker` here, because it # TODO: doesn't currently understand markers with arguments. Extend it? # - # with AutorefMarker("_o42"): + # with ExpandedAutorefMarker("_o42"): def isexpandedautorefblock(tree): if not (type(tree) is With and len(tree.items) == 1): return False ctxmanager = tree.items[0].context_expr return (type(ctxmanager) is Call and - isx(ctxmanager.func, "AutorefMarker") and + isx(ctxmanager.func, "ExpandedAutorefMarker") and len(ctxmanager.args) == 1 and type(ctxmanager.args[0]) in (Constant, Str)) # Python 3.8+: ast.Constant def getreferent(tree): return getconstant(tree.items[0].context_expr.args[0]) @@ -164,9 +164,9 @@ def transform(self, tree): # # expands to: # - # with AutorefMarker('_o5'): + # with ExpandedAutorefMarker('_o5'): # _o5 = e - # with AutorefMarker('_o4'): + # with ExpandedAutorefMarker('_o4'): # _o4 = (lambda _ar13: (_ar13[1] if _ar13[0] else e2))(_autoref_resolve((_o5, 'e2'))) # (lambda _ar9: (_ar9[1] if _ar9[0] else e))(_autoref_resolve((_o4, _o5, 'e'))) # @@ -183,9 +183,9 @@ def transform(self, tree): # # expands to: # - # with AutorefMarker('outer'): + # with ExpandedAutorefMarker('outer'): # outer = e - # with AutorefMarker('inner'): + # with ExpandedAutorefMarker('inner'): # inner = (lambda _ar17: (_ar17[1] if _ar17[0] else e2))(_autoref_resolve((outer, 'e2'))) # outer # <-- !!! # @@ -198,7 +198,7 @@ def transform(self, tree): else: add_to_resolver_list(tree, q[n[o]]) # _autoref_resolve((p, "x")) --> _autoref_resolve((p, o, "x")) return tree - elif type(tree) is Call and isx(tree.func, "AutorefMarker"): # nested autorefs + elif type(tree) is Call and isx(tree.func, "ExpandedAutorefMarker"): # nested autorefs return tree elif type(tree) is Name and (type(tree.ctx) is Load or not tree.ctx) and tree.id not in referents: tree = makeautoreference(tree) @@ -220,5 +220,5 @@ def transform(self, tree): for stmt in block_body: newbody.append(AutorefTransformer(referents=always_skip + [o]).visit(stmt)) - return wrapwith(item=q[h[AutorefMarker](u[o])], + return wrapwith(item=q[h[ExpandedAutorefMarker](u[o])], body=newbody) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 7655a050..ff6e6b9b 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -296,7 +296,7 @@ def transform_starred(tree, dstarred=False): # _autoref_resolve doesn't need any special handling elif (isdo(tree) or is_decorator(tree.func, "namelambda") or any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, expanded_lazy_name) or - any(isx(tree.func, s) for s in ("_autoref_resolve", "AutorefMarker"))): + any(isx(tree.func, s) for s in ("_autoref_resolve", "ExpandedAutorefMarker"))): # here we know the operator (.func) to be one of specific names; # don't transform it to avoid confusing lazyrec[] (important if this # is an inner call in the arglist of an outer, lazy call, since it diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 807a2685..4fb8b3f3 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -30,7 +30,7 @@ from .util import (isx, isec, detect_callec, detect_lambda, has_tco, sort_lambda_decorators, - suggest_decorator_index, ContinuationsMarker, wrapwith, isexpandedmacromarker) + suggest_decorator_index, ExpandedContinuationsMarker, wrapwith, isexpandedmacromarker) from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView from .ifexprs import aif @@ -91,7 +91,7 @@ def tco(block_body): for stmt in block_body: # skip nested, already expanded "with continuations" blocks # (needed to support continuations in the Lispython dialect, which applies tco globally) - if isexpandedmacromarker("ContinuationsMarker", stmt): + if isexpandedmacromarker("ExpandedContinuationsMarker", stmt): new_block_body.append(stmt) continue @@ -523,7 +523,7 @@ def transform(self, tree): # Leave a marker so "with tco", if applied, can ignore the expanded "with continuations" block # (needed to support continuations in the Lispython dialect, since it applies tco globally.) - return wrapwith(item=q[h[ContinuationsMarker]], + return wrapwith(item=q[h[ExpandedContinuationsMarker]], body=new_block_body) # ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 873a1abe..93b825ce 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -428,10 +428,10 @@ def isexpandedmacromarker(typename, tree): Example. If ``tree`` is the AST for the following code:: - with ContinuationsMarker: + with ExpandedContinuationsMarker: ... - then ``isexpandedmacromarker("ContinuationsMarker", tree)`` returns ``True``. + then ``isexpandedmacromarker("ExpandedContinuationsMarker", tree)`` returns ``True``. **NOTE**: The markers this function detects remain in the AST at run time; they inherit from `unpythonic.syntax.util.UnpythonicExpandedMacroMarker`. @@ -479,13 +479,13 @@ def __enter__(cls): def __exit__(cls, exctype, excvalue, traceback): pass # pragma: no cover -class ContinuationsMarker(metaclass=UnpythonicExpandedMacroMarker): +class ExpandedContinuationsMarker(metaclass=UnpythonicExpandedMacroMarker): """AST marker for an expanded "with continuations" block.""" pass # pragma: no cover # This one must be "instantiated", because we need to pass information at -# macro expansion time using the ctor call syntax, e.g. `AutorefMarker("o")`. -class AutorefMarker(metaclass=UnpythonicExpandedMacroMarker): +# macro expansion time using the ctor call syntax, e.g. `ExpandedAutorefMarker("o")`. +class ExpandedAutorefMarker(metaclass=UnpythonicExpandedMacroMarker): """AST marker for an expanded "with autoref[o]" block.""" def __init__(self, varname): self.varname = varname # not needed, but doesn't hurt either. From ae7a4f83bc9c9cda13892200c5a0adf5339b6ea3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:11:48 +0300 Subject: [PATCH 024/832] add TODO --- unpythonic/syntax/autoref.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 8b6481ff..85d7c540 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -74,6 +74,12 @@ # In reality, we also capture-and-assign the autoref'd expr into a gensym'd variable (instead of referring # to ``o`` and ``p`` directly), so that arbitrary expressions can be autoref'd without giving them # a name in user code. +# +# TODO: Consider whether we could use a `mcpyrate.markers.ASTMarker` (which could +# TODO: be deleted before the code reaches run time, instead of leaving it in like +# TODO: `UnpythonicExpandedMacroMarker`s are). May need a postprocess hook in the +# TODO: expander, so that we could register a function that deletes autoref markers +# TODO: at the expander's global postprocess pass. @passthrough_lazy_args def _autoref_resolve(args): From 4225e957f4e2b81321334fa5ac72880de1115b66 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:11:59 +0300 Subject: [PATCH 025/832] improve ignore list (always_skip) in autoref --- unpythonic/syntax/autoref.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 85d7c540..8ce06df6 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -16,6 +16,7 @@ from .nameutil import isx from .util import wrapwith, ExpandedAutorefMarker from .letdoutil import isdo, islet, ExpandedDoView, ExpandedLetView +from .testingtools import _test_function_names from ..lazyutil import force1, passthrough_lazy_args @@ -218,10 +219,12 @@ def transform(self, tree): # We are a second-pass macro (inside out), so any first-pass macro invocations, # as well as any second-pass macro invocations inside the `with autoref` block, # have already expanded by the time we run our transformer. - always_skip = ['letter', 'dof', 'namelambda', 'curry', 'currycall', 'lazy', 'lazyrec', 'maybe_force_args', - # test framework stuff - 'unpythonic_assert', 'unpythonic_assert_signals', 'unpythonic_assert_raises', - 'callsite_filename', 'returns_normally'] + always_skip = ['letter', 'dof', # let/do subsystem + 'namelambda', # lambdatools subsystem + 'curry', 'curryf' 'currycall', # autocurry subsystem + 'lazy', 'lazyrec', 'maybe_force_args', # lazify subsystem + # the test framework subsystem + 'callsite_filename', 'returns_normally'] + _test_function_names newbody = [Assign(targets=[q[n[o]]], value=args[0])] for stmt in block_body: newbody.append(AutorefTransformer(referents=always_skip + [o]).visit(stmt)) From e352a579618670b04d9ccf07d56e96035c2432b1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:12:14 +0300 Subject: [PATCH 026/832] variable naming in docstring code example --- unpythonic/syntax/nameutil.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index ec4f7fc4..b8e2eadc 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -16,7 +16,7 @@ def isx(tree, x, accept_attr=True): Alternatively, ``x`` may be a predicate that accepts a ``str`` and returns whether it matches, to support more complex matching - (e.g. ``lambda s: s.startswith("foo")``). + (e.g. ``lambda name: name.startswith("foo")``). Both bare names and attributes can be recognized, to support both from-imports and regular imports of ``somemodule.x``. From a9f90e4c6604eb2ff8fe76b142dd173feb8269b6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:12:28 +0300 Subject: [PATCH 027/832] remove unused import --- unpythonic/syntax/tailtools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 4fb8b3f3..49923282 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -36,7 +36,7 @@ from ..dynassign import dyn from ..it import uniqify -from ..fun import identity, orf +from ..fun import identity from ..tco import trampolined, jump from ..lazyutil import passthrough_lazy_args From 60fb2c6299f7eeb0c0f688ad29ae3e32a2ace0aa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:12:43 +0300 Subject: [PATCH 028/832] update comments --- unpythonic/syntax/testingtools.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 721f261e..04779a36 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -28,14 +28,11 @@ from .util import isx -from ..test import fixtures +from ..test import fixtures # unpythonic.test.fixtures, regular (non-macro) code belonging to the framework # ----------------------------------------------------------------------------- # Helper for other macros to detect uses of the ones we define here. -# TODO: Detect asserters only? Now this breaks the handling of the[] in a prefix block. -# TODO: It should be handled like any expr, but currently it's skipped because it's listed here. - # Note the unexpanded `error[]` macro is distinguishable from a call to # the function `unpythonic.conditions.error`, because a macro invocation # is an `ast.Subscript`, whereas a function call is an `ast.Call`. From 3c0bd8be313b5e5b85d9389f285915a593e7da26 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:12:51 +0300 Subject: [PATCH 029/832] mark a TODO --- unpythonic/syntax/testingtools.py | 1 + 1 file changed, 1 insertion(+) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 04779a36..5c713a47 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -36,6 +36,7 @@ # Note the unexpanded `error[]` macro is distinguishable from a call to # the function `unpythonic.conditions.error`, because a macro invocation # is an `ast.Subscript`, whereas a function call is an `ast.Call`. +# TODO: Maybe these lists should be public, autoref already uses the list of functions. _test_asserter_names = ["test", "test_signals", "test_raises", "error", "fail", "warn"] _test_function_names = ["unpythonic_assert", "unpythonic_assert_signals", From 23b52d919622d9896a92282dc616932bf1f222dd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:23:58 +0300 Subject: [PATCH 030/832] porting: pass macro arguments to test macros using brackets --- doc/macros.md | 16 ++++++------- unpythonic/syntax/__init__.py | 2 +- unpythonic/syntax/testingtools.py | 8 +++---- unpythonic/syntax/tests/test_lazify.py | 2 +- unpythonic/syntax/tests/test_letdo.py | 2 +- unpythonic/syntax/tests/test_letdoutil.py | 8 +++---- .../syntax/tests/testing_testingtools.py | 8 +++---- unpythonic/tests/test_assignonce.py | 8 +++---- unpythonic/tests/test_collections.py | 14 +++++------ unpythonic/tests/test_conditions.py | 24 +++++++++---------- unpythonic/tests/test_dispatch.py | 6 ++--- unpythonic/tests/test_dynassign.py | 2 +- unpythonic/tests/test_ec.py | 2 +- unpythonic/tests/test_env.py | 14 +++++------ unpythonic/tests/test_fploop.py | 16 ++++++------- unpythonic/tests/test_fun.py | 4 ++-- unpythonic/tests/test_fup.py | 2 +- unpythonic/tests/test_let.py | 2 +- unpythonic/tests/test_lispylet.py | 2 +- unpythonic/tests/test_llist.py | 2 +- unpythonic/tests/test_seq.py | 2 +- unpythonic/tests/test_tco.py | 2 +- 22 files changed, 74 insertions(+), 74 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 0c32281d..cc478234 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1671,7 +1671,7 @@ We provide the low-level syntactic constructs `test[]`, `test_raises[]` and `tes By default, the `test[expr]` macro asserts that the value of `expr` is truthy. If you want to assert only that `expr` runs to completion normally, use `test[returns_normally(expr)]`. -The test macros also come in block variants, `with test`, `with test_raises(exctype)`, `with test_signals(exctype)`. +The test macros also come in block variants, `with test`, `with test_raises[exctype]`, `with test_signals[exctype]`. As usual in test frameworks, the test constructs behave somewhat like `assert`, with the difference that a failure or error will not abort the whole unit (unless explicitly asked to do so). There is no return value; upon success, the test constructs return `None`. Upon failure (test assertion not satisfied) or error (unexpected exception or signal), the failure or error is reported, and further tests continue running. @@ -1761,23 +1761,23 @@ with test: body ... return expr -with test(message): +with test[message]: body ... -with test(message): +with test[message]: body ... return expr -with test_raises(exctype): +with test_raises[exctype]: body ... -with test_raises(exctype, message): +with test_raises[exctype, message]: body ... -with test_signals(exctype): +with test_signals[exctype]: body ... -with test_signals(exctype, message): +with test_signals[exctype, message]: body ... ``` @@ -1798,7 +1798,7 @@ By default, a `with test` block asserts just that it completes normally. If you (Another way to view the default behavior is that the `with test` macro injects a `return True` at the end of the block, if there is no `return`. This is actually how the default behavior is implemented.) -The `with test_raises(exctype)` and `with test_signals(exctype)` blocks assert that the block raises (respectively, signals) the declared exception (condition) type. These blocks are implicitly lifted into functions, too, but they do not check the return value. For them, **not** raising/signaling the declared exception/condition type is considered a test failure. Raising/signaling some other (hence unexpected) exception/condition type is considered an error. +The `with test_raises[exctype]` and `with test_signals[exctype]` blocks assert that the block raises (respectively, signals) the declared exception (condition) type. These blocks are implicitly lifted into functions, too, but they do not check the return value. For them, **not** raising/signaling the declared exception/condition type is considered a test failure. Raising/signaling some other (hence unexpected) exception/condition type is considered an error. #### `the`: capture the value of interesting subexpressions diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 1b242bd1..195a98c9 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -2462,7 +2462,7 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 ... return expr # optional - with test(message): + with test[message]: body0 ... return expr # optional diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 5c713a47..21a5d494 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -550,14 +550,14 @@ def test_block(block_body, args): filename = q[h[callsite_filename]()] asserter = q[h[unpythonic_assert]] - # with test(message): + # with test[message]: if len(args) == 1: message = args[0] # with test: elif len(args) == 0: message = q[None] else: - raise SyntaxError('Expected `with test:` or `with test(message):`') + raise SyntaxError('Expected `with test:` or `with test[message]:`') # Before we edit the tree, get the source code in its pre-transformation # state, so we can include that into the test failure message. @@ -620,10 +620,10 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): ln = q[u[first_stmt.lineno]] if hasattr(first_stmt, "lineno") else q[None] filename = q[h[callsite_filename]()] - # with test_raises(exctype, message): + # with test_raises[exctype, message]: if len(args) == 2: exctype, message = args - # with test_raises(exctype): + # with test_raises[exctype]: elif len(args) == 1: exctype = args[0] message = q[None] diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index e5b18be0..b2b4c4d6 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -456,7 +456,7 @@ def func2(x): test[func1(21) == 42] print("*** This error case SHOULD PRINT A WARNING:", file=stderr) - with test_raises(RuntimeError): + with test_raises[RuntimeError]: @trampolined def func3(): return jump(42) diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index fc6abf61..e02d6da4 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -397,7 +397,7 @@ def test13(): x = "the unused local x" # noqa: F841, this `x` being unused is the point of this test. # pragma: no cover test[test13() == "the env x"] - with test_raises(NameError, "should have tried to access the deleted nonlocal x"): + with test_raises[NameError, "should have tried to access the deleted nonlocal x"]: x = "the nonlocal x" @dlet((x, "the env x")) def test14(): diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index 5ca5832a..8167bf33 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -168,7 +168,7 @@ def f2(): test_raises[TypeError, UnexpandedEnvAssignView(q[x]), # noqa: F821 "not an env assignment"] - with test_raises(TypeError, "name must be str"): + with test_raises[TypeError, "name must be str"]: view.name = 1234 # -------------------------------------------------------------------------------- @@ -239,7 +239,7 @@ def f3(): test[len(view.bindings) == 2] test[unparse(view.bindings[0]) == "(z, 21)"] test[unparse(view.bindings[1]) == "(t, 2)"] - with test_raises(TypeError, "decorator let does not have an accessible body"): + with test_raises[TypeError, "decorator let does not have an accessible body"]: view.body = q[x] # noqa: F821 test_raises[TypeError, @@ -308,7 +308,7 @@ def f4(): test_raises[TypeError, view.body, "decorator let does not have an accessible body"] - with test_raises(TypeError, "decorator let does not have an accessible body"): + with test_raises[TypeError, "decorator let does not have an accessible body"]: view.body = q[x] # noqa: F821 test[view.envname is None] # dlet decorator doesn't have an envname, either @@ -404,7 +404,7 @@ def f5(): test_raises[TypeError, view.body, "decorator let does not have an accessible body"] - with test_raises(TypeError, "decorator let does not have an accessible body"): + with test_raises[TypeError, "decorator let does not have an accessible body"]: view.body = q[x] # noqa: F821 test[view.envname is not None] # dletrec decorator has envname in the bindings diff --git a/unpythonic/syntax/tests/testing_testingtools.py b/unpythonic/syntax/tests/testing_testingtools.py index 431f5d77..1187c560 100644 --- a/unpythonic/syntax/tests/testing_testingtools.py +++ b/unpythonic/syntax/tests/testing_testingtools.py @@ -214,17 +214,17 @@ def counter(): # return a + a == 4 # # # A test block can have a failure message: - # with test("should be three, no?"): + # with test["should be three, no?"]: # a = 2 # return a + a == 3 # # # Similarly, there are also `with test_raises` and `with test_signals` blocks, # # though they don't support `return` - they always assert that the block # # raises or signals, respectively. - # with test_raises(RuntimeError): + # with test_raises[RuntimeError]: # raise RuntimeError() # - # with test_raises(RuntimeError, "should have raised"): + # with test_raises[RuntimeError, "should have raised"]: # raise RuntimeError() # # # By default, for test failure reporting, `test[]` captures as "result": @@ -272,7 +272,7 @@ def counter(): # with testset("normal return, don't care about value"): # # There's also a block variant that asserts the block completes normally # # (no exception or signal). - # with test("block variant"): + # with test["block variant"]: # print("hello world") # # # To get that effect in the expression variant, call `returns_normally`: diff --git a/unpythonic/tests/test_assignonce.py b/unpythonic/tests/test_assignonce.py index 5968e6c3..ccdf9e6f 100644 --- a/unpythonic/tests/test_assignonce.py +++ b/unpythonic/tests/test_assignonce.py @@ -7,17 +7,17 @@ def runtests(): with assignonce() as e: - with test("basic usage"): + with test["basic usage"]: e.a = 2 e.b = 3 - with test_raises(AttributeError, "should not be able to redefine an already defined name"): + with test_raises[AttributeError, "should not be able to redefine an already defined name"]: e.a = 5 - with test("rebind"): + with test["rebind"]: e.set("a", 42) # rebind - with test_raises(AttributeError, "should not be able to rebind an unbound name"): + with test_raises[AttributeError, "should not be able to rebind an unbound name"]: e.set("c", 3) if __name__ == '__main__': # pragma: no cover diff --git a/unpythonic/tests/test_collections.py b/unpythonic/tests/test_collections.py index 1015fab8..d318607d 100644 --- a/unpythonic/tests/test_collections.py +++ b/unpythonic/tests/test_collections.py @@ -104,7 +104,7 @@ def f(b): b4 << cat # same as b4.set(cat) test[unbox(b4) is cat] - with test_raises(TypeError, "box is mutable, should not be hashable"): + with test_raises[TypeError, "box is mutable, should not be hashable"]: d = {} d[b] = "foo" @@ -253,7 +253,7 @@ class Zee: with testset("frozendict"): d3 = frozendict({'a': 1, 'b': 2}) test[d3['a'] == 1] - with test_raises(TypeError, "frozendict is immutable, should not be writable"): + with test_raises[TypeError, "frozendict is immutable, should not be writable"]: d3['c'] = 42 d4 = frozendict(d3, a=42) # functional update @@ -393,11 +393,11 @@ class Zee: lst = list(range(5)) v = view(lst)[2:] - with test_raises(TypeError): + with test_raises[TypeError]: v[2, 3] = 42 # multidimensional indexing not supported - with test_raises(IndexError): + with test_raises[IndexError]: v[9001] = 42 - with test_raises(IndexError): + with test_raises[IndexError]: v[-9001] = 42 # read-only live view for sequences @@ -411,7 +411,7 @@ class Zee: test[type(v[1:]) is roview] # slicing a read-only view gives another read-only view test[v[1:] == [3, 4, 5]] test_raises[TypeError, view(v[1:])] # cannot create a writable view into a read-only view - with test_raises(TypeError, "read-only view should not support item assignment"): + with test_raises[TypeError, "read-only view should not support item assignment"]: v[2] = 3 test_raises[AttributeError, v.reverse()] # read-only view does not support in-place reverse @@ -459,7 +459,7 @@ class Zee: test[s2 == (1, 2, 23, 42, 5)] test[tpl == (1, 2, 3, 4, 5)] - with test_raises(TypeError): + with test_raises[TypeError]: ShadowedSequence(s4, "la la la", "new value") # not a valid index specification # no-op ShadowedSequence is allowed diff --git a/unpythonic/tests/test_conditions.py b/unpythonic/tests/test_conditions.py index 999c890a..0f7b00ed 100644 --- a/unpythonic/tests/test_conditions.py +++ b/unpythonic/tests/test_conditions.py @@ -91,7 +91,7 @@ def highlevel(): # When the "proceed" restart is invoked, it causes the `cerror()` call in # the low-level code to return normally. So execution resumes from where it # left off, never mind that a condition occurred. - with test("basic usage proceed"): # barrier against stray exceptions/signals + with test["basic usage proceed"]: # barrier against stray exceptions/signals with handlers((OddNumberError, proceed)): # We would like to: # `test[lowlevel() == list(range(10))]` @@ -116,22 +116,22 @@ def highlevel(): # The restart name "use_value" is commonly used for the use case "resume with this value", # so the library has a eponymous function to invoke it. - with test("basic usage use_value"): + with test["basic usage use_value"]: with handlers((OddNumberError, lambda c: use_value(c.x))): result = lowlevel() test[result == list(range(10))] - with test("basic usage double"): + with test["basic usage double"]: with handlers((OddNumberError, lambda c: invoke("double", c.x))): result = lowlevel() test[result == [0, 2 * 1, 2, 2 * 3, 4, 2 * 5, 6, 2 * 7, 8, 2 * 9]] - with test("basic usage drop"): + with test["basic usage drop"]: with handlers((OddNumberError, lambda c: invoke("drop", c.x))): result = lowlevel() test[result == [0, 2, 4, 6, 8]] - with test("basic usage bail"): + with test["basic usage bail"]: try: with handlers((OddNumberError, lambda c: invoke("bail", c.x))): lowlevel() @@ -161,7 +161,7 @@ def lowlevel(): out.append(k) return out def highlevel(): - with test("basic usage use_value 2"): + with test["basic usage use_value 2"]: with handlers((OddNumberError, lambda c: use_value(42))): result = lowlevel() test[result == [0, 42, 2, 42, 4, 42, 6, 42, 8, 42]] @@ -222,7 +222,7 @@ def highlevel1(): # Use case where we want to resume at the low level (in a real-world application, repairing the error). # Note we need new code only at the high level; the mid and low levels remain as-is. def highlevel2(): - with test("resume at low level"): + with test["resume at low level"]: with handlers((TellMeHowToRecover, lambda c: invoke("resume_low", "resumed at low level"))): result = midlevel() test[result == "resumed at low level > normal exit from low level > normal exit from mid level"] @@ -230,7 +230,7 @@ def highlevel2(): # Use case where we want to resume at the mid level (in a real-world application, skipping the failed part). def highlevel3(): - with test("resume at mid level"): + with test["resume at mid level"]: with handlers((TellMeHowToRecover, lambda c: invoke("resume_mid", "resumed at mid level"))): result = midlevel() test[result == "resumed at mid level > normal exit from mid level"] @@ -449,7 +449,7 @@ def invoke_if_exists(restart_name): # # Note we place the `test_raises` construct on the outside, to avoid intercepting # the `signal(JustACondition)`. - with test_raises(NoItDidntExist, "nonexistent restart"): + with test_raises[NoItDidntExist, "nonexistent restart"]: with handlers((JustACondition, lambda: invoke_if_exists("myrestart"))): signal(JustACondition()) finding() @@ -470,7 +470,7 @@ def errorcases(): test_raises[ControlError, invoke("woo")] # error case: invoke an undefined restart - with test_signals(ControlError, "should yell when trying to invoke a nonexistent restart"): + with test_signals[ControlError, "should yell when trying to invoke a nonexistent restart"]: with restarts(foo=(lambda x: x)): invoke("bar") @@ -482,10 +482,10 @@ def errorcases(): test_signals[TypeError, invoke(42)] # invalid bindings - with test_signals(TypeError): + with test_signals[TypeError]: with restarts(myrestart=42): # name=callable, ... pass # pragma: no cover - with test_signals(TypeError): + with test_signals[TypeError]: with handlers(("ha ha ha", 42)): # (excspec, callable), ... pass # pragma: no cover errorcases() diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index fa222725..a285ba00 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -103,7 +103,7 @@ def f2(x: int): def f3(x: typing.Any): # not @generic! return False - with test_raises(TypeError, "should not be able to @generic_for a non-generic function"): + with test_raises[TypeError, "should not be able to @generic_for a non-generic function"]: @generic_for(f3) def f4(x: int): return x @@ -221,7 +221,7 @@ def instmeth(self, x: float): test_raises[TypeError, jack(3.14)] # jack only accepts int or str with testset("error cases"): - with test_raises(TypeError, "@typed should only accept a single method"): + with test_raises[TypeError, "@typed should only accept a single method"]: @typed def errorcase1(x: int): pass # pragma: no cover @@ -229,7 +229,7 @@ def errorcase1(x: int): def errorcase1(x: str): # noqa: F811 pass # pragma: no cover - with test_raises(TypeError, "@generic should complain about missing type annotations"): + with test_raises[TypeError, "@generic should complain about missing type annotations"]: @generic def errorcase2(x): pass # pragma: no cover diff --git a/unpythonic/tests/test_dynassign.py b/unpythonic/tests/test_dynassign.py index c6846334..d3457b2a 100644 --- a/unpythonic/tests/test_dynassign.py +++ b/unpythonic/tests/test_dynassign.py @@ -81,7 +81,7 @@ def threadtest(q): test[noimplicits(dyn.items()) == (("a", 1), ("b", 2), ("c", 23), ("d", 4))] dyn.a = 42 # update occurs in the nearest enclosing dynamic scope that has the name bound test[noimplicits(dyn.items()) == (("a", 42), ("b", 2), ("c", 23), ("d", 4))] - with test_raises(AttributeError, "should not be able to update unbound dynamic variable"): + with test_raises[AttributeError, "should not be able to update unbound dynamic variable"]: dyn.e = 5 # subscript notation also works for updating diff --git a/unpythonic/tests/test_ec.py b/unpythonic/tests/test_ec.py index febde782..eb63de72 100644 --- a/unpythonic/tests/test_ec.py +++ b/unpythonic/tests/test_ec.py @@ -81,7 +81,7 @@ def inner(): test[result == 42] with testset("error case"): - with test_raises(RuntimeError, "should not be able to call an ec instance outside its dynamic extent"): + with test_raises[RuntimeError, "should not be able to call an ec instance outside its dynamic extent"]: @call_ec def erroneous(ec): return ec diff --git a/unpythonic/tests/test_env.py b/unpythonic/tests/test_env.py index 5eeb45fc..ec0efbd8 100644 --- a/unpythonic/tests/test_env.py +++ b/unpythonic/tests/test_env.py @@ -105,7 +105,7 @@ def runtests(): with testset("error cases"): with env(x=1) as e: e.finalize() - with test_raises(AttributeError, "should not be able to add new bindings to a finalized environment"): + with test_raises[AttributeError, "should not be able to add new bindings to a finalized environment"]: e.y = 42 # undefined name @@ -116,28 +116,28 @@ def runtests(): test_raises[AttributeError, e.set("foo", 42)] # invalid, set() only modifies existing bindings with env() as e: - with test_raises(ValueError, "should detect invalid identifier in __setitem__"): + with test_raises[ValueError, "should detect invalid identifier in __setitem__"]: e["∞"] = 1 # invalid identifier in store context (__setitem__) with env() as e: - with test_raises(ValueError, "should detect invalid identifier in __getitem__"): + with test_raises[ValueError, "should detect invalid identifier in __getitem__"]: e["∞"] # invalid identifier in load context (__getitem__) with env() as e: - with test_raises(ValueError, "should detect invalid identifier in __delitem__"): + with test_raises[ValueError, "should detect invalid identifier in __delitem__"]: del e["∞"] # invalid identifier in del context (__delitem__) with env() as e: - with test_raises(AttributeError, "overwriting a reserved name should not be allowed"): + with test_raises[AttributeError, "overwriting a reserved name should not be allowed"]: e.set = {1, 2, 3} with env(x=1) as e: e.finalize() - with test_raises(TypeError, "deleting binding from finalized environment should not be allowed"): + with test_raises[TypeError, "deleting binding from finalized environment should not be allowed"]: del e.x with env() as e: - with test_raises(AttributeError, "deleting nonexistent binding should not be allowed"): + with test_raises[AttributeError, "deleting nonexistent binding should not be allowed"]: del e.x with env(x=1, y=2) as e: diff --git a/unpythonic/tests/test_fploop.py b/unpythonic/tests/test_fploop.py index 43dc1de4..5dd2b3f5 100644 --- a/unpythonic/tests/test_fploop.py +++ b/unpythonic/tests/test_fploop.py @@ -61,27 +61,27 @@ def s(loop, acc=0, i=0): test[s == 35] with testset("error cases"): - with test_raises(ValueError, "@looped: should detect invalid definition, no loop parameter"): + with test_raises[ValueError, "@looped: should detect invalid definition, no loop parameter"]: @looped def s(): fail["Should not be reached because the definition is faulty."] # pragma: no cover - with test_raises(ValueError, "@looped: should detect invalid definition, extra parameter not initialized"): + with test_raises[ValueError, "@looped: should detect invalid definition, extra parameter not initialized"]: @looped def s(loop, myextra): fail["Should not be reached because the definition is faulty."] # pragma: no cover - with test_raises(ValueError, "@looped_over: should detect invalid definition, no (loop, x, acc) parameters for loop body"): + with test_raises[ValueError, "@looped_over: should detect invalid definition, no (loop, x, acc) parameters for loop body"]: @looped_over(range(10), acc=()) def s(): fail["Should not be reached because the definition is faulty."] # pragma: no cover - with test_raises(ValueError, "@looped_over: should detect invalid definition, no acc parameter for loop body"): + with test_raises[ValueError, "@looped_over: should detect invalid definition, no acc parameter for loop body"]: @looped_over(range(10), acc=()) def s(loop, x): fail["Should not be reached because the definition is faulty."] # pragma: no cover - with test_raises(ValueError, "@looped_over: should detect invalid definition, extra parameter not initialized"): + with test_raises[ValueError, "@looped_over: should detect invalid definition, extra parameter not initialized"]: @looped_over(range(10), acc=()) def s(loop, x, acc, myextra): fail["Should not be reached because the definition is faulty."] # pragma: no cover @@ -327,7 +327,7 @@ def result(loop, brk, acc=0, i=0): return loop(acc + i, i + 1) # provide the additional parameters test[result == 45] - with test_raises(ValueError): + with test_raises[ValueError]: @breakably_looped def result(loop): # missing `brk` parameter pass # pragma: no cover @@ -363,12 +363,12 @@ def s(loop, x, acc, cnt, brk): return loop(acc + x) # pragma: no cover test[s == 0] - with test_raises(ValueError): + with test_raises[ValueError]: @breakably_looped_over(range(10), acc=0) def s(loop, x, acc): # missing `cnt` and `brk` parameters return loop(acc + x) # pragma: no cover - with test_raises(ValueError): + with test_raises[ValueError]: @breakably_looped_over(range(10), acc=0) def s(loop, x, acc, cnt): # missing `brk` parameter return loop(acc + x) # pragma: no cover diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index e743cc2b..798da7f3 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -180,11 +180,11 @@ def f(x): # note f takes only one arg # raise `TypeError`. def double(x): return 2 * x - with test_raises(TypeError, "leftover args should not be allowed by default"): + with test_raises[TypeError, "leftover args should not be allowed by default"]: curry(double, 2, "foo") # To disable the error, use this trick to explicitly state you want to do so: - with test("leftover args should be allowed with manually created surrounding context"): + with test["leftover args should be allowed with manually created surrounding context"]: with dyn.let(curry_context=["whatever"]): # any human-readable label is fine. # a `with test` can optionally return a value, which becomes the asserted expr. return curry(double, 2, "foo") == (4, "foo") diff --git a/unpythonic/tests/test_fup.py b/unpythonic/tests/test_fup.py index da9f64b6..335ca0b0 100644 --- a/unpythonic/tests/test_fup.py +++ b/unpythonic/tests/test_fup.py @@ -98,7 +98,7 @@ def runtests(): test[out == (2, 3, 2, 3, 2, 3, 42, 3, 2, 3)] with testset("error cases"): - with test_raises(IndexError, "should detect replacement sequence too short"): + with test_raises[IndexError, "should detect replacement sequence too short"]: tup = (1, 2, 3, 4, 5) out = fupdate(tup, slice(1, None, 2), (10,)) # need 2 items, have 1 diff --git a/unpythonic/tests/test_let.py b/unpythonic/tests/test_let.py index fa8dd9a1..496f95d3 100644 --- a/unpythonic/tests/test_let.py +++ b/unpythonic/tests/test_let.py @@ -151,7 +151,7 @@ def result(*, env): body=lambda e: e.set('y', 3)), "e.y should not be defined"] - with test_raises(AttributeError, "let environment should be final (should not be able to create new bindings in it inside the let body)"): + with test_raises[AttributeError, "let environment should be final (should not be able to create new bindings in it inside the let body)"]: @blet(x=1) def error1(*, env): env.y = 2 # error, cannot introduce new bindings into a let environment diff --git a/unpythonic/tests/test_lispylet.py b/unpythonic/tests/test_lispylet.py index cad4ed72..8d3ec2c6 100644 --- a/unpythonic/tests/test_lispylet.py +++ b/unpythonic/tests/test_lispylet.py @@ -142,7 +142,7 @@ def result(*, env): e.set('y', 3)), "e.y should not be defined"] - with test_raises(AttributeError, "let environment should be final (should not be able to create new bindings in it inside the let body)"): + with test_raises[AttributeError, "let environment should be final (should not be able to create new bindings in it inside the let body)"]: @blet((('x', 1),)) def error1(*, env): env.y = 2 # error, cannot introduce new bindings into a let environment diff --git a/unpythonic/tests/test_llist.py b/unpythonic/tests/test_llist.py index 053d3ac2..c6798cf1 100644 --- a/unpythonic/tests/test_llist.py +++ b/unpythonic/tests/test_llist.py @@ -22,7 +22,7 @@ def runtests(): test_raises[TypeError, car("sedan")] test_raises[TypeError, cdr("disc")] - with test_raises(TypeError, "cons cells should be immutable"): + with test_raises[TypeError, "cons cells should be immutable"]: c.car = 3 test[the[c == c]] diff --git a/unpythonic/tests/test_seq.py b/unpythonic/tests/test_seq.py index c556df63..5218a677 100644 --- a/unpythonic/tests/test_seq.py +++ b/unpythonic/tests/test_seq.py @@ -67,7 +67,7 @@ def runtests(): lambda x, y: (x * 2, y + 1)) test[(a, b) == (4, 3)] - with test_raises(TypeError, "should error when the curry context exits with args remaining"): + with test_raises[TypeError, "should error when the curry context exits with args remaining"]: a, b = pipec((1, 2), lambda x: x + 1, lambda x: x * 2) diff --git a/unpythonic/tests/test_tco.py b/unpythonic/tests/test_tco.py index e82aa97d..647dcb1a 100644 --- a/unpythonic/tests/test_tco.py +++ b/unpythonic/tests/test_tco.py @@ -81,7 +81,7 @@ def withec(ec): print("*** These error cases SHOULD PRINT A WARNING:", file=stderr) print("** Attempted jump into an inert data value:", file=stderr) - with test_raises(RuntimeError): + with test_raises[RuntimeError]: @trampolined def errorcase1(): return jump(42) From ba79b337fa90fbd076fbbfc6797bdbb11304b091 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:24:35 +0300 Subject: [PATCH 031/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 195a98c9..5c46b254 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -95,8 +95,6 @@ # TODO: `make_dynvar` needs to be better advertised in the docs. A workflow example would also be nice. -# TODO: Brackets: use "with test[...]" instead of "with test(...)" in the test modules - # TODO: Remove any unused `expander` kwargs from the macro interface # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. From 1a54a469198e696a4b779cb4780de419b13a8a53 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:26:54 +0300 Subject: [PATCH 032/832] update compatibility note --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2d191286..139fd2ab 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ None required. The officially supported language versions are **CPython 3.8** and **PyPy3 3.7**. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). -The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 7.3.4 (language version 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. +The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. ### Documentation From 30c42846c6ad234e160058a00242e52d43576dbf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:43:55 +0300 Subject: [PATCH 033/832] update changelog --- CHANGELOG.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 46a8a7e3..203b094b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,11 @@ This edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. -**Minimum Python version is now 3.6**. For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). +**Minimum Python language version is now 3.6**. For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). **New**: -- `with namedlambda` now understands the walrus operator, too. In `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) +- `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), and `call_cc[]` (for `with continuations`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. - `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. - Python 3.8 and 3.9 support added. @@ -14,25 +14,29 @@ This edition concentrates on upgrading our dependencies, namely the macro expand **Non-breaking changes**: - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. +- CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). +- CI: Test coverage improved to 94%. **Breaking changes**: -- Migrate to the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander; MacroPy support dropped. - - This facilitates future development of the macro parts of `unpythonic`. - - Macro arguments are now passed using brackets `macroname[args]` instead of parentheses. +- Migrate to the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander; **MacroPy support dropped**. This change facilitates future development of the macro parts of `unpythonic`. + - **Macro arguments are now passed using brackets** `macroname[args]` instead of parentheses. - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. - - If you already only need to run on Python 3.9 and later, please use brackets. We currently plan to eventually drop support for parentheses to pass macro arguments, when Python 3.9 becomes the minimum supported language version. + - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. - As a result of the new macro expander, macro test coverage should now be reported correctly. - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - - Any imports of these in user code should be modified to point to the new locations. + - **Any imports of these constructs in user code should be modified to point to the new locations.** + - Unlike `macropy`'s `Lazy`, our `Lazy` does not define `__call__`; instead, it defines the method `force`, which has the same effect (it computes if necessary, and then returns the value of the promise). - The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else it is available to be used as a regular variable. - `f[]` now respects nesting: an invocation of `f[]` will not descend into another nested `f[]`. - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import. Now you **must** import also the macro `f` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `f` is currently bound to. -- Rename the `curry` macro to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. -- The internal utility class `unpythonic.syntax.util.ASTMarker` has been renamed to `UnpythonicExpandedMacroMarker` to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. -- Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. -- Python 3.4 and 3.5 support dropped, as these language versions have reached end-of-life. +- **Rename the `curry` macro** to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. +- Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. + - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) + - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. +- Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. --- From 8d488741fdb8a20eae4da82c01b301289219c900 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:56:26 +0300 Subject: [PATCH 034/832] be more specific --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 139fd2ab..c36e436b 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f As of [7bb1198](https://github.com/Technologicat/unpythonic/commit/7bb1198605087f1dd7ca292e33afd53e5aa9721d), the initial porting effort of `unpythonic` to Python 3.8 and the new [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander is complete. In fact, if you want to play around with 0.15-pre, the code is already in `master`. -The codebase already fully works, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. +The codebase already fully works on all supported Python versions, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. For details, see [the 0.15 milestone](https://github.com/Technologicat/unpythonic/milestone/1). From 74957d53085bbdf0b970471c3303dbaec51b518d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 01:56:33 +0300 Subject: [PATCH 035/832] PyPy3 now has language version 3.7, too --- doc/readings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/readings.md b/doc/readings.md index d41a1866..c3a09676 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -61,7 +61,7 @@ The common denominator is programming. Some relate to language design, some to c - [Clean Code for Python](https://github.com/zedr/clean-code-python) - *Software engineering principles, from Robert C. Martin's book [Clean Code](https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882), adapted for Python.* -- [PyPy3](http://pypy.org/), fast, JIT-ing Python 3 that's mostly a drop-in replacement for CPython 3.6. Macro expanders (`macropy`, `mcpyrate`) work, too. +- [PyPy3](http://pypy.org/), fast, JIT-ing Python 3 that's mostly a drop-in replacement for CPythons 3.6 and 3.7. As of April 2021, support for 3.8 is in the works. Macro expanders (`macropy`, `mcpyrate`) work, too. - [Brython](https://brython.info/): Python 3 in the browser, as a replacement for JavaScript. - No separate compile step - the compiler is implemented in JS. Including a script tag of type text/python invokes it. From 03b19c7522f0c5cf1b15a850d62eec5dcc9f2486 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:02:56 +0300 Subject: [PATCH 036/832] fix borked links to unit tests in docs --- README.md | 4 ++-- doc/features.md | 12 ++++++------ doc/macros.md | 6 +++--- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index c36e436b..5b2d14de 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ This depends on the purpose of each feature, as well as ease-of-use consideratio ### Examples -Small, limited-space overview of the overall flavor. There's a lot more that doesn't fit here, especially in the pure-Python feature set. See the [full documentation](doc/features.md) and [unit tests](unpythonic/test/) for more examples. +Small, limited-space overview of the overall flavor. There's a lot more that doesn't fit here, especially in the pure-Python feature set. See the [full documentation](doc/features.md) and [unit tests](unpythonic/tests/) for more examples. #### Unpythonic in 30 seconds: Pure Python @@ -622,4 +622,4 @@ Thanks to [TUT](http://www.tut.fi/en/home) for letting me teach [RAK-19006 in sp Links to blog posts, online articles and papers on topics relevant in the context of `unpythonic` have been collected to [a separate document](doc/readings.md). -If you like both FP and numerics, we have [some examples](unpythonic/test/test_fpnumerics.py) based on various internet sources. +If you like both FP and numerics, we have [some examples](unpythonic/tests/test_fpnumerics.py) based on various internet sources. diff --git a/doc/features.md b/doc/features.md index c864e6d6..6d08e379 100644 --- a/doc/features.md +++ b/doc/features.md @@ -65,7 +65,7 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``ulp``: unit in last place](#ulp-unit-in-last-place) - [``async_raise``: inject an exception to another thread](#async_raise-inject-an-exception-to-another-thread) *(CPython only)* -For many examples, see [the unit tests](unpythonic/test/), the docstrings of the individual features, and this guide. +For many examples, see [the unit tests](unpythonic/tests/), the docstrings of the individual features, and this guide. *This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out-of-date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* @@ -334,7 +334,7 @@ print(tuple((k, dyn[k]) for k in dyn)) Finally, ``dyn`` supports membership testing as ``"x" in dyn``, ``"y" not in dyn``, where the string is the name of the dynvar whose presence is being tested. -For some more details, see [the unit tests](../unpythonic/test/test_dynassign.py). +For some more details, see [the unit tests](../unpythonic/tests/test_dynassign.py). ### Relation to similar features in Lisps @@ -2105,7 +2105,7 @@ Another question arises due to Python having builtin support for object persiste (Scenario: during second and later runs, a program first initializes, which causes the singleton instance to be created, just like during the first run of that program. Then the program loads state from a pickle file, containing (among other data) the state the singleton instance was in when the program previously shut down. In this scenario, considering the singleton, the data in the file is more relevant than the defaults the program initialization feeds in. Hence the default should be to replace the state of the existing singleton instance with the data from the pickle file.) -Our `Singleton` abstraction is the result of these pythonifications applied to the classic pattern. For more documentation and examples, see the unit tests in [`unpythonic/test/test_singleton.py`](../unpythonic/test/test_singleton.py). +Our `Singleton` abstraction is the result of these pythonifications applied to the classic pattern. For more documentation and examples, see the unit tests in [`unpythonic/tests/test_singleton.py`](../unpythonic/tests/test_singleton.py). **NOTE**: A related pattern is the *[Borg](http://code.activestate.com/recipes/66531-singleton-we-dont-need-no-stinkin-singleton-the-bo/)*, a.k.a. *Monostate*. [After considering the matter](https://github.com/Technologicat/unpythonic/issues/22), it was felt that in the context of Python, it offers no advantages over the singleton abstraction, while eliminating a useful feature: the singleton abstraction allows using the object identity check (`is`) to check if a name refers to the singleton instance. For this reason, `unpythonic` provides `Singleton`, but no `Borg`. If you feel this is unjust, please let me know - this decision can be revisited, if a situation in which a `Borg` is more appropriate than a `Singleton` comes up. @@ -2985,7 +2985,7 @@ The ``generic`` decorator allows creating multiple-dispatch generic functions wi We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. -For what kind of things can be done with this, see particularly the [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) example in [`unpythonic.test.test_dispatch`](../unpythonic/test/test_dispatch.py). +For what kind of things can be done with this, see particularly the [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) example in [`unpythonic.tests.test_dispatch`](../unpythonic/tests/test_dispatch.py). #### ``generic``: multiple dispatch with type annotation syntax @@ -3051,7 +3051,7 @@ assert gargle(42, 6.022e23, "hello") == "int, float, str" assert gargle(1, 2, 3) == "int" # as many as in the [int, float, str] case. Still resolves correctly. ``` -See [the unit tests](../unpythonic/test/test_dispatch.py) for more. For which features of the ``typing`` stdlib module are supported, see ``isoftype`` below. +See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the ``typing`` stdlib module are supported, see ``isoftype`` below. Inspired by the [multi-methods of CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) (the Common Lisp Object System), and the [generic functions of Julia](https://docs.julialang.org/en/v1/manual/methods/). @@ -3173,7 +3173,7 @@ assert isoftype(3.14, typing.SupportsRound) assert isoftype([1, 2, 3], typing.Sized) ``` -See [the unit tests](../unpythonic/test/test_typecheck.py) for more. +See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. **CAUTION**: Callables are just checked for being callable; no further analysis is done. Type-checking callables properly requires a much more complex type checker. diff --git a/doc/macros.md b/doc/macros.md index cc478234..1793877e 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1661,9 +1661,9 @@ If you want to turn coloring off (e.g. for redirecting stderr to a file), see th The following is an overview of the framework. For details, look at the docstrings of the various constructs in `unpythonic.test.fixtures` (which provides much of this), those of the test macros, and finally, the automated tests of `unpythonic` itself. -How to test code using conditions and restarts can be found in [`unpythonic.test.test_conditions`](../unpythonic/test/test_conditions.py). +How to test code using conditions and restarts can be found in [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py). -How to test macro utilities (e.g. syntax transformer functions that operate on ASTs) can be found in [`unpythonic.syntax.test.test_letdoutil`](../unpythonic/syntax/test/test_letdoutil.py). +How to test macro utilities (e.g. syntax transformer functions that operate on ASTs) can be found in [`unpythonic.syntax.tests.test_letdoutil`](../unpythonic/syntax/tests/test_letdoutil.py). #### Overview @@ -1721,7 +1721,7 @@ Additional tools for code using **conditions and restarts**: The `catch_signals` context manager controls the signal barrier of `with testset` and the `test` family of syntactic constructs. It is provided for writing tests for code that uses conditions and restarts. -Used as `with catch_signals(False)`, it disables the signal barrier. Within the dynamic extent of the block, an uncaught signal (in the sense of `unpythonic.conditions.signal` and its sisters) is not considered an error. This can be useful, because sometimes leaving a signal uncaught is the right thing to do. See [`unpythonic.test.test_conditions`](../unpythonic/test/test_conditions.py) for examples. +Used as `with catch_signals(False)`, it disables the signal barrier. Within the dynamic extent of the block, an uncaught signal (in the sense of `unpythonic.conditions.signal` and its sisters) is not considered an error. This can be useful, because sometimes leaving a signal uncaught is the right thing to do. See [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py) for examples. It can be nested. Used as `with catch_signals(True)`, it re-enables the barrier, if currently disabled. From 9d80f5777f282de2ee0300dd5caaed09f1152b3c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:03:09 +0300 Subject: [PATCH 037/832] we use mcpyrate now --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 1793877e..31ca8442 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2024,7 +2024,7 @@ This Elisp snippet can be used to add syntax highlighting for keywords specific ```elisp (defun my/unpythonic-syntax-highlight-setup () - "Set up additional syntax highlighting for `unpythonic.syntax' and MacroPy in Python mode." + "Set up additional syntax highlighting for `unpythonic.syntax' and `mcpyrate` in Python mode." ;; adapted from code in dash.el (let ((new-keywords '("test" "test_raises" "test_signals" "fail" "the" "error" "warn" ; both testing macros and condition signaling protocols From dc7aa7da29e02f197dcd2670d5ea96714893d6e5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:07:12 +0300 Subject: [PATCH 038/832] revise recommendation for `locals` arg --- unpythonic/net/server.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/net/server.py b/unpythonic/net/server.py index b8b07f1e..19844ec6 100644 --- a/unpythonic/net/server.py +++ b/unpythonic/net/server.py @@ -32,14 +32,14 @@ With that out of the way, to enable the server in your app:: from unpythonic.net import server - server.start(locals=globals()) + server.start(locals={}) The `locals=...` argument sets the top-level namespace for variables for use by the REPL. It is shared between REPL sessions. Using `locals=globals()` makes the REPL directly use the calling module's top-level scope. If you want a clean environment, where you must access any -modules through `sys.modules`, use `locals={}`. +modules through `sys.modules`, use `locals={}` (recommended). To connect to a running REPL server (with tab completion and Ctrl+C support):: From 4a1b83d8686503b27cd78c2ea466898b84fd4356 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:09:05 +0300 Subject: [PATCH 039/832] update etymology in docstring --- unpythonic/net/server.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/unpythonic/net/server.py b/unpythonic/net/server.py index 19844ec6..e1cd14ec 100644 --- a/unpythonic/net/server.py +++ b/unpythonic/net/server.py @@ -104,6 +104,11 @@ The `socketserverREPL` package uses the same default, and actually its `repl_tool.py` can talk to this server (but doesn't currently feature remote tab completion). + +The default port for the control channel is 8128, because it's for +*completing* things, and https://en.wikipedia.org/wiki/Perfect_number +This is the first one above 1024, and was already known to Nicomachus +around 100 CE. """ # TODO: use logging module instead of server-side print From 238aef53b84bbae2950e8ffa26627a84d852d81e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:26:22 +0300 Subject: [PATCH 040/832] exorcise some more outdated mentions of MacroPy --- unpythonic/syntax/__init__.py | 6 +++--- unpythonic/syntax/letdo.py | 16 ---------------- unpythonic/syntax/letdoutil.py | 2 +- unpythonic/syntax/tests/test_conts.py | 2 +- unpythonic/syntax/tests/test_lambdatools.py | 1 - 5 files changed, 5 insertions(+), 22 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 5c46b254..54ba4e65 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -1835,9 +1835,9 @@ def dbg(tree, *, args, syntax, expander, **kw): # noqa: F811 ``xxx`` is the original line number before macro expansion, if available in the AST node of the expression, otherwise ``None``. (Some macros might - not care about inserting line numbers, because MacroPy fixes any missing - line numbers at the end; this is why it might be missing at some locations - in any specific macro-enabled program.) + not care about inserting line numbers, because `mcpyrate` fixes any missing + line numbers in a postprocess step; this is why it might be missing at some + locations in any specific macro-enabled program.) A default implementation of the debug printer is provided and automatically assigned as the default value for `dyn.dbgprint_expr`. diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 12e84eb3..aa5334bb 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -75,22 +75,6 @@ def _letimpl(bindings, body, mode): # this case, because our syntaxes always require at least one binding. # So this check is here just to protect against use with no bindings directly # from other syntax transformers, which in theory could attempt anything. - # - # TODO: update this comment for mcpyrate - # The reason the macro layer never calls us with no bindings is technical. - # In the macro interface, with no bindings, the macro's `args` are `()` - # whether it was invoked as `let()[...]` or just `let[...]`. Thus, - # there is no way to distinguish, in the macro layer, between these - # two. We can't use `UnexpandedLetView` to do the dirty work of AST - # analysis, because MacroPy does too much automatically: in the macro - # layer, `tree` is only the part inside the brackets. So we really - # can't see whether the part outside the brackets was a Call with no - # arguments, or just a Name - both cases get treated exactly the same, - # as a macro invocation with empty `args`. - # - # The latter form, `let[...]`, is used by the haskelly syntax - # `let[(...) in ...]`, `let[..., where(...)]` - and in these cases, - # both the bindings and the body reside inside the brackets. return body # pragma: no cover bindings = dyn._macro_expander.visit(bindings) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 101d3934..3e4438b0 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -625,7 +625,7 @@ def _setbindings(self, newbindings): # update name in the namelambda(...) thev.func.args[0] = Constant(value=f"letrec_binding_{newk_string}") # Python 3.8+: ast.Constant # Macro-generated nodes may be missing source location information, - # in which case we let MacroPy fix it later. + # in which case we let `mcpyrate` fix it later. # This is mainly an issue for the unit tests of this module, which macro-generate the "old" data. if hasattr(oldb, "lineno") and hasattr(oldb, "col_offset"): newelts.append(Tuple(elts=[newk, thev], lineno=oldb.lineno, col_offset=oldb.col_offset)) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 966f6f53..e19af79b 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -478,7 +478,7 @@ def pt(maxn): test[out == pts] with testset("integration with autoreturn and autocurry simultaneously"): - with autocurry: # major slowdown, but works; must be in a separate "with" # TODO: why separate? https://github.com/azazel75/macropy/issues/21 + with autocurry: # major slowdown, but works with autoreturn, continuations: stack = [] def amb(lst, cc): # noqa: F811, the previous one is no longer used. diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index c884c5c7..b68ba445 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -140,7 +140,6 @@ def decorated(*args, **kwargs): test[f5.__name__ == "f5"] # also autocurry with a lambda as the last argument is recognized - # TODO: fix MacroPy #21 properly; https://github.com/azazel75/macropy/issues/21 with testset("namedlambda, naming an autocurried last arg"): with namedlambda: with autocurry: From d93c1ec91691041c838d6657dc62692021ead41e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:31:42 +0300 Subject: [PATCH 041/832] dlet should also expand body and args itself --- unpythonic/syntax/__init__.py | 36 ++++++++++++----------------------- unpythonic/syntax/letdo.py | 2 ++ 2 files changed, 14 insertions(+), 24 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 54ba4e65..6681fa6c 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -534,10 +534,8 @@ def count(): if syntax != "decorator": raise SyntaxError("dlet is a decorator macro only") - args = expander.visit(args) - tree = expander.visit(tree) - - return _destructure_and_apply_let(tree, args, expander, _dlet) + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _dlet) @parametricmacro def dletseq(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -557,10 +555,8 @@ def g(a): if syntax != "decorator": raise SyntaxError("dletseq is a decorator macro only") - args = expander.visit(args) - tree = expander.visit(tree) - - return _destructure_and_apply_let(tree, args, expander, _dletseq) + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _dletseq) @parametricmacro def dletrec(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -580,10 +576,8 @@ def f(x): if syntax != "decorator": raise SyntaxError("dletrec is a decorator macro only") - args = expander.visit(args) - tree = expander.visit(tree) - - return _destructure_and_apply_let(tree, args, expander, _dletrec) + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _dletrec) @parametricmacro def blet(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -599,10 +593,8 @@ def result(): if syntax != "decorator": raise SyntaxError("blet is a decorator macro only") - args = expander.visit(args) - tree = expander.visit(tree) - - return _destructure_and_apply_let(tree, args, expander, _blet) + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _blet) @parametricmacro def bletseq(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -620,10 +612,8 @@ def result(): if syntax != "decorator": raise SyntaxError("bletseq is a decorator macro only") - args = expander.visit(args) - tree = expander.visit(tree) - - return _destructure_and_apply_let(tree, args, expander, _bletseq) + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _bletseq) @parametricmacro def bletrec(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -652,10 +642,8 @@ def result(): if syntax != "decorator": raise SyntaxError("bletrec is a decorator macro only") - args = expander.visit(args) - tree = expander.visit(tree) - - return _destructure_and_apply_let(tree, args, expander, _bletrec) + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _bletrec) # ----------------------------------------------------------------------------- # Imperative code in expression position. diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index aa5334bb..038afeb9 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -203,11 +203,13 @@ def _dletimpl(bindings, body, mode, kind): assert kind in ("decorate", "call") if type(body) not in (FunctionDef, AsyncFunctionDef): raise SyntaxError("Expected a function definition to decorate") # pragma: no cover + body = dyn._macro_expander.visit(body) if not bindings: # Similarly as above, this cannot trigger from the macro layer no # matter what that layer does. This is here to optimize away a `dlet` # with no bindings, when used directly from other syntax transformers. return body # pragma: no cover + bindings = dyn._macro_expander.visit(bindings) names, values = zip(*[b.elts for b in bindings]) # --> (k1, ..., kn), (v1, ..., vn) names = [k.id for k in names] # any duplicates will be caught by env at run-time From bff7df43f05546bffa863abe6cc0b037392b1dc0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:31:57 +0300 Subject: [PATCH 042/832] parameter naming --- unpythonic/syntax/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 6681fa6c..907ee282 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -497,14 +497,14 @@ def letrec(tree, *, args, syntax, expander, **kw): # noqa: F811 # node (so we could see the exact original syntax). # # allow_call_in_name_position: used by let_syntax to allow template definitions. -def _destructure_and_apply_let(tree, args, macro_expander, let_expander_function, allow_call_in_name_position=False): +def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, allow_call_in_name_position=False): with dyn.let(_macro_expander=macro_expander): # implicit do (extra bracket notation) needs this. if args: bs = _canonize_bindings(args, allow_call_in_name_position=allow_call_in_name_position) - return let_expander_function(bindings=bs, body=tree) + return let_transformer(bindings=bs, body=tree) # haskelly syntax, let[(...) in ...], let[..., where(...)] view = _UnexpandedLetView(tree) # note "tree" here is only the part inside the brackets - return let_expander_function(bindings=view.bindings, body=view.body) + return let_transformer(bindings=view.bindings, body=view.body) # ----------------------------------------------------------------------------- # Decorator versions, for "let over def". From cf0f639e563841d24e3ce2ea582a426ff6764ba6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:35:58 +0300 Subject: [PATCH 043/832] exorcise unused parameters in macro interface --- unpythonic/syntax/__init__.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 907ee282..c390f43f 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -648,7 +648,7 @@ def result(): # ----------------------------------------------------------------------------- # Imperative code in expression position. -def local(tree, *, syntax, invocation, **kw): # noqa: F811 +def local(tree, *, syntax, **kw): # noqa: F811 """[syntax] Declare a local name in a "do". Usage:: @@ -675,7 +675,7 @@ def local(tree, *, syntax, invocation, **kw): # noqa: F811 raise SyntaxError("local is an expr macro only") # pragma: no cover return _local(tree) -def delete(tree, *, syntax, invocation, **kw): # noqa: F811 +def delete(tree, *, syntax, **kw): # noqa: F811 """[syntax] Delete a previously declared local name in a "do". Usage:: @@ -1203,7 +1203,7 @@ def foo(n): # ----------------------------------------------------------------------------- -def autoreturn(tree, *, syntax, expander, **kw): # noqa: F811 +def autoreturn(tree, *, syntax, **kw): # noqa: F811 """[syntax, block] Implicit "return" in tail position, like in Lisps. Each ``def`` function definition lexically within the ``with autoreturn`` @@ -1775,7 +1775,7 @@ def result(ec): # ----------------------------------------------------------------------------- @parametricmacro -def nb(tree, *, args, syntax, expander, **kw): # noqa: F811 +def nb(tree, *, args, syntax, **kw): # noqa: F811 """[syntax, block] Ultralight math notebook. Auto-print top-level expressions, auto-assign last result as _. @@ -2245,7 +2245,7 @@ def lazy(tree, *, syntax, **kw): # noqa: F811 # Expand outside in. Ordering shouldn't matter here. return _lazy(tree) -def lazyrec(tree, *, syntax, expander, **kw): # noqa: F811 +def lazyrec(tree, *, syntax, **kw): # noqa: F811 """[syntax, expr] Delay items in a container literal, recursively. Essentially, this distributes ``lazy[]`` into the items inside a literal @@ -2290,7 +2290,7 @@ def lazyrec(tree, *, syntax, expander, **kw): # noqa: F811 # ----------------------------------------------------------------------------- -def prefix(tree, *, syntax, expander, **kw): # noqa: F811 +def prefix(tree, *, syntax, **kw): # noqa: F811 """[syntax, block] Write Python like Lisp: the first item is the operator. Example:: From 5c5d6c0179f8bc8a02b09525c4e60042df32e141 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:36:43 +0300 Subject: [PATCH 044/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index c390f43f..bac68554 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -95,8 +95,6 @@ # TODO: `make_dynvar` needs to be better advertised in the docs. A workflow example would also be nice. -# TODO: Remove any unused `expander` kwargs from the macro interface - # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. # TODO: Test the q[t[...]] implementation in do0[] From 205a5149a5ba05bbaa5b63aaf12b2c78ec899881 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 02:38:03 +0300 Subject: [PATCH 045/832] shorten --- unpythonic/syntax/__init__.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index bac68554..7bd70ac3 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -244,10 +244,7 @@ def autoref(tree, *, args, syntax, expander, **kw): # noqa: F811 if not args: raise SyntaxError("autoref requires an argument, the object to be auto-referenced") - if "optional_vars" in kw: - target = kw["optional_vars"] - else: - target = None + target = kw.get("optional_vars", None) tree = expander.visit(tree) From 1fb686b4495f1086c06c0a345688c8960d42eba7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 03:51:11 +0300 Subject: [PATCH 046/832] maintainability: reorganize macro interfaces Since `mcpyrate` can re-export macros, the macro interfaces now live in the individual modules under `unpythonic.syntax`, in the same place as the syntax transformers for those macros. This also helps keep the docstrings in the same place. No more monolithic `__init__` module, hooray! --- unpythonic/syntax/__init__.py | 2577 +---------------------------- unpythonic/syntax/autocurry.py | 57 +- unpythonic/syntax/autoref.py | 54 +- unpythonic/syntax/dbg.py | 102 +- unpythonic/syntax/forall.py | 10 +- unpythonic/syntax/ifexprs.py | 73 +- unpythonic/syntax/lambdatools.py | 222 ++- unpythonic/syntax/lazify.py | 408 ++++- unpythonic/syntax/letdo.py | 527 +++++- unpythonic/syntax/letsyntax.py | 158 +- unpythonic/syntax/nb.py | 31 +- unpythonic/syntax/prefix.py | 129 +- unpythonic/syntax/tailtools.py | 669 +++++++- unpythonic/syntax/testingtools.py | 559 +++++-- 14 files changed, 2782 insertions(+), 2794 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7bd70ac3..eb287ad2 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -4,11 +4,7 @@ Requires `mcpyrate`. """ -from mcpyrate import parametricmacro -from mcpyrate.expander import MacroExpander -from mcpyrate.utils import extract_bindings - -from ..dynassign import make_dynvar, dyn +from ..dynassign import make_dynvar # -------------------------------------------------------------------------------- # This module contains the macro interface and docstrings; the submodules @@ -100,9 +96,6 @@ # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. -# TODO: With `mcpyrate`, we could move the macro interface functions to -# TODO: the submodules, and have just re-exports here. - # TODO: macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" # TODO: Some macros look up others; convert lookups to mcpyrate style (accounting for as-imports) @@ -147,41 +140,29 @@ # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... -# Syntax transformers and internal utilities -from .autoref import autoref as _autoref -from .autocurry import autocurry as _autocurry -from .dbg import dbg_block as _dbg_block, dbg_expr as _dbg_expr -from .forall import forall as _forall -from .ifexprs import aif as _aif, cond as _cond -from .lambdatools import (multilambda as _multilambda, - namedlambda as _namedlambda, - f as _f, - envify as _envify) -from .lazify import lazy as _lazy, lazify as _lazify, lazyrec as _lazyrec -from .letdo import (local as _local, delete as _delete, - do as _do, do0 as _do0, - let as _let, letseq as _letseq, letrec as _letrec, - dlet as _dlet, dletseq as _dletseq, dletrec as _dletrec, - blet as _blet, bletseq as _bletseq, bletrec as _bletrec) -from .letdoutil import (UnexpandedLetView as _UnexpandedLetView, - canonize_bindings as _canonize_bindings) -from .letsyntax import (let_syntax_expr as _let_syntax_expr, - let_syntax_block as _let_syntax_block) -from .nb import nb as _nb -from .prefix import prefix as _prefix -from .tailtools import (autoreturn as _autoreturn, tco as _tco, - continuations as _continuations) -from .testingtools import (test_expr as _test_expr, - test_expr_signals as _test_expr_signals, - test_expr_raises as _test_expr_raises, - test_block as _test_block, - test_block_signals as _test_block_signals, - test_block_raises as _test_block_raises, - fail_expr as _fail_expr, - error_expr as _error_expr, - warn_expr as _warn_expr) - -# Re-exports (for client code that uses us) +# Re-exports - macro interfaces +from .autocurry import autocurry # noqa: F401 +from .autoref import autoref # noqa: F401 +from .dbg import dbg # noqa: F401 +from .forall import forall # noqa: F401 +from .ifexprs import aif, cond # noqa: F401 +from .lambdatools import multilambda, namedlambda, f, quicklambda, envify # noqa: F401 +from .lazify import lazy, lazyrec, lazify # noqa: F401 +from .letdo import (let, letseq, letrec, # noqa: F401 + dlet, dletseq, dletrec, + blet, bletseq, bletrec, + local, delete, do, do0) +from .letsyntax import let_syntax, abbrev # noqa: F401 +from .nb import nb # noqa: F401 +from .prefix import prefix # noqa: F401 +from .tailtools import (autoreturn, # noqa: F401 + tco, + continuations, call_cc) +from .testingtools import (the, test, # noqa: F401 + test_signals, test_raises, + fail, error, warn) + +# Re-exports - regular code from .dbg import dbgprint_block, dbgprint_expr # noqa: F401, re-export for re-use in a decorated variant. from .forall import insist, deny # noqa: F401 from .ifexprs import it # noqa: F401 @@ -189,2519 +170,9 @@ from .lazify import force, force1 # noqa: F401 from .letsyntax import block, expr # noqa: F401 from .prefix import q, u, kw # noqa: F401 # TODO: bad names, `mcpyrate` uses them too. -from .tailtools import call_cc # noqa: F401 -from .testingtools import the # noqa: F401 # We use `dyn` to pass the `expander` parameter to the macro implementations. class _NoExpander: def visit(self, tree): raise NotImplementedError("Macro expander instance has not been set in `dyn`.") make_dynvar(_macro_expander=_NoExpander()) - -# ----------------------------------------------------------------------------- - -# The "kw" we have here is the parameter from mcpyrate; the "kw" we export (that -# flake8 thinks conflicts with this) is the runtime stub for our `prefix` macro. -@parametricmacro -def autoref(tree, *, args, syntax, expander, **kw): # noqa: F811 - """Implicitly reference attributes of an object. - - Example:: - - e = env(a=1, b=2) - c = 3 - with autoref[e]: - a - b - c - - The transformation is applied in ``Load`` context only. ``Store`` and ``Del`` - are not redirected. - - Useful e.g. with the ``.mat`` file loader of SciPy. - - **CAUTION**: `autoref` is essentially the `with` construct of JavaScript - (which is completely different from Python's meaning of `with`), which is - nowadays deprecated. See: - - https://www.ecma-international.org/ecma-262/6.0/#sec-with-statement - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/with - https://2ality.com/2011/06/with-statement.html - - **CAUTION**: The auto-reference `with` construct was deprecated in JavaScript - **for security reasons**. Since the autoref'd object **will hijack all name - lookups**, use `with autoref` only with an object you trust! - - **CAUTION**: `with autoref` also complicates static code analysis or makes it - outright infeasible, for the same reason. It is impossible to statically know - whether something that looks like a bare name in the source code is actually - a true bare name, or a reference to an attribute of the autoref'd object. - That status can also change at any time, since the lookup is dynamic, and - attributes can be added and removed dynamically. - """ - if syntax != "block": - raise SyntaxError("autoref is a block macro only") - if not args: - raise SyntaxError("autoref requires an argument, the object to be auto-referenced") - - target = kw.get("optional_vars", None) - - tree = expander.visit(tree) - - return _autoref(block_body=tree, args=args, asname=target) - -# ----------------------------------------------------------------------------- - -def aif(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Anaphoric if. - - Usage:: - - aif[test, then, otherwise] - - aif[[pre, ..., test], - [post_true, ..., then], # "then" branch - [post_false, ..., otherwise]] # "otherwise" branch - - Inside the ``then`` and ``otherwise`` branches, the magic identifier ``it`` - (which is always named literally ``it``) refers to the value of ``test``. - - This expands into a ``let`` and an expression-form ``if``. - - Each part may consist of multiple expressions by using brackets around it; - those brackets create a `do` environment (see `unpythonic.syntax.do`). - - To represent a single expression that is a literal list, use extra - brackets: ``[[1, 2, 3]]``. - """ - if syntax != "expr": - raise SyntaxError("aif is an expr macro only") - - # Expand outside-in, but the implicit do[] needs the expander. - with dyn.let(_macro_expander=expander): - return _aif(tree) - -def cond(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Lispy cond; like "a if p else b", but has "elif". - - Usage:: - - cond[test1, then1, - test2, then2, - ... - otherwise] - - cond[[pre1, ..., test1], [post1, ..., then1], - [pre2, ..., test2], [post2, ..., then2], - ... - [postn, ..., otherwise]] - - This allows human-readable multi-branch conditionals in an expression position. - - Each part may consist of multiple expressions by using brackets around it; - those brackets create a `do` environment (see `unpythonic.syntax.do`). - - To represent a single expression that is a literal list, use extra - brackets: ``[[1, 2, 3]]``. - """ - if syntax != "expr": - raise SyntaxError("cond is an expr macro only") - - # Expand outside-in, but the implicit do[] needs the expander. - with dyn.let(_macro_expander=expander): - return _cond(tree) - -# ----------------------------------------------------------------------------- - -def autocurry(tree, *, syntax, expander, **kw): # technically a list of trees, the body of the with block # noqa: F811 - """[syntax, block] Automatic currying. - - Usage:: - - from unpythonic.syntax import macros, autocurry - - with autocurry: - ... - - All **function calls** and **function definitions** (``def``, ``lambda``) - *lexically* inside the ``with autocurry`` block are automatically curried. - - **CAUTION**: Some builtins are uninspectable or may report their arities - incorrectly; in those cases, ``curry`` may fail, occasionally in mysterious - ways. - - The function ``unpythonic.arity.arities``, which ``unpythonic.fun.curry`` - internally uses, has a workaround for the inspectability problems of all - builtins in the top-level namespace (as of Python 3.7), but e.g. methods - of builtin types are not handled. - - Lexically inside a ``with autocurry`` block, the auto-curried function calls - will skip the curry if the function is uninspectable, instead of raising - ``TypeError`` as usual. - - Example:: - - from unpythonic.syntax import macros, autocurry - from unpythonic import foldr, composerc as compose, cons, nil, ll - - with autocurry: - def add3(a, b, c): - return a + b + c - assert add3(1)(2)(3) == 6 - assert add3(1, 2)(3) == 6 - assert add3(1)(2, 3) == 6 - assert add3(1, 2, 3) == 6 - - mymap = lambda f: foldr(compose(cons, f), nil) - double = lambda x: 2 * x - assert mymap(double, ll(1, 2, 3)) == ll(2, 4, 6) - - # The definition was auto-curried, so this works here too. - assert add3(1)(2)(3) == 6 - """ - if syntax != "block": - raise SyntaxError("autocurry is a block macro only") - - tree = expander.visit(tree) - - return _autocurry(block_body=tree) - -# ----------------------------------------------------------------------------- - -@parametricmacro -def let(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Introduce expression-local variables. - - This is sugar on top of ``unpythonic.lispylet.let``. - - Usage:: - - let[(k0, v0), ...][body] - let[(k0, v0), ...][[body0, ...]] - - where ``body`` is an expression. The names bound by ``let`` are local; - they are available in ``body``, and do not exist outside ``body``. - - Alternative haskelly syntax is also available:: - - let[((k0, v0), ...) in body] - let[((k0, v0), ...) in [body0, ...]] - let[body, where((k0, v0), ...)] - let[[body0, ...], where((k0, v0), ...)] - - For a body with multiple expressions, use an extra set of brackets, - as shown above. This inserts a ``do``. Only the outermost extra brackets - are interpreted specially; all others in the bodies are interpreted - as usual, as lists. - - Note that in the haskelly syntax, the extra brackets for a multi-expression - body should enclose only the ``body`` part. - - Each ``name`` in the same ``let`` must be unique. - - Assignment to let-bound variables is supported with syntax such as ``x << 42``. - This is an expression, performing the assignment, and returning the new value. - - In a multiple-expression body, also an internal definition context exists - for local variables that are not part of the ``let``; see ``do`` for details. - - Technical points: - - - In reality, the let-bound variables live in an ``unpythonic.env``. - This macro performs the magic to make them look (and pretty much behave) - like lexical variables. - - - Compared to ``unpythonic.lispylet.let``, the macro version needs no quotes - around variable names in bindings. - - - The body is automatically wrapped in a ``lambda e: ...``. - - - For all ``x`` in bindings, the macro transforms lookups ``x --> e.x``. - - - Lexical scoping is respected (so ``let`` constructs can be nested) - by actually using a unique name (gensym) instead of just ``e``. - - - In the case of a multiple-expression body, the ``do`` transformation - is applied first to ``[body0, ...]``, and the result becomes ``body``. - """ - if syntax != "expr": - raise SyntaxError("let is an expr macro only") - - # The `let[]` family of macros expands inside out. - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _let) - -@parametricmacro -def letseq(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Let with sequential binding (like Scheme/Racket let*). - - Like ``let``, but bindings take effect sequentially. Later bindings - shadow earlier ones if the same name is used multiple times. - - Expands to nested ``let`` expressions. - """ - if syntax != "expr": - raise SyntaxError("letseq is an expr macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _letseq) - -@parametricmacro -def letrec(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Let with mutually recursive binding. - - Like ``let``, but bindings can see other bindings in the same ``letrec``. - - Each ``name`` in the same ``letrec`` must be unique. - - The definitions are processed sequentially, left to right. A definition - may refer to any previous definition. If ``value`` is callable (lambda), - it may refer to any definition, including later ones. - - This is useful for locally defining mutually recursive functions. - """ - if syntax != "expr": - raise SyntaxError("letrec is an expr macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _letrec) - -# NOTE: At the macro interface, the invocations `let()[...]` (empty args) -# and `let[...]` (no args) were indistinguishable in MacroPy. This was a -# problem, because it might be that the user wrote the body but simply -# forgot to put anything in the parentheses. (There's `do[]` if you need -# a `let` without making any bindings.) -# -# In `mcpyrate`, `let()[...]` is a syntax error. The preferred syntax, -# when using macro arguments, is `let[...][...]`. When this is not -# possible (in decorator position up to Python 3.8), then `let(...)[...]` -# is acceptable. But empty brackets/parentheses are not accepted. Thus, -# we will have an empty `args` list only when there are no brackets/parentheses -# for the macro arguments part. -# -# So when `args` is empty, this function assumes haskelly let syntax -# `let[(...) in ...]` or `let[..., where(...)]`. In these cases, -# both the bindings and the body reside inside the brackets (i.e., -# in the AST contained in the `tree` argument). -# -# Since the brackets/parentheses must be deleted when no macro arguments -# are given, this is now the correct assumption to make. -# -# But note that if needed elsewhere, `mcpyrate` has the `invocation` kwarg -# in the macro interface that gives a copy of the whole macro invocation -# node (so we could see the exact original syntax). -# -# allow_call_in_name_position: used by let_syntax to allow template definitions. -def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, allow_call_in_name_position=False): - with dyn.let(_macro_expander=macro_expander): # implicit do (extra bracket notation) needs this. - if args: - bs = _canonize_bindings(args, allow_call_in_name_position=allow_call_in_name_position) - return let_transformer(bindings=bs, body=tree) - # haskelly syntax, let[(...) in ...], let[..., where(...)] - view = _UnexpandedLetView(tree) # note "tree" here is only the part inside the brackets - return let_transformer(bindings=view.bindings, body=view.body) - -# ----------------------------------------------------------------------------- -# Decorator versions, for "let over def". - -@parametricmacro -def dlet(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, decorator] Decorator version of let, for 'let over def'. - - Example:: - - @dlet[(x, 0)] - def count(): - x << x + 1 - return x - assert count() == 1 - assert count() == 2 - - **CAUTION**: function arguments, local variables, and names declared as - ``global`` or ``nonlocal`` in a given lexical scope shadow names from the - ``let`` environment *for the entirety of that lexical scope*. (This is - modeled after Python's standard scoping rules.) - - **CAUTION**: assignment to the let environment is ``name << value``; - the regular syntax ``name = value`` creates a local variable in the - lexical scope of the ``def``. - """ - if syntax != "decorator": - raise SyntaxError("dlet is a decorator macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _dlet) - -@parametricmacro -def dletseq(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, decorator] Decorator version of letseq, for 'letseq over def'. - - Expands to nested function definitions, each with one ``dlet`` decorator. - - Example:: - - @dletseq[(x, 1), - (x, x+1), - (x, x+2)] - def g(a): - return a + x - assert g(10) == 14 - """ - if syntax != "decorator": - raise SyntaxError("dletseq is a decorator macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _dletseq) - -@parametricmacro -def dletrec(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, decorator] Decorator version of letrec, for 'letrec over def'. - - Example:: - - @dletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))] - def f(x): - return evenp(x) - assert f(42) is True - assert f(23) is False - - Same cautions apply as to ``dlet``. - """ - if syntax != "decorator": - raise SyntaxError("dletrec is a decorator macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _dletrec) - -@parametricmacro -def blet(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, decorator] def --> let block. - - Example:: - - @blet[(x, 21)] - def result(): - return 2*x - assert result == 42 - """ - if syntax != "decorator": - raise SyntaxError("blet is a decorator macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _blet) - -@parametricmacro -def bletseq(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, decorator] def --> letseq block. - - Example:: - - @bletseq[(x, 1), - (x, x+1), - (x, x+2)] - def result(): - return x - assert result == 4 - """ - if syntax != "decorator": - raise SyntaxError("bletseq is a decorator macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _bletseq) - -@parametricmacro -def bletrec(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, decorator] def --> letrec block. - - Example:: - - @bletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))] - def result(): - return evenp(42) - assert result is True - - Because names inside a ``def`` have mutually recursive scope, - an almost equivalent pure Python solution (no macros) is:: - - from unpythonic.misc import call - - @call - def result(): - evenp = lambda x: (x == 0) or oddp(x - 1) - oddp = lambda x: (x != 0) and evenp(x - 1) - return evenp(42) - assert result is True - """ - if syntax != "decorator": - raise SyntaxError("bletrec is a decorator macro only") - - with dyn.let(_macro_expander=expander): - return _destructure_and_apply_let(tree, args, expander, _bletrec) - -# ----------------------------------------------------------------------------- -# Imperative code in expression position. - -def local(tree, *, syntax, **kw): # noqa: F811 - """[syntax] Declare a local name in a "do". - - Usage:: - - local[name << value] - - Only meaningful in a ``do[...]``, ``do0[...]``, or an implicit ``do`` - (extra bracket syntax). - - The declaration takes effect starting from next item in the ``do``, i.e. - the item that comes after the ``local[]``. It will not shadow nonlocal - variables of the same name in any earlier items of the same ``do``, and - in the item making the definition, the old bindings are still in effect - on the RHS. - - This means that if you want, you can declare a local ``x`` that takes its - initial value from a nonlocal ``x``, by ``local[x << x]``. Here the ``x`` - on the RHS is the nonlocal one (since the declaration has not yet taken - effect), and the ``x`` on the LHS is the name given to the new local variable - that only exists inside the ``do``. Any references to ``x`` in any further - items in the same ``do`` will point to the local ``x``. - """ - if syntax != "expr": - raise SyntaxError("local is an expr macro only") # pragma: no cover - return _local(tree) - -def delete(tree, *, syntax, **kw): # noqa: F811 - """[syntax] Delete a previously declared local name in a "do". - - Usage:: - - delete[name] - - Only meaningful in a ``do[...]``, ``do0[...]``, or an implicit ``do`` - (extra bracket syntax). - - The deletion takes effect starting from the next item; hence, the - deleted local variable will no longer shadow nonlocal variables of - the same name in any later items of the same `do`. - - Note ``do[]`` supports local variable deletion, but the ``let[]`` - constructs don't, by design. - """ - if syntax != "expr": - raise SyntaxError("delete is an expr macro only") # pragma: no cover - return _delete(tree) - -def do(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Stuff imperative code into an expression position. - - Return value is the value of the last expression inside the ``do``. - See also ``do0``. - - Usage:: - - do[body0, ...] - - Example:: - - do[local[x << 42], - print(x), - x << 23, - x] - - This is sugar on top of ``unpythonic.seq.do``, but with some extra features. - - - To declare and initialize a local name, use ``local[name << value]``. - - The operator ``local`` is syntax, not really a function, and it - only exists inside a ``do``. There is also an operator ``delete`` - to delete a previously declared local name in the ``do``. - - Both ``local`` and ``delete``, if used, should be imported as macros. - - - By design, there is no way to create an uninitialized variable; - a value must be given at declaration time. Just use ``None`` - as an explicit "no value" if needed. - - - Names declared within the same ``do`` must be unique. Re-declaring - the same name is an expansion-time error. - - - To assign to an already declared local name, use ``name << value``. - - **local name declarations** - - A ``local`` declaration comes into effect in the expression following - the one where it appears. Thus:: - - result = [] - let((lst, []))[do[result.append(lst), # the let "lst" - local[lst << lst + [1]], # LHS: do "lst", RHS: let "lst" - result.append(lst)]] # the do "lst" - assert result == [[], [1]] - - **Syntactic ambiguity** - - These two cases cannot be syntactically distinguished: - - - Just one body expression, which is a literal tuple or list, - - - Multiple body expressions, represented as a literal tuple or list. - - ``do`` always uses the latter interpretation. - - Whenever there are multiple expressions in the body, the ambiguity does not - arise, because then the distinction between the sequence of expressions itself - and its items is clear. - - Examples:: - - do[1, 2, 3] # --> tuple, 3 - do[(1, 2, 3)] # --> tuple, 3 (since in Python, the comma creates tuples; - # parentheses are only used for disambiguation) - do[[1, 2, 3]] # --> list, 3 - do[[[1, 2, 3]]] # --> list containing a list, [1, 2, 3] - do[([1, 2, 3],)] # --> tuple containing a list, [1, 2, 3] - do[[1, 2, 3],] # --> tuple containing a list, [1, 2, 3] - do[[(1, 2, 3)]] # --> list containing a tuple, (1, 2, 3) - do[((1, 2, 3),)] # --> tuple containing a tuple, (1, 2, 3) - do[(1, 2, 3),] # --> tuple containing a tuple, (1, 2, 3) - - It is possible to use ``unpythonic.misc.pack`` to create a tuple from - given elements: ``do[pack(1, 2, 3)]`` is interpreted as a single-item body - that creates a tuple (by calling a function). - - Note the outermost brackets belong to the ``do``; they don't yet create a list. - - In the *use brackets to denote a multi-expr body* syntax (e.g. ``multilambda``, - ``let`` constructs), the extra brackets already create a list, so in those - uses, the ambiguity does not arise. The transformation inserts not only the - word ``do``, but also the outermost brackets. For example:: - - let[(x, 1), - (y, 2)][[ - [x, y]]] - - transforms to:: - - let[(x, 1), - (y, 2)][do[[ # "do[" is inserted between the two opening brackets - [x, y]]]] # and its closing "]" is inserted here - - which already gets rid of the ambiguity. - - **Notes** - - Macros are expanded in an inside-out order, so a nested ``let`` shadows - names, if the same names appear in the ``do``:: - - do[local[x << 17], - let[(x, 23)][ - print(x)], # 23, the "x" of the "let" - print(x)] # 17, the "x" of the "do" - - The reason we require local names to be declared is to allow write access - to lexically outer environments from inside a ``do``:: - - let[(x, 17)][ - do[x << 23, # no "local[...]"; update the "x" of the "let" - local[y << 42], # "y" is local to the "do" - print(x, y)]] - - With the extra bracket syntax, the latter example can be written as:: - - let[(x, 17)][[ - x << 23, - local[y << 42], - print(x, y)]] - - It's subtly different in that the first version has the do-items in a tuple, - whereas this one has them in a list, but the behavior is exactly the same. - - Python does it the other way around, requiring a ``nonlocal`` statement - to re-bind a name owned by an outer scope. - - The ``let`` constructs solve this problem by having the local bindings - declared in a separate block, which plays the role of ``local``. - """ - if syntax != "expr": - raise SyntaxError("do is an expr macro only") - with dyn.let(_macro_expander=expander): - return _do(tree) - -def do0(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Like do, but return the value of the first expression.""" - if syntax != "expr": - raise SyntaxError("do0 is an expr macro only") - with dyn.let(_macro_expander=expander): - return _do0(tree) - -# ----------------------------------------------------------------------------- - -# TODO: change the block() construct to block[], for syntactic consistency -@parametricmacro -def let_syntax(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr/block] Introduce local **syntactic** bindings. - - **Expression variant**:: - - let_syntax[(lhs, rhs), ...][body] - let_syntax[(lhs, rhs), ...][[body0, ...]] - - Alternative haskelly syntax:: - - let_syntax[((lhs, rhs), ...) in body] - let_syntax[((lhs, rhs), ...) in [body0, ...]] - - let_syntax[body, where((lhs, rhs), ...)] - let_syntax[[body0, ...], where((lhs, rhs), ...)] - - **Block variant**:: - - with let_syntax: - with block as xs: # capture a block of statements - bare name - ... - with block(a, ...) as xs: # capture a block of statements - template - ... - with expr as x: # capture a single expression - bare name - ... - with expr(a, ...) as x: # capture a single expression - template - ... - body0 - ... - - A single expression can be a ``do[]`` if multiple expressions are needed. - - The bindings are applied **at macro expansion time**, substituting - the expression on the RHS for each instance of the corresponding LHS. - Each substitution gets a fresh copy. - - This is useful to e.g. locally abbreviate long function names at macro - expansion time (with zero run-time overhead), or to splice in several - (possibly parametric) instances of a common pattern. - - In the expression variant, ``lhs`` may be: - - - A bare name (e.g. ``x``), or - - - A simple template of the form ``f(x, ...)``. The names inside the - parentheses declare the formal parameters of the template (that can - then be used in the body). - - In the block variant: - - - The **as-part** specifies the name of the LHS. - - - If a template, the formal parameters are declared on the ``block`` - or ``expr``, not on the as-part (due to syntactic limitations). - - **Templates** - - To make parametric substitutions, use templates. - - Templates support only positional arguments, with no default values. - - Even in block templates, parameters are always expressions (because they - use the function-call syntax at the use site). - - In the body of the ``let_syntax``, a template is used like a function call. - Just like in an actual function call, when the template is substituted, - any instances of its formal parameters on its RHS get replaced by the - argument values from the "call" site; but ``let_syntax`` performs this - at macro-expansion time. - - Note each instance of the same formal parameter gets a fresh copy of the - corresponding argument value. - - **Substitution order** - - This is a two-step process. In the first step, we apply template substitutions. - In the second step, we apply bare name substitutions to the result of the - first step. (So RHSs of templates may use any of the bare-name definitions.) - - Within each step, the substitutions are applied **in the order specified**. - So if the bindings are ``((x, y), (y, z))``, then ``x`` transforms to ``z``. - But if the bindings are ``((y, z), (x, y))``, then ``x`` transforms to ``y``, - and only an explicit ``y`` at the use site transforms to ``z``. - - **Notes** - - Inspired by Racket's ``let-syntax`` and ``with-syntax``, see: - https://docs.racket-lang.org/reference/let.html - https://docs.racket-lang.org/reference/stx-patterns.html - - **CAUTION**: This is essentially a toy macro system inside the real - macro system, implemented with the real macro system. - - The usual caveats of macro systems apply. Especially, we support absolutely - no form of hygiene. Be very, very careful to avoid name conflicts. - - ``let_syntax`` is meant only for simple local substitutions where the - elimination of repetition can shorten the code and improve readability. - - If you need to do something complex, prefer writing a real macro directly - in `mcpyrate`. - """ - if syntax not in ("expr", "block"): - raise SyntaxError("let_syntax is an expr and block macro only") - - tree = expander.visit(tree) - - if syntax == "expr": - return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, allow_call_in_name_position=True) - else: # syntax == "block": - return _let_syntax_block(block_body=tree) - -@parametricmacro -def abbrev(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr/block] Exactly like ``let_syntax``, but expands outside in. - - Because this variant expands before any macros in the body, it can locally - rename other macros, e.g.:: - - abbrev[(m, macrowithverylongname)][ - m[tree1] if m[tree2] else m[tree3]] - - **CAUTION**: Because ``abbrev`` expands outside-in, and does not respect - boundaries of any nested ``abbrev`` invocations, it will not lexically scope - the substitutions. Instead, the outermost ``abbrev`` expands first, and then - any inner ones expand with whatever substitutions they have remaining. - - If the same name is used on the LHS in two or more nested ``abbrev``, - any inner ones will likely raise an error (unless the outer substitution - just replaces a name with another), because also the names on the LHS - in the inner ``abbrev`` will undergo substitution when the outer - ``abbrev`` expands. - """ - if syntax not in ("expr", "block"): - raise SyntaxError("abbrev is an expr and block macro only") - - # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. - - if syntax == "expr": - return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, allow_call_in_name_position=True) - else: - return _let_syntax_block(block_body=tree) - -# ----------------------------------------------------------------------------- - -def forall(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Nondeterministic evaluation. - - Fully based on AST transformation, with real lexical variables. - Like Haskell's do-notation, but here specialized for the List monad. - - Example:: - - # pythagorean triples - pt = forall[z << range(1, 21), # hypotenuse - x << range(1, z+1), # shorter leg - y << range(x, z+1), # longer leg - insist(x*x + y*y == z*z), - (x, y, z)] - assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), - (8, 15, 17), (9, 12, 15), (12, 16, 20)) - """ - if syntax != "expr": - raise SyntaxError("forall is an expr macro only") - - tree = expander.visit(tree) - - return _forall(exprs=tree) - -# ----------------------------------------------------------------------------- - -def multilambda(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] Supercharge your lambdas: multiple expressions, local variables. - - For all ``lambda`` lexically inside the ``with multilambda`` block, - ``[...]`` denotes a multiple-expression body with an implicit ``do``:: - - lambda ...: [expr0, ...] --> lambda ...: do[expr0, ...] - - Only the outermost set of brackets around the body of a ``lambda`` denotes - a multi-expression body; the rest are interpreted as lists, as usual. - - Examples:: - - with multilambda: - echo = lambda x: [print(x), x] - assert echo("hi there") == "hi there" - - count = let[(x, 0)][ - lambda: [x << x + 1, - x]] - assert count() == 1 - assert count() == 2 - - mk12 = lambda: [[1, 2]] - assert mk12() == [1, 2] - - For local variables, see ``do``. - """ - if syntax != "block": - raise SyntaxError("multilambda is a block macro only") - - # Expand outside in. - # multilambda should expand first before any let[], do[] et al. that happen - # to be inside the block, to avoid misinterpreting implicit lambdas - # generated by those constructs. - with dyn.let(_macro_expander=expander): # implicit do (extra bracket notation) needs this. - return _multilambda(block_body=tree) - -def namedlambda(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] Name lambdas implicitly. - - Lexically inside a ``with namedlambda`` block, any literal ``lambda`` - that is assigned to a name using one of the supported assignment forms - is named to have the name of the LHS of the assignment. The name is - captured at macro expansion time. - - Naming modifies the original function object. - - We support: - - - Single-item assignments to a local name, ``f = lambda ...: ...`` - - - Named expressions (a.k.a. walrus operator, Python 3.8+), - ``f := lambda ...: ...`` - - - Assignments to unpythonic environments, ``f << (lambda ...: ...)`` - - - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any - let syntax supported by unpythonic (here using the haskelly let-in - just as an example). - - Support for other forms of assignment might or might not be added in a - future version. - - Example:: - - with namedlambda: - f = lambda x: x**3 # assignment: name as "f" - - let[(x, 42), (g, None), (h, None)][[ - g << (lambda x: x**2), # env-assignment: name as "g" - h << f, # still "f" (no literal lambda on RHS) - (g(x), h(x))]] - - foo = let[(f7, lambda x: x) in f7] # let-binding: name as "f7" - - The naming is performed using the function ``unpythonic.misc.namelambda``, - which will update ``__name__``, ``__qualname__`` and ``__code__.co_name``. - """ - if syntax != "block": - raise SyntaxError("namedlambda is a block macro only") - - # Two-pass macro. We pass in the expander to allow the macro to decide when to recurse. - with dyn.let(_macro_expander=expander): - return _namedlambda(block_body=tree) - -def f(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Underscore notation (quick lambdas) for Python. - - Usage:: - - f[body] - - The ``f[]`` macro creates a lambda. Each underscore in ``body`` - introduces a new parameter. - - Example:: - - func = f[_ * _] - - expands to:: - - func = lambda a0, a1: a0 * a1 - - The underscore is interpreted magically by ``f[]``; but ``_`` itself - is not a macro, and has no special meaning outside ``f[]``. The underscore - does **not** need to be imported for ``f[]`` to recognize it. - - The macro does not descend into any nested ``f[]``. - """ - if syntax != "expr": - raise SyntaxError("f is an expr macro only") - - # This macro expands outside in, but needs `expander` for introspection. - with dyn.let(_macro_expander=expander): - return _f(tree) - -def quicklambda(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] Make ``f`` quick lambdas expand first. - - To be able to transform correctly, the block macros in ``unpythonic.syntax`` - that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all - ``lambda`` definitions written with Python's standard ``lambda``. - - However, the ``f`` macro uses the syntax ``f[...]``, which (to the analyzer) - does not look like a lambda definition. This macro changes the expansion - order, forcing any ``f[...]`` lexically inside the block to expand before - any other macros do. - - Any expression of the form ``f[...]``, where ``f`` is any name bound in the - current macro expander to the macro `unpythonic.syntax.f`, is understood as - a quick lambda. (In plain English, this respects as-imports of the macro ``f``.) - - Example - a quick multilambda:: - - from unpythonic.syntax import macros, multilambda, quicklambda, f, local - - with quicklambda, multilambda: - func = f[[local[x << _], - local[y << _], - x + y]] - assert func(1, 2) == 3 - - (This is of course rather silly, as an unnamed argument can only be mentioned - once. If we're giving names to them, a regular ``lambda`` is shorter to write. - The point is, this combo is now possible.) - """ - if syntax != "block": - raise SyntaxError("quicklambda is a block macro only") - - # This macro expands outside in. - # - # In `mcpyrate`, expander instances are cheap - so we create a second expander - # to which we register only the `f` macro, under whatever names it appears in - # the original expander. Thus it leaves all other macros alone. This is the - # official `mcpyrate` way to immediately expand only some particular macros - # inside the current macro invocation. - bindings = extract_bindings(expander.bindings, f) - return MacroExpander(bindings, expander.filename).visit(tree) - -def envify(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] Make formal parameters live in an unpythonic env. - - The purpose is to allow overwriting formals using unpythonic's - expression-assignment ``name << value``. The price is that the references - to the arguments are copied into an env whenever an envified function is - entered. - - Example - PG's accumulator puzzle (http://paulgraham.com/icad.html):: - - with envify: - def foo(n): - return lambda i: n << n + i - - Or even shorter:: - - with autoreturn, envify: - def foo(n): - lambda i: n << n + i - """ - if syntax != "block": - raise SyntaxError("envify is a block macro only") - - # Two-pass macro. - with dyn.let(_macro_expander=expander): - return _envify(block_body=tree) - -# ----------------------------------------------------------------------------- - -def autoreturn(tree, *, syntax, **kw): # noqa: F811 - """[syntax, block] Implicit "return" in tail position, like in Lisps. - - Each ``def`` function definition lexically within the ``with autoreturn`` - block is examined, and if the last item within the body is an expression - ``expr``, it is transformed into ``return expr``. - - If the last item is an if/elif/else block, the transformation is applied - to the last item in each of its branches. - - If the last item is a ``with`` or ``async with`` block, the transformation - is applied to the last item in its body. - - If the last item is a try/except/else/finally block, the rules are as follows. - If an ``else`` clause is present, the transformation is applied to the last - item in it; otherwise, to the last item in the ``try`` clause. Additionally, - in both cases, the transformation is applied to the last item in each of the - ``except`` clauses. The ``finally`` clause is not transformed; the intention - is it is usually a finalizer (e.g. to release resources) that runs after the - interesting value is already being returned by ``try``, ``else`` or ``except``. - - Example:: - - with autoreturn: - def f(): - "I'll just return this" - assert f() == "I'll just return this" - - def g(x): - if x == 1: - "one" - elif x == 2: - "two" - else: - "something else" - assert g(1) == "one" - assert g(2) == "two" - assert g(42) == "something else" - - **CAUTION**: If the final ``else`` is omitted, as often in Python, then - only the ``else`` item is in tail position with respect to the function - definition - likely not what you want. - - So with ``autoreturn``, the final ``else`` should be written out explicitly, - to make the ``else`` branch part of the same if/elif/else block. - - **CAUTION**: ``for``, ``async for``, ``while`` are currently not analyzed; - effectively, these are defined as always returning ``None``. If the last item - in your function body is a loop, use an explicit return. - - **CAUTION**: With ``autoreturn`` enabled, functions no longer return ``None`` - by default; the whole point of this macro is to change the default return - value. - - The default return value is ``None`` only if the tail position contains - a statement (because in a sense, a statement always returns ``None``). - """ - if syntax != "block": - raise SyntaxError("autoreturn is a block macro only") - - # Expand outside in. Any nested macros should get clean standard Python, - # not having to worry about implicit "return" statements. - return _autoreturn(block_body=tree) - -def tco(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] Implicit tail-call optimization (TCO). - - Examples:: - - with tco: - evenp = lambda x: (x == 0) or oddp(x - 1) - oddp = lambda x: (x != 0) and evenp(x - 1) - assert evenp(10000) is True - - with tco: - def evenp(x): - if x == 0: - return True - return oddp(x - 1) - def oddp(x): - if x != 0: - return evenp(x - 1) - return False - assert evenp(10000) is True - - This is based on a strategy similar to MacroPy's tco macro, but using - the TCO machinery from ``unpythonic.tco``. - - This recursively handles also builtins ``a if p else b``, ``and``, ``or``; - and from ``unpythonic.syntax``, ``do[]``, ``let[]``, ``letseq[]``, ``letrec[]``, - when used in computing a return value. (``aif[]`` and ``cond[]`` also work.) - - Note only calls **in tail position** will be TCO'd. Any other calls - are left as-is. Tail positions are: - - - The whole return value, if it is just a single call. - - - Both ``a`` and ``b`` branches of ``a if p else b`` (but not ``p``). - - - The last item in an ``and``/``or``. If these are nested, only the - last item in the whole expression involving ``and``/``or``. E.g. in:: - - (a and b) or c - a and (b or c) - - in either case, only ``c`` is in tail position, regardless of the - values of ``a``, ``b``. - - - The last item in a ``do[]``. - - - In a ``do0[]``, this is the implicit item that just returns the - stored return value. - - - The argument of a call to an escape continuation. The ``ec(...)`` call - itself does not need to be in tail position; escaping early is the - whole point of an ec. - - All function definitions (``def`` and ``lambda``) lexically inside the block - undergo TCO transformation. The functions are automatically ``@trampolined``, - and any tail calls in their return values are converted to ``jump(...)`` - for the TCO machinery. - - Note in a ``def`` you still need the ``return``; it marks a return value. - But see ``autoreturn``:: - - with autoreturn, tco: - def evenp(x): - if x == 0: - True - else: - oddp(x - 1) - def oddp(x): - if x != 0: - evenp(x - 1) - else: - False - assert evenp(10000) is True - - **CAUTION**: regarding escape continuations, only basic uses of ecs created - via ``call_ec`` are currently detected as being in tail position. Any other - custom escape mechanisms are not supported. (This is mainly of interest for - lambdas, which have no ``return``, and for "multi-return" from a nested - function.) - - *Basic use* is defined as either of these two cases:: - - # use as decorator - @call_ec - def result(ec): - ... - - # use directly on a literal lambda - result = call_ec(lambda ec: ...) - - When macro expansion of the ``with tco`` block starts, names of escape - continuations created **anywhere lexically within** the ``with tco`` block - are captured. Lexically within the block, any call to a function having - any of the captured names, or as a fallback, one of the literal names - ``ec``, ``brk``, ``throw`` is interpreted as invoking an escape - continuation. - """ - if syntax != "block": - raise SyntaxError("tco is a block macro only") - - # Two-pass macro. - with dyn.let(_macro_expander=expander): - return _tco(block_body=tree) - -def continuations(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] call/cc for Python. - - This allows saving the control state and then jumping back later - (in principle, any time later). Some possible use cases: - - - Tree traversal (possibly a cartesian product of multiple trees, with the - current position in each tracked automatically). - - - McCarthy's amb operator. - - - Generators. (Python already has those, so only for teaching.) - - This is a very loose pythonification of Paul Graham's continuation-passing - macros, which implement continuations by chaining closures and passing the - continuation semi-implicitly. For details, see chapter 20 in On Lisp: - - http://paulgraham.com/onlisp.html - - Continuations are most readily implemented when the program is written in - continuation-passing style (CPS), but that is unreadable for humans. - The purpose of this macro is to partly automate the CPS transformation, so - that at the use site, we can write CPS code in a much more readable fashion. - - A ``with continuations`` block implies TCO; the same rules apply as in a - ``with tco`` block. Furthermore, ``with continuations`` introduces the - following additional rules: - - - Functions which make use of continuations, or call other functions that do, - must be defined within a ``with continuations`` block, using the usual - ``def`` or ``lambda`` forms. - - - All function definitions in a ``with continuations`` block, including - any nested definitions, have an implicit formal parameter ``cc``, - **even if not explicitly declared** in the formal parameter list. - - If declared explicitly, ``cc`` must be in a position that can accept a - default value. - - This means ``cc`` must be declared either as by-name-only:: - - with continuations: - def myfunc(a, b, *, cc): - ... - - f = lambda *, cc: ... - - or as the last parameter that has no default:: - - with continuations: - def myfunc(a, b, cc): - ... - - f = lambda cc: ... - - Then the continuation machinery will automatically set the default value - of ``cc`` to the default continuation (``identity``), which just returns - its arguments. - - The most common use case for explicitly declaring ``cc`` is that the - function is the target of a ``call_cc[]``; then it helps readability - to make the ``cc`` parameter explicit. - - - A ``with continuations`` block will automatically transform all - function definitions and ``return`` statements lexically contained - within the block to use the continuation machinery. - - - ``return somevalue`` actually means a tail-call to ``cc`` with the - given ``somevalue``. - - Multiple values can be returned as a ``tuple``. Tupleness is tested - at run-time. - - Any tuple return value is automatically unpacked to the positional - args of ``cc``. To return multiple things as one without the implicit - unpacking, use a ``list``. - - - An explicit ``return somefunc(arg0, ..., k0=v0, ...)`` actually means - a tail-call to ``somefunc``, with its ``cc`` automatically set to our - ``cc``. Hence this inserts a call to ``somefunc`` before proceeding - with our current continuation. (This is most often what we want when - making a tail-call from a continuation-enabled function.) - - Here ``somefunc`` **must** be a continuation-enabled function; - otherwise the TCO chain will break and the result is immediately - returned to the top-level caller. - - (If the call succeeds at all; the ``cc`` argument is implicitly - filled in and passed by name. Regular functions usually do not - accept a named parameter ``cc``, let alone know what to do with it.) - - - Just like in ``with tco``, a lambda body is analyzed as one big - return-value expression. This uses the exact same analyzer; for example, - ``do[]`` (including any implicit ``do[]``) and the ``let[]`` expression - family are supported. - - - Calls from functions defined in one ``with continuations`` block to those - defined in another are ok; there is no state or context associated with - the block. - - - Much of the language works as usual. - - Any non-tail calls can be made normally. Regular functions can be called - normally in any non-tail position. - - Continuation-enabled functions behave as regular functions when - called normally; only tail calls implicitly set ``cc``. A normal call - uses ``identity`` as the default ``cc``. - - - For technical reasons, the ``return`` statement is not allowed at the - top level of the ``with continuations:`` block. (Because a continuation - is essentially a function, ``return`` would behave differently based on - whether it is placed lexically before or after a ``call_cc[]``.) - - If you absolutely need to terminate the function surrounding the - ``with continuations:`` block from inside the block, use an exception - to escape; see ``call_ec``, ``catch``, ``throw``. - - **Capturing the continuation**: - - Inside a ``with continuations:`` block, the ``call_cc[]`` statement - captures a continuation. (It is actually a macro, for technical reasons.) - - For various possible program topologies that continuations may introduce, see - the clarifying pictures under ``doc/`` in the source distribution. - - Syntax:: - - x = call_cc[func(...)] - *xs = call_cc[func(...)] - x0, ... = call_cc[func(...)] - x0, ..., *xs = call_cc[func(...)] - call_cc[func(...)] - - Conditional variant:: - - x = call_cc[f(...) if p else g(...)] - *xs = call_cc[f(...) if p else g(...)] - x0, ... = call_cc[f(...) if p else g(...)] - x0, ..., *xs = call_cc[f(...) if p else g(...)] - call_cc[f(...) if p else g(...)] - - Assignment targets: - - - To destructure a multiple-values (from a tuple return value), - use a tuple assignment target (comma-separated names, as usual). - - - The last assignment target may be starred. It is transformed into - the vararg (a.k.a. ``*args``) of the continuation function. - (It will capture a whole tuple, or any excess items, as usual.) - - - To ignore the return value (useful if ``func`` was called only to - perform its side-effects), just omit the assignment part. - - Conditional variant: - - - ``p`` is any expression. If truthy, ``f(...)`` is called, and if falsey, - ``g(...)`` is called. - - - Each of ``f(...)``, ``g(...)`` may be ``None``. A ``None`` skips the - function call, proceeding directly to the continuation. Upon skipping, - all assignment targets (if any are present) are set to ``None``. - The starred assignment target (if present) gets the empty tuple. - - - The main use case of the conditional variant is for things like:: - - with continuations: - k = None - def setk(cc): - global k - k = cc - def dostuff(x): - call_cc[setk() if x > 10 else None] # capture only if x > 10 - ... - - To keep things relatively straightforward, a ``call_cc[]`` is only - allowed to appear **at the top level** of: - - - the ``with continuations:`` block itself - - a ``def`` or ``async def`` - - Nested defs are ok; here *top level* only means the top level of the - *currently innermost* ``def``. - - If you need to place ``call_cc[]`` inside a loop, use ``@looped`` et al. - from ``unpythonic.fploop``; this has the loop body represented as the - top level of a ``def``. - - Multiple ``call_cc[]`` statements in the same function body are allowed. - These essentially create nested closures. - - **Main differences to Scheme and Racket**: - - Compared to Scheme/Racket, where ``call/cc`` will capture also expressions - occurring further up in the call stack, our ``call_cc`` may be need to be - placed differently (further out, depending on what needs to be captured) - due to the delimited nature of the continuations implemented here. - - Scheme and Racket implicitly capture the continuation at every position, - whereas we do it explicitly, only at the use sites of the ``call_cc`` macro. - - Also, since there are limitations to where a ``call_cc[]`` may appear, some - code may need to be structured differently to do some particular thing, if - porting code examples originally written in Scheme or Racket. - - Unlike ``call/cc`` in Scheme/Racket, ``call_cc`` takes **a function call** - as its argument, not just a function reference. Also, there's no need for - it to be a one-argument function; any other args can be passed in the call. - The ``cc`` argument is filled implicitly and passed by name; any others are - passed exactly as written in the client code. - - **Technical notes**: - - The ``call_cc[]`` statement essentially splits its use site into *before* - and *after* parts, where the *after* part (the continuation) can be run - a second and further times, by later calling the callable that represents - the continuation. This makes a computation resumable from a desired point. - - The return value of the continuation is whatever the original function - returns, for any ``return`` statement that appears lexically after the - ``call_cc[]``. - - The effect of ``call_cc[]`` is that the function call ``func(...)`` in - the brackets is performed, with its ``cc`` argument set to the lexically - remaining statements of the current ``def`` (at the top level, the rest - of the ``with continuations`` block), represented as a callable. - - The continuation itself ends there (it is *delimited* in this particular - sense), but it will chain to the ``cc`` of the function it appears in. - This is termed the *parent continuation* (**pcc**), stored in the internal - variable ``_pcc`` (which defaults to ``None``). - - Via the use of the pcc, here ``f`` will maintain the illusion of being - just one function, even though a ``call_cc`` appears there:: - - def f(*, cc): - ... - call_cc[g(1, 2, 3)] - ... - - The continuation is a closure. For its pcc, it will use the value the - original function's ``cc`` had when the definition of the continuation - was executed (for that particular instance of the closure). Hence, calling - the original function again with its ``cc`` set to something else will - produce a new continuation instance that chains into that new ``cc``. - - The continuation's own ``cc`` will be ``identity``, to allow its use just - like any other function (also as argument of a ``call_cc`` or target of a - tail call). - - When the pcc is set (not ``None``), the effect is to run the pcc first, - and ``cc`` only after that. This preserves the whole captured tail of a - computation also in the presence of nested ``call_cc`` invocations (in the - above example, this would occur if also ``g`` used ``call_cc``). - - Continuations are not accessible by name (their definitions are named by - gensym). To get a reference to a continuation instance, stash the value - of the ``cc`` argument somewhere while inside the ``call_cc``. - - The function ``func`` called by a ``call_cc[func(...)]`` is (almost) the - only place where the ``cc`` argument is actually set. There it is the - captured continuation. Roughly everywhere else, ``cc`` is just ``identity``. - - Tail calls are an exception to this rule; a tail call passes along the current - value of ``cc``, unless overridden manually (by setting the ``cc=...`` kwarg - in the tail call). - - When the pcc is set (not ``None``) at the site of the tail call, the - machinery will create a composed continuation that runs the pcc first, - and ``cc`` (whether current or manually overridden) after that. This - composed continuation is then passed to the tail call as its ``cc``. - - **Tips**: - - - Once you have a captured continuation, one way to use it is to set - ``cc=...`` manually in a tail call, as was mentioned. Example:: - - def main(): - call_cc[myfunc()] # call myfunc, capturing the current cont... - ... # ...which is the rest of "main" - - def myfunc(cc): - ourcc = cc # save the captured continuation (sent by call_cc[]) - def somefunc(): - return dostuff(..., cc=ourcc) # and use it here - somestack.append(somefunc) - - In this example, when ``somefunc`` is eventually called, it will tail-call - ``dostuff`` and then proceed with the continuation ``myfunc`` had - at the time when that instance of the ``somefunc`` closure was created. - (This pattern is essentially how to build the ``amb`` operator.) - - - Instead of setting ``cc``, you can also overwrite ``cc`` with a captured - continuation inside a function body. That overrides the continuation - for the rest of the dynamic extent of the function, not only for a - particular tail call:: - - def myfunc(cc): - ourcc = cc - def somefunc(): - cc = ourcc - return dostuff(...) - somestack.append(somefunc) - - - A captured continuation can also be called manually; it's just a callable. - - The assignment targets, at the ``call_cc[]`` use site that spawned this - particular continuation, specify its call signature. All args are - positional, except the implicit ``cc``, which is by-name-only. - - - Just like in Scheme/Racket's ``call/cc``, the values that get bound - to the ``call_cc[]`` assignment targets on second and further calls - (when the continuation runs) are the arguments given to the continuation - when it is called (whether implicitly or manually). - - - Setting ``cc`` to ``unpythonic.fun.identity``, while inside a ``call_cc``, - will short-circuit the rest of the computation. In such a case, the - continuation will not be invoked automatically. A useful pattern for - suspend/resume. - - - However, it is currently not possible to prevent the rest of the tail - of a captured continuation (the pcc) from running, apart from manually - setting ``_pcc`` to ``None`` before executing a ``return``. Note that - doing that is not strictly speaking supported (and may be subject to - change in a future version). - - - When ``call_cc[]`` appears inside a function definition: - - - It tail-calls ``func``, with its ``cc`` set to the captured - continuation. - - - The return value of the function containing one or more ``call_cc[]`` - statements is the return value of the continuation. - - - When ``call_cc[]`` appears at the top level of ``with continuations``: - - - A normal call to ``func`` is made, with its ``cc`` set to the captured - continuation. - - - In this case, if the continuation is called later, it always - returns ``None``, because the use site of ``call_cc[]`` is not - inside a function definition. - - - If you need to insert just a tail call (no further statements) before - proceeding with the current continuation, no need for ``call_cc[]``; - use ``return func(...)`` instead. - - The purpose of ``call_cc[func(...)]`` is to capture the current - continuation (the remaining statements), and hand it to ``func`` - as a first-class value. - - - To combo with ``multilambda``, use this ordering:: - - with multilambda, continuations: - ... - - - Some very limited comboability with ``call_ec``. May be better to plan - ahead, using ``call_cc[]`` at the appropriate outer level, and then - short-circuit (when needed) by setting ``cc`` to ``identity``. - This avoids the need to have both ``call_cc`` and ``call_ec`` at the - same time. - - - ``unpythonic.ec.call_ec`` can be used normally **lexically before any** - ``call_cc[]``, but (in a given function) after at least one ``call_cc[]`` - has run, the ``ec`` ceases to be valid. This is because our ``call_cc[]`` - actually splits the function into *before* and *after* parts, and - **tail-calls** the *after* part. - - (Wrapping the ``def`` in another ``def``, and placing the ``call_ec`` - on the outer ``def``, does not help either, because even the outer - function has exited by the time *the continuation* is later called - the second and further times.) - - Usage of ``call_ec`` while inside a ``with continuations`` block is:: - - with continuations: - @call_ec - def result(ec): - print("hi") - ec(42) - print("not reached") - assert result == 42 - - result = call_ec(lambda ec: do[print("hi"), - ec(42), - print("not reached")]) - - Note the signature of ``result``. Essentially, ``ec`` is a function - that raises an exception (to escape to a dynamically outer context), - whereas the implicit ``cc`` is the closure-based continuation handled - by the continuation machinery. - - See the ``tco`` macro for details on the ``call_ec`` combo. - """ - if syntax != "block": - raise SyntaxError("continuations is a block macro only") - - # Two-pass macro. - with dyn.let(_macro_expander=expander): - return _continuations(block_body=tree) - -# ----------------------------------------------------------------------------- - -@parametricmacro -def nb(tree, *, args, syntax, **kw): # noqa: F811 - """[syntax, block] Ultralight math notebook. - - Auto-print top-level expressions, auto-assign last result as _. - - A custom print function can be supplied as an argument. - - Example:: - - with nb: - 2 + 3 - 42 * _ - - from sympy import * - with nb[pprint]: - x, y = symbols("x, y") - x * y - 3 * _ - """ - if syntax != "block": - raise SyntaxError("nb is a block macro only") - - # Expand outside in. This macro is so simple and orthogonal the - # ordering doesn't matter. This is cleaner. - return _nb(body=tree, args=args) - -# ----------------------------------------------------------------------------- - -@parametricmacro -def dbg(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr/block] Debug-print expressions including their source code. - - **Expression variant**: - - Example:: - - dbg[25 + 17] # --> [file.py:100] (25 + 17): 42 - - The transformation is:: - - dbg[expr] --> dyn.dbgprint_expr(k, v, filename=__file__, lineno=xxx) - - where ``k`` is the source code of the expression and ``v`` is its value, - and `dyn` is `unpythonic.dynassign.dyn` (hygienically captured, so you - don't need to import it just to use the `dbg[]` macro). - - ``xxx`` is the original line number before macro expansion, if available - in the AST node of the expression, otherwise ``None``. (Some macros might - not care about inserting line numbers, because `mcpyrate` fixes any missing - line numbers in a postprocess step; this is why it might be missing at some - locations in any specific macro-enabled program.) - - A default implementation of the debug printer is provided and automatically - assigned as the default value for `dyn.dbgprint_expr`. - - To customize the debug printing, set your custom printer function to the - dynvar ``dbgprint_expr``, using `with dyn.let(dbgprint_expr=...)`. - - The custom function, beside performing any printing/logging as a side effect, - **must** return the value ``v``, so that surrounding an expression with - ``dbg[...]`` does not alter its value. - - If you want to use the default implementation as part of your customized one - (e.g. if you want to decorate that with some logging code), it is available as - `unpythonic.syntax.dbgprint_expr`. - - **Block variant**: - - Lexically within the block, any call to ``print`` (alternatively, if specified, - the optional custom print function), prints both the expression source code - and the corresponding value. - - A custom print function can be supplied as an argument. To implement a - custom print function, see the default implementation ``dbgprint_block`` - for the signature. - - If you want to use the default implementation as part of your customized one, - it is available as `unpythonic.syntax.dbgprint_block`. - - Examples:: - - with dbg: - x = 2 - print(x) # --> [file.py:100] x: 2 - - with dbg: - x = 2 - y = 3 - print(x, y) # --> [file.py:100] x: 2, y: 3 - print(x, y, sep="\n") # --> [file.py:100] x: 2 - # [file.py:100] y: 3 - - prt = lambda *args, **kwargs: print(*args) - with dbg[prt]: - x = 2 - prt(x) # --> ('x',) (2,) - print(x) # --> 2 - - with dbg[prt]: - x = 2 - y = 17 - prt(x, y, 1 + 2) # --> ('x', 'y', '(1 + 2)'), (2, 17, 3)) - - **CAUTION**: The source code is back-converted from the AST representation; - hence its surface syntax may look slightly different to the original (e.g. - extra parentheses). See ``mcpyrate.unparse``. - """ - if syntax not in ("expr", "block"): - raise SyntaxError("dbg is an expr and block macro only") - - tree = expander.visit(tree) - - if syntax == "expr": - return _dbg_expr(tree) - else: # syntax == "block": - return _dbg_block(body=tree, args=args) - -# ----------------------------------------------------------------------------- - -def lazify(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, block] Call-by-need for Python. - - In a ``with lazify`` block, function arguments are evaluated only when - actually used, at most once each, and in the order in which they are - actually used. Promises are automatically forced on access. - - Automatic lazification applies to arguments in function calls and to - let-bindings, since they play a similar role. **No other binding forms - are auto-lazified.** - - Automatic lazification uses the ``lazyrec[]`` macro, which recurses into - certain types of container literals, so that the lazification will not - interfere with unpacking. See its docstring for details. - - Comboing with other block macros in ``unpythonic.syntax`` is supported, - including ``curry`` and ``continuations``. - - Silly contrived example:: - - with lazify: - def my_if(p, a, b): - if p: - return a # b never evaluated in this code path... - else: - return b # a never evaluated in this code path... - - # ...hence the divisions by zero here are never performed. - assert my_if(True, 23, 1/0) == 23 - assert my_if(False, 1/0, 42) == 42 - - Note ``my_if`` is a run-of-the-mill runtime function, not a macro. Only the - ``with lazify`` is imbued with any magic. - - Like ``with continuations``, no state or context is associated with a - ``with lazify`` block, so lazy functions defined in one block may call - those defined in another. Calls between lazy and strict code are also - supported (in both directions), without requiring any extra effort. - - Evaluation of each lazified argument is guaranteed to occur at most once; - the value is cached. Order of evaluation of lazy arguments is determined - by the (dynamic) order in which the lazy code actually uses them. - - Essentially, the above code expands into:: - - from unpythonic.syntax import macros, lazy - from unpythonic.syntax import force - - def my_if(p, a, b): - if force(p): - return force(a) - else: - return force(b) - assert my_if(lazy[True], lazy[23], lazy[1/0]) == 23 - assert my_if(lazy[False], lazy[1/0], lazy[42]) == 42 - - plus some clerical details to allow lazy and strict code to be mixed. - - Just passing through a lazy argument to another lazy function will - not trigger evaluation, even when it appears in a computation inlined - to the argument list:: - - with lazify: - def g(a, b): - return a - def f(a, b): - return g(2*a, 3*b) - assert f(21, 1/0) == 42 - - The division by zero is never performed, because the value of ``b`` is - not needed to compute the result (worded less magically, that promise is - never forced in the code path that produces the result). Essentially, - the above code expands into:: - - from unpythonic.syntax import macros, lazy - from unpythonic.syntax import force - - def g(a, b): - return force(a) - def f(a, b): - return g(lazy[2*force(a)], lazy[3*force(b)]) - assert f(lazy[21], lazy[1/0]) == 42 - - This relies on the magic of closures to capture f's ``a`` and ``b`` into - the promises. - - But be careful; **assignments are not auto-lazified**, so the following does - **not** work:: - - with lazify: - def g(a, b): - return a - def f(a, b): - c = 3*b # not in an arglist, b gets evaluated! - return g(2*a, c) - assert f(21, 1/0) == 42 - - To avoid that, explicitly wrap the computation into a ``lazy[]``. For why - assignment RHSs are not auto-lazified, see the section on pitfalls below. - - In calls, bare references (name, subscript, attribute) are detected and for - them, re-thunking is skipped. For example:: - - def g(a): - return a - def f(a): - return g(a) - assert f(42) == 42 - - expands into:: - - def g(a): - return force(a) - def f(a): - return g(a) # <-- no lazy[force(a)] since "a" is just a name - assert f(lazy[42]) == 42 - - When resolving references, subscripts and attributes are forced just enough - to obtain the containing object from a promise, if any; for example, the - elements of a list ``lst`` will not be evaluated just because the user code - happens to use ``lst.append(...)``; this only forces the object ``lst`` - itself. - - A ``lst`` appearing by itself evaluates the whole list. Similarly, ``lst[0]`` - by itself evaluates only the first element, and ``lst[:-1]`` by itself - evaluates all but the last element. The index expression in a subscript is - fully forced, because its value is needed to determine which elements of the - subscripted container are to be accessed. - - **Mixing lazy and strict code** - - Lazy code is allowed to call strict functions and vice versa, without - requiring any additional effort. - - Keep in mind what this implies: when calling a strict function, any arguments - given to it will be evaluated! - - In the other direction, when calling a lazy function from strict code, the - arguments are evaluated by the caller before the lazy code gets control. - The lazy code gets just the evaluated values. - - If you have, in strict code, an argument expression you want to pass lazily, - use syntax like ``f(lazy[...], ...)``. If you accidentally do this in lazy - code, it shouldn't break anything; ``with lazify`` detects any argument - expressions that are already promises, and just passes them through. - - **Forcing promises manually** - - This is mainly useful if you ``lazy[]`` or ``lazyrec[]`` something explicitly, - and want to compute its value outside a ``with lazify`` block. - - We provide the functions ``force1`` and ``force``. - - Using ``force1``, if ``x`` is a ``lazy[]`` promise, it will be forced, - and the resulting value is returned. If ``x`` is not a promise, - ``x`` itself is returned, à la Racket. - - The function ``force``, in addition, descends into containers (recursively). - When an atom ``x`` (i.e. anything that is not a container) is encountered, - it is processed using ``force1``. - - Mutable containers are updated in-place; for immutables, a new instance is - created. Any container with a compatible ``collections.abc`` is supported. - (See ``unpythonic.collections.mogrify`` for details.) In addition, as - special cases ``unpythonic.collections.box`` and ``unpythonic.llist.cons`` - are supported. - - **Tips, tricks and pitfalls** - - You can mix and match bare data values and promises, since ``force(x)`` - evaluates to ``x`` when ``x`` is not a promise. - - So this is just fine:: - - with lazify: - def f(x): - x = 2*21 # assign a bare data value - print(x) # the implicit force(x) evaluates to x - f(17) - - If you want to manually introduce a promise, use ``lazy[]``:: - - from unpythonic.syntax import macros, lazify, lazy - - with lazify: - def f(x): - x = lazy[2*21] # assign a promise - print(x) # the implicit force(x) evaluates the promise - f(17) - - If you have a container literal and want to lazify it recursively in a - position that does not auto-lazify, use ``lazyrec[]`` (see its docstring - for details):: - - from unpythonic.syntax import macros, lazify, lazyrec - - with lazify: - def f(x): - return x[:-1] - lst = lazyrec[[1, 2, 3/0]] - assert f(lst) == [1, 2] - - For non-literal containers, use ``lazy[]`` for each item as appropriate:: - - def f(lst): - lst.append(lazy["I'm lazy"]) - lst.append(lazy["Don't call me lazy, I'm just evaluated later!"]) - - Keep in mind, though, that ``lazy[]`` will introduce a lambda, so there's - the usual pitfall:: - - from unpythonic.syntax import macros, lazify, lazy - - with lazify: - lst = [] - for x in range(3): # DANGER: only one "x", mutated imperatively - lst.append(lazy[x]) # all these closures capture the same "x" - print(lst[0]) # 2 - print(lst[1]) # 2 - print(lst[2]) # 2 - - So to capture the value instead of the name, use the usual workaround, - the wrapper lambda (here written more readably as a let, which it really is):: - - from unpythonic.syntax import macros, lazify, lazy, let - - with lazify: - lst = [] - for x in range(3): - lst.append(let[(y, x) in lazy[y]]) - print(lst[0]) # 0 - print(lst[1]) # 1 - print(lst[2]) # 2 - - Be careful not to ``lazy[]`` or ``lazyrec[]`` too much:: - - with lazify: - a = 10 - a = lazy[2*a] # 20, right? - print(a) # crash! - - Why does this example crash? The expanded code is:: - - with lazify: - a = 10 - a = lazy[2*force(a)] - print(force(a)) - - The ``lazy[]`` sets up a promise, which will force ``a`` *at the time when - the containing promise is forced*, but at that time the name ``a`` points - to a promise, which will force... - - The fundamental issue is that ``a = 2*a`` is an imperative update; if you - need to do that, just let Python evaluate the RHS normally (i.e. use the - value the name ``a`` points to *at the time when the RHS runs*). - - Assigning a lazy value to a new name evaluates it, because any read access - triggers evaluation:: - - with lazify: - def g(x): - y = x # the "x" on the RHS triggers the implicit force - print(y) # bare data value - f(2*21) - - Inspired by Haskell, Racket's (delay) and (force), and lazy/racket. - - **Combos** - - Introducing the *HasThon* programming language (it has 100% more Thon than - popular brands):: - - with autocurry, lazify: # or continuations, autocurry, lazify if you want those - def add2first(a, b, c): - return a + b - assert add2first(2)(3)(1/0) == 5 - - def f(a, b): - return a - assert let[((c, 42), - (d, 1/0)) in f(c)(d)] == 42 - assert letrec[((c, 42), - (d, 1/0), - (e, 2*c)) in f(e)(d)] == 84 - - assert letrec[((c, 42), - (d, 1/0), - (e, 2*c)) in [local[x << f(e)(d)], - x/4]] == 21 - - Works also with continuations. Rules: - - - Also continuations are transformed into lazy functions. - - - ``cc`` built by chain_conts is treated as lazy, **itself**; then it's - up to the continuations chained by it to decide whether to force their - arguments. - - - The default continuation ``identity`` is strict, so that return values - from a continuation-enabled computation will be forced. - - Example:: - - with continuations, lazify: - k = None - def setk(*args, cc): - nonlocal k - k = cc - return args[0] - def doit(): - lst = ['the call returned'] - *more, = call_cc[setk('A', 1/0)] - return lst + [more[0]] - assert doit() == ['the call returned', 'A'] - assert k('again') == ['the call returned', 'again'] - assert k('thrice', 1/0) == ['the call returned', 'thrice'] - - For a version with comments, see ``unpythonic/syntax/test/test_lazify.py``. - - **CAUTION**: Call-by-need is a low-level language feature that is difficult - to bolt on after the fact. Some things might not work. - - **CAUTION**: The functions in ``unpythonic.fun`` are lazify-aware (so that - e.g. curry and compose work with lazy functions), as are ``call`` and - ``callwith`` in ``unpythonic.misc``, but the rest of ``unpythonic`` is not. - - **CAUTION**: Argument passing by function call, and let-bindings are - currently the only binding constructs to which auto-lazification is applied. - """ - if syntax != "block": - raise SyntaxError("lazify is a block macro only") - - # Two-pass macro. - with dyn.let(_macro_expander=expander): - return _lazify(body=tree) - -# The `lazy` macro comes from `demo/promise.py` in `mcpyrate`. -def lazy(tree, *, syntax, **kw): # noqa: F811 - """[syntax, expr] Delay an expression (lazy evaluation). - - This macro injects a lambda to delay evaluation, and encapsulates - the result into a *promise* (an `unpythonic.lazyutil.Lazy` object). - - In Racket, this operation is known as `delay`. - """ - if syntax != "expr": - raise SyntaxError("lazy is an expr macro only") - - # Expand outside in. Ordering shouldn't matter here. - return _lazy(tree) - -def lazyrec(tree, *, syntax, **kw): # noqa: F811 - """[syntax, expr] Delay items in a container literal, recursively. - - Essentially, this distributes ``lazy[]`` into the items inside a literal - ``list``, ``tuple``, ``set``, ``frozenset``, ``unpythonic.collections.box`` - or ``unpythonic.llist.cons``, and into the values of a literal ``dict`` or - ``unpythonic.collections.frozendict``. - - Because this is a macro and must work by names only, only this fixed set of - container types is supported. - - The container itself is not lazified, only the items inside it are, to keep - the lazification from interfering with unpacking. This allows things such as - ``f(*lazyrec[(1*2*3, 4*5*6)])`` to work as expected. - - See also ``lazy[]`` (the effect on each item) and ``unpythonic.syntax.force`` - (the inverse of ``lazyrec[]``). - - For an atom, ``lazyrec[]`` has the same effect as ``lazy[]``:: - - lazyrec[dostuff()] --> lazy[dostuff()] - - For a container literal, ``lazyrec[]`` descends into it:: - - lazyrec[(2*21, 1/0)] --> (lazy[2*21], lazy[1/0]) - lazyrec[{'a': 2*21, 'b': 1/0}] --> {'a': lazy[2*21], 'b': lazy[1/0]} - - Constructor call syntax for container literals is also supported:: - - lazyrec[list(2*21, 1/0)] --> [lazy[2*21], lazy[1/0]] - - Nested container literals (with any combination of known types) are - processed recursively, for example:: - - lazyrec[((2*21, 1/0), (1+2+3, 4+5+6))] --> ((lazy[2*21], lazy[1/0]), - (lazy[1+2+3], lazy[4+5+6])) - """ - if syntax != "expr": - raise SyntaxError("lazyrec is an expr macro only") - - # Expand outside in. Ordering shouldn't matter here. - return _lazyrec(tree) - -# ----------------------------------------------------------------------------- - -def prefix(tree, *, syntax, **kw): # noqa: F811 - """[syntax, block] Write Python like Lisp: the first item is the operator. - - Example:: - - with prefix: - (print, "hello world") - t1 = (q, 1, 2, (3, 4), 5) - x = 42 - t2 = (q, 17, 23, x) - (print, t1, t2) - - Lexically inside a ``with prefix``: - - - A bare ``q`` at the head of a tuple is the quote operator. It increases - the quote level by one. - - It actually just tells the macro that this tuple (and everything in it, - recursively) is not a function call. - - Variables can be used as usual, there is no need to unquote them. - - - A bare ``u`` at the head of a tuple is the unquote operator, which - decreases the quote level by one. In other words, in:: - - with prefix: - t = (q, 1, 2, (u, print, 3), (print, 4), 5) - (print, t) - - the third item will call ``print(3)`` and evaluate to its return value - (in this case ``None``, since it's ``print``), whereas the fourth item - is a tuple with the two items ``(, 4)``. - - - Quote/unquote operators are parsed from the start of the tuple until - no more remain. Then any remaining items are either returned quoted - (if quote level > 0), or evaluated as a function call and replaced - by the return value. - - - How to pass named args:: - - from unpythonic.misc import call - - with prefix: - (f, kw(myarg=3)) # ``kw(...)`` (syntax, not really a function!) - call(f, myarg=3) # in a call(), kwargs are ok - f(myarg=3) # or just use Python's usual function call syntax - - One ``kw`` operator may include any number of named args (and **only** - named args). The tuple may have any number of ``kw`` operators. - - All named args are collected from ``kw`` operators in the tuple - when writing the final function call. If the same kwarg has been - specified by multiple ``kw`` operators, the rightmost definition wins. - - **Note**: Python itself prohibits having repeated named args in the **same** - ``kw`` operator, because it uses the function call syntax. If you get a - `SyntaxError: keyword argument repeated` with no useful traceback, - check any recent ``kw`` operators you have added in prefix blocks. - - A ``kw(...)`` operator in a quoted tuple (not a function call) is an error. - - Current limitations: - - - passing ``*args`` and ``**kwargs`` not supported. - - Workarounds: ``call(...)``; Python's usual function call syntax. - - - For ``*args``, to keep it lispy, maybe you want ``unpythonic.fun.apply``; - this allows syntax such as ``(apply, f, 1, 2, lst)``. - - **CAUTION**: This macro is experimental, not intended for production use. - """ - if syntax != "block": - raise SyntaxError("prefix is a block macro only") - - # Expand outside in. Any nested macros should get clean standard Python, - # not having to worry about tuples possibly denoting function calls. - return _prefix(block_body=tree) - -# ----------------------------------------------------------------------------- - -@parametricmacro -def test(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr/block] Make a test assertion. For writing automated tests. - - **Testing overview**: - - Use the `test[]`, `test_raises[]`, `test_signals[]`, `fail[]`, `error[]` - and `warn[]` macros inside a `with testset()`, as appropriate. - - See `testset` and `session` in the module `unpythonic.test.fixtures`, - as well as the docstrings of any constructs exported from that module. - - See below for tips and tricks. - - Finally, see the unit tests of `unpythonic` itself for examples. - - **Expression variant**: - - Syntax:: - - test[expr] - test[expr, message] - - The test succeeds if `expr` evaluates to truthy. The `message` - is used in forming the error message if the test fails or errors. - - If you want to assert just that an expression runs to completion - normally, and don't care about the return value:: - - from unpythonic.test.fixtures import returns_normally - - test[returns_normally(expr)] - test[returns_normally(expr), message] - - This can be useful for testing functions with side effects; sometimes - what is important is that the function completes normally. - - What `test[expr]` captures for reporting as "result" in the failure - message, if the test fails: - - - If a `the[...]` mark is present, the subexpression marked as `the[...]`. - At most one `the[]` may appear in a single `test[...]`. - - Else if `expr` is a comparison, the LHS (leftmost term in case of - a chained comparison). So e.g. `test[x < 3]` needs no annotation - to do the right thing. This is a common use case, hence automatic. - - Else the whole `expr`. - - The `the[...]` mark is useful in tests involving comparisons:: - - test[lower_limit < the[computeitem(...)]] - test[lower_limit < the[computeitem(...)] < upper_limit] - test[myconstant in the[computeset(...)]] - - If your interesting part is on the LHS, `the[]` is optional, although - allowed (to explicitly document intent). These have the same effect:: - - test[the[computeitem(...)] in myitems] - test[computeitem(...) in myitems] - - The `the[...]` mark passes the value through, and does not affect the - evaluation order of user code. - - The `the[]` mark can be imported as a macro from this module, so that - its appearance in your source code won't confuse `flake8`, and you'll - get a nice macro-expansion-time error if it accidentally appears outside - a `test[]` or `with test:`. - - **Block variant**: - - A test that requires statements (e.g. assignments) can be written as a - `with test` block:: - - with test: - body0 - ... - return expr # optional - - with test[message]: - body0 - ... - return expr # optional - - The test block is automatically lifted into a function, so it introduces - **a local scope**. Use the `nonlocal` or `global` declarations if you need - to mutate something defined on the outside. - - If there is a `return` at the top level of the block, that is the return - value from the test; it is what will be asserted. - - If there is no `return`, the test asserts that the block completes normally, - just like a `test[returns_normally(...)]` does for an expression. - - The asymmetry in syntax reflects the asymmetry between expressions and - statements in Python. Likewise, the fact that `with test` requires `return` - to return a value, but `test[...]` doesn't, is similar to the difference - between `def` and `lambda`. - - In the block variant, the "result" capture rules apply to the return value - designated by `return`. To override, the `the[]` mark can be used for - capturing the value of any one expression inside the block. The mark - doesn't have to be in the `return`. - - At most one `the[]` may appear in the same `with test` block. - - **Failure and error signaling**: - - Upon a test failure, `test[]` will *signal* a `TestFailure` using the - *cerror* (correctable error) protocol, via unpythonic's condition - system, which is a pythonification of Common Lisp's condition system. - See `unpythonic.conditions`. - - If a test fails to run to completion due to an uncaught exception or an - unhandled signal (e.g. an `error` or `cerror` condition), `TestError` - is signaled instead, so the caller can easily tell apart which case - occurred. - - Finally, when a `warn[]` runs, `TestWarning` is signaled. - - These condition types are defined in `unpythonic.test.fixtures`. - They inherit from `TestingException`, defined in the same module. - Beside the human-readable message, these exception types contain - attributes with programmatically inspectable information about - what happened. See the docstring of `TestingException`. - - *Signaling* a condition, instead of *raising* an exception, allows the - surrounding code (inside the test framework) to install a handler that - invokes the `proceed` restart (if there is such in scope), so upon a test - failure or error, the test suite resumes. - - **Disabling the signal barrier**: - - As implied above, `test[]` (likewise `with test:`) forms a barrier that - alerts the user about uncaught signals, and stops those signals from - propagating further. If your `with handlers` block that needs to see - the signal is outside the `test` invocation, or if allowing a signal to - go uncaught is part of normal operation (e.g. `warn` signals are often - not caught, because the only reason to do so is to muffle the warning), - use a `with catch_signals(False):` block (from the module - `unpythonic.test.fixtures`) to disable the signal barrier:: - - from unpythonic.test.fixtures import catch_signals - - with catch_signals(False): - test[...] - - Another way to avoid catching signals that should not be caught by the - test framework is to rearrange the `test[]` so that the expression being - asserted cannot result in an uncaught signal. For example, save the result - of a computation into a variable first, and then use it in the `test[]`, - instead of invoking that computation inside the `test[]`. See - `unpythonic.test.test_conditions` for examples. - - Exceptions are always caught by `test[]`, because exceptions do not support - resumption; unlike with signals, the inner level of the call stack is already - destroyed by the time the exception is caught by the test construct. - """ - if syntax not in ("expr", "block"): - raise SyntaxError("test is an expr and block macro only") - - # Two-pass macros. - with dyn.let(_macro_expander=expander): - if syntax == "expr": - if args: - raise SyntaxError("test[] in expression mode does not take macro arguments") - return _test_expr(tree) - else: # syntax == "block": - return _test_block(block_body=tree, args=args) - -@parametricmacro -def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr/block] Like `test`, but expect the expression to signal a condition. - - "Signal" as in `unpythonic.conditions.signal` and its sisters. - - Syntax:: - - test_signals[exctype, expr] - test_signals[exctype, expr, message] - - with test_signals[exctype]: - body0 - ... - - with test_signals[exctype, message]: - body0 - ... - - Example:: - - test_signals[ValueError, myfunc()] - test_signals[ValueError, myfunc(), "failure message"] - - The test succeeds, if `expr` signals a condition of type `exctype`, and the - signal propagates into the (implicit) handler inside the `test_signals[]` - construct. - - If `expr` returns normally, the test fails. - - If `expr` signals some other type of condition, or raises an exception, the - test errors. - - **Differences to `test[]`, `with test`**: - - As the focus of this construct is on signaling vs. returning normally, the - `the[]` mark is not supported. The block variant does not support `return`. - """ - if syntax not in ("expr", "block"): - raise SyntaxError("test_signals is an expr and block macro only") - - # Two-pass macros. - with dyn.let(_macro_expander=expander): - if syntax == "expr": - if args: - raise SyntaxError("test_signals[] in expression mode does not take macro arguments") - return _test_expr_signals(tree) - else: # syntax == "block": - return _test_block_signals(block_body=tree, args=args) - -@parametricmacro -def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 - """[syntax, expr/block] Like `test`, but expect the expression to raise an exception. - - Syntax:: - - test_raises[exctype, expr] - test_raises[exctype, expr, message] - - with test_raises[exctype]: - body0 - ... - - with test_raises[exctype, message]: - body0 - ... - - Example:: - - test_raises[TypeError, issubclass(1, int)] - test_raises[ValueError, myfunc()] - test_raises[ValueError, myfunc(), "failure message"] - - The test succeeds, if `expr` raises an exception of type `exctype`, and the - exception propagates into the (implicit) handler inside the `test_raises[]` - construct. - - If `expr` returns normally, the test fails. - - If `expr` signals a condition, or raises some other type of exception, the - test errors. - - **Differences to `test[]`, `with test`**: - - As the focus of this construct is on raising vs. returning normally, the - `the[]` mark is not supported. The block variant does not support `return`. - """ - if syntax not in ("expr", "block"): - raise SyntaxError("test_raises is an expr and block macro only") - - with dyn.let(_macro_expander=expander): - if syntax == "expr": - if args: - raise SyntaxError("test_raises[] in expression mode does not take macro arguments") - return _test_expr_raises(tree) - else: # syntax == "block": - return _test_block_raises(block_body=tree, args=args) - -def fail(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Produce a test failure, unconditionally. - - Useful to e.g. mark a line of code that should not be reached in automated - tests, reaching which is therefore a test failure. - - Usage:: - - fail["human-readable reason"] - - which has the same effect as:: - - test[False, "human-readable reason"] - - except in the case of `fail[]`, the error message generating machinery is - special-cased to omit the source code expression, because it explicitly - states that the intent of the "test" is not actually to perform a test. - - See also `error[]`, `warn[]`. - """ - if syntax != "expr": - raise SyntaxError("fail is an expr macro only") - - # Expand outside in. The ordering shouldn't matter here. - # The underlying `test` machinery needs to access the expander. - with dyn.let(_macro_expander=expander): - return _fail_expr(tree) - -def error(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Produce a test error, unconditionally. - - Useful to e.g. indicate to the user that an optional dependency that could - be used to run some integration test is not installed. - - Usage:: - - error["human-readable reason"] - - See also `warn[]`, `fail[]`. - """ - if syntax != "expr": - raise SyntaxError("error is an expr macro only") - - # Expand outside in. The ordering shouldn't matter here. - # The underlying `test` machinery needs to access the expander. - with dyn.let(_macro_expander=expander): - return _error_expr(tree) - -def warn(tree, *, syntax, expander, **kw): # noqa: F811 - """[syntax, expr] Produce a test warning, unconditionally. - - Useful to e.g. indicate that the Python interpreter or version the - tests are running on does not support a particular test, or to alert - about a non-essential TODO. - - A warning does not increase the failure count, so it will not cause - your CI workflow to break. - - Usage:: - - warn["human-readable reason"] - - See also `error[]`, `fail[]`. - """ - if syntax != "expr": - raise SyntaxError("warn is an expr macro only") - - # Expand outside in. The ordering shouldn't matter here. - # The underlying `test` machinery needs to access the expander. - with dyn.let(_macro_expander=expander): - return _warn_expr(tree) - -# ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index f88d9a80..bcd3c9f5 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -16,9 +16,64 @@ # "curryf" and "currycall" to detect an auto-curried expression with a final lambda. from ..fun import curry as curryf, _currycall as currycall + +def autocurry(tree, *, syntax, expander, **kw): # technically a list of trees, the body of the with block + """[syntax, block] Automatic currying. + + Usage:: + + from unpythonic.syntax import macros, autocurry + + with autocurry: + ... + + All **function calls** and **function definitions** (``def``, ``lambda``) + *lexically* inside the ``with autocurry`` block are automatically curried. + + **CAUTION**: Some builtins are uninspectable or may report their arities + incorrectly; in those cases, ``curry`` may fail, occasionally in mysterious + ways. + + The function ``unpythonic.arity.arities``, which ``unpythonic.fun.curry`` + internally uses, has a workaround for the inspectability problems of all + builtins in the top-level namespace (as of Python 3.7), but e.g. methods + of builtin types are not handled. + + Lexically inside a ``with autocurry`` block, the auto-curried function calls + will skip the curry if the function is uninspectable, instead of raising + ``TypeError`` as usual. + + Example:: + + from unpythonic.syntax import macros, autocurry + from unpythonic import foldr, composerc as compose, cons, nil, ll + + with autocurry: + def add3(a, b, c): + return a + b + c + assert add3(1)(2)(3) == 6 + assert add3(1, 2)(3) == 6 + assert add3(1)(2, 3) == 6 + assert add3(1, 2, 3) == 6 + + mymap = lambda f: foldr(compose(cons, f), nil) + double = lambda x: 2 * x + assert mymap(double, ll(1, 2, 3)) == ll(2, 4, 6) + + # The definition was auto-curried, so this works here too. + assert add3(1)(2)(3) == 6 + """ + if syntax != "block": + raise SyntaxError("autocurry is a block macro only") + + tree = expander.visit(tree) + + return _autocurry(block_body=tree) + + _iscurry = lambda name: name in ("curry", "currycall") -def autocurry(block_body): +def _autocurry(block_body): class AutoCurryTransformer(ASTTransformer): def transform(self, tree): # Ignore hygienically captured values, and don't recurse in them. diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 8ce06df6..a62fcc54 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -8,7 +8,7 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 -from mcpyrate import gensym +from mcpyrate import gensym, parametricmacro from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer @@ -82,6 +82,56 @@ # TODO: expander, so that we could register a function that deletes autoref markers # TODO: at the expander's global postprocess pass. +@parametricmacro +def autoref(tree, *, args, syntax, expander, **kw): + """Implicitly reference attributes of an object. + + Example:: + + e = env(a=1, b=2) + c = 3 + with autoref[e]: + a + b + c + + The transformation is applied in ``Load`` context only. ``Store`` and ``Del`` + are not redirected. + + Useful e.g. with the ``.mat`` file loader of SciPy. + + **CAUTION**: `autoref` is essentially the `with` construct of JavaScript + (which is completely different from Python's meaning of `with`), which is + nowadays deprecated. See: + + https://www.ecma-international.org/ecma-262/6.0/#sec-with-statement + https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/with + https://2ality.com/2011/06/with-statement.html + + **CAUTION**: The auto-reference `with` construct was deprecated in JavaScript + **for security reasons**. Since the autoref'd object **will hijack all name + lookups**, use `with autoref` only with an object you trust! + + **CAUTION**: `with autoref` also complicates static code analysis or makes it + outright infeasible, for the same reason. It is impossible to statically know + whether something that looks like a bare name in the source code is actually + a true bare name, or a reference to an attribute of the autoref'd object. + That status can also change at any time, since the lookup is dynamic, and + attributes can be added and removed dynamically. + """ + if syntax != "block": + raise SyntaxError("autoref is a block macro only") + if not args: + raise SyntaxError("autoref requires an argument, the object to be auto-referenced") + + target = kw.get("optional_vars", None) + + tree = expander.visit(tree) + + return _autoref(block_body=tree, args=args, asname=target) + +# -------------------------------------------------------------------------------- + @passthrough_lazy_args def _autoref_resolve(args): *objs, s = [force1(x) for x in args] @@ -90,7 +140,7 @@ def _autoref_resolve(args): return True, force1(getattr(o, s)) return False, None -def autoref(block_body, args, asname): +def _autoref(block_body, args, asname): if len(args) != 1: raise SyntaxError("expected exactly one argument, the expr to implicitly reference") # pragma: no cover if not block_body: diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 8fb7bf7a..ed720d07 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -6,20 +6,110 @@ The printing can be customized; see ``dbgprint_block`` and ``dbgprint_expr``. """ -__all__ = ["dbgprint_block", "dbg_block", - "dbgprint_expr", "dbg_expr"] +__all__ = ["dbg", "dbgprint_block", "dbgprint_expr"] from ast import Call, Name, keyword from mcpyrate.quotes import macros, q, u, a, t, h # noqa: F401 -from mcpyrate import unparse +from mcpyrate import parametricmacro, unparse from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer from ..dynassign import dyn, make_dynvar from ..misc import callsite_filename +@parametricmacro +def dbg(tree, *, args, syntax, expander, **kw): + """[syntax, expr/block] Debug-print expressions including their source code. + + **Expression variant**: + + Example:: + + dbg[25 + 17] # --> [file.py:100] (25 + 17): 42 + + The transformation is:: + + dbg[expr] --> dyn.dbgprint_expr(k, v, filename=__file__, lineno=xxx) + + where ``k`` is the source code of the expression and ``v`` is its value, + and `dyn` is `unpythonic.dynassign.dyn` (hygienically captured, so you + don't need to import it just to use the `dbg[]` macro). + + ``xxx`` is the original line number before macro expansion, if available + in the AST node of the expression, otherwise ``None``. (Some macros might + not care about inserting line numbers, because `mcpyrate` fixes any missing + line numbers in a postprocess step; this is why it might be missing at some + locations in any specific macro-enabled program.) + + A default implementation of the debug printer is provided and automatically + assigned as the default value for `dyn.dbgprint_expr`. + + To customize the debug printing, set your custom printer function to the + dynvar ``dbgprint_expr``, using `with dyn.let(dbgprint_expr=...)`. + + The custom function, beside performing any printing/logging as a side effect, + **must** return the value ``v``, so that surrounding an expression with + ``dbg[...]`` does not alter its value. + + If you want to use the default implementation as part of your customized one + (e.g. if you want to decorate that with some logging code), it is available as + `unpythonic.syntax.dbgprint_expr`. + + **Block variant**: + + Lexically within the block, any call to ``print`` (alternatively, if specified, + the optional custom print function), prints both the expression source code + and the corresponding value. + + A custom print function can be supplied as an argument. To implement a + custom print function, see the default implementation ``dbgprint_block`` + for the signature. + + If you want to use the default implementation as part of your customized one, + it is available as `unpythonic.syntax.dbgprint_block`. + + Examples:: + + with dbg: + x = 2 + print(x) # --> [file.py:100] x: 2 + + with dbg: + x = 2 + y = 3 + print(x, y) # --> [file.py:100] x: 2, y: 3 + print(x, y, sep="\n") # --> [file.py:100] x: 2 + # [file.py:100] y: 3 + + prt = lambda *args, **kwargs: print(*args) + with dbg[prt]: + x = 2 + prt(x) # --> ('x',) (2,) + print(x) # --> 2 + + with dbg[prt]: + x = 2 + y = 17 + prt(x, y, 1 + 2) # --> ('x', 'y', '(1 + 2)'), (2, 17, 3)) + + **CAUTION**: The source code is back-converted from the AST representation; + hence its surface syntax may look slightly different to the original (e.g. + extra parentheses). See ``mcpyrate.unparse``. + """ + if syntax not in ("expr", "block"): + raise SyntaxError("dbg is an expr and block macro only") + + tree = expander.visit(tree) + + if syntax == "expr": + return _dbg_expr(tree) + else: # syntax == "block": + return _dbg_block(body=tree, args=args) + +# -------------------------------------------------------------------------------- + def dbgprint_block(ks, vs, *, filename=None, lineno=None, sep=", ", **kwargs): """Default debug printer for the ``dbg`` macro, block variant. @@ -71,7 +161,7 @@ def dbgprint_block(ks, vs, *, filename=None, lineno=None, sep=", ", **kwargs): else: print(header + sep.join(f"{k}: {v}" for k, v in zip(ks, vs)), **kwargs) -def dbg_block(body, args): +def _dbg_block(body, args): if args: # custom print function hook # TODO: add support for Attribute to support using a method as a custom print function # (the problem is we must syntactically find matches in the AST, and AST nodes don't support comparison) @@ -100,6 +190,8 @@ def transform(self, tree): return self.generic_visit(tree) return DbgBlockTransformer().visit(body) +# -------------------------------------------------------------------------------- + def dbgprint_expr(k, v, *, filename, lineno): """Default debug printer for the ``dbg`` macro, expression variant. @@ -137,7 +229,7 @@ def dbgprint_expr(k, v, *, filename, lineno): print(f"[{filename}:{lineno}] {k}: {v}") return v # IMPORTANT! (passthrough; debug printing is a side effect) -def dbg_expr(tree): +def _dbg_expr(tree): ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] # Careful here! We must `h[]` the `dyn`, but not `dbgprint_expr` itself, diff --git a/unpythonic/syntax/forall.py b/unpythonic/syntax/forall.py index e497174a..b4423e75 100644 --- a/unpythonic/syntax/forall.py +++ b/unpythonic/syntax/forall.py @@ -15,7 +15,7 @@ from ..amb import insist, deny # for re-export only # noqa: F401 -def forall(exprs): +def forall(tree, *, syntax, expander, **kw): """[syntax, expr] Nondeterministic evaluation. Fully based on AST transformation, with real lexical variables. @@ -32,6 +32,14 @@ def forall(exprs): assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), (8, 15, 17), (9, 12, 15), (12, 16, 20)) """ + if syntax != "expr": + raise SyntaxError("forall is an expr macro only") + + tree = expander.visit(tree) + + return _forall(exprs=tree) + +def _forall(exprs): if type(exprs) is not Tuple: # pragma: no cover, let's not test macro expansion errors. raise SyntaxError("forall body: expected a sequence of comma-separated expressions") itemno = 0 diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index 866d16bc..2bc1e0fa 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -8,14 +8,47 @@ from mcpyrate.quotes import macros, q, a # noqa: F811, F401 -from .letdo import implicit_do, let +from .letdo import implicit_do, _let -def aif(tree): +from ..dynassign import dyn + +# -------------------------------------------------------------------------------- + +def aif(tree, *, syntax, expander, **kw): + """[syntax, expr] Anaphoric if. + + Usage:: + + aif[test, then, otherwise] + + aif[[pre, ..., test], + [post_true, ..., then], # "then" branch + [post_false, ..., otherwise]] # "otherwise" branch + + Inside the ``then`` and ``otherwise`` branches, the magic identifier ``it`` + (which is always named literally ``it``) refers to the value of ``test``. + + This expands into a ``let`` and an expression-form ``if``. + + Each part may consist of multiple expressions by using brackets around it; + those brackets create a `do` environment (see `unpythonic.syntax.do`). + + To represent a single expression that is a literal list, use extra + brackets: ``[[1, 2, 3]]``. + """ + if syntax != "expr": + raise SyntaxError("aif is an expr macro only") + + # Expand outside-in, but the implicit do[] needs the expander. + with dyn.let(_macro_expander=expander): + return _aif(tree) + +def _aif(tree): test, then, otherwise = [implicit_do(x) for x in tree.elts] bindings = [q[(it, a[test])]] body = q[a[then] if it else a[otherwise]] # TODO: we should use a hygienically captured macro here. - return let(bindings, body) + return _let(bindings, body) # TODO: `mcpyrate` has a rudimentary capability like Racket's "syntax-parameterize". # TODO: Make `it` a name macro that errors out unless it appears inside an `aif`. @@ -31,7 +64,39 @@ def __repr__(self): # pragma: no cover, we have a repr just in case one of thes return "" it = it() -def cond(tree): +# -------------------------------------------------------------------------------- + +def cond(tree, *, syntax, expander, **kw): + """[syntax, expr] Lispy cond; like "a if p else b", but has "elif". + + Usage:: + + cond[test1, then1, + test2, then2, + ... + otherwise] + + cond[[pre1, ..., test1], [post1, ..., then1], + [pre2, ..., test2], [post2, ..., then2], + ... + [postn, ..., otherwise]] + + This allows human-readable multi-branch conditionals in an expression position. + + Each part may consist of multiple expressions by using brackets around it; + those brackets create a `do` environment (see `unpythonic.syntax.do`). + + To represent a single expression that is a literal list, use extra + brackets: ``[[1, 2, 3]]``. + """ + if syntax != "expr": + raise SyntaxError("cond is an expr macro only") + + # Expand outside-in, but the implicit do[] needs the expander. + with dyn.let(_macro_expander=expander): + return _cond(tree) + +def _cond(tree): if type(tree) is not Tuple: raise SyntaxError("Expected cond[test1, then1, test2, then2, ..., otherwise]") # pragma: no cover def build(elts): diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 19383633..27468a76 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -3,7 +3,8 @@ __all__ = ["multilambda", "namedlambda", - "f", # for quicklambda + "f", + "quicklambda", "envify"] from ast import (Lambda, List, Name, Assign, Subscript, Call, FunctionDef, @@ -14,6 +15,7 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym +from mcpyrate.expander import MacroExpander from mcpyrate.quotes import is_captured_value from mcpyrate.splicing import splice_expression from mcpyrate.utils import extract_bindings @@ -24,12 +26,208 @@ from ..env import env from .astcompat import getconstant, Str, NamedExpr -from .letdo import do +from .letdo import _do from .letdoutil import islet, isenvassign, UnexpandedLetView, UnexpandedEnvAssignView, ExpandedDoView from .util import (is_decorated_lambda, isx, has_deco, destructure_decorated_lambda, detect_lambda) -def multilambda(block_body): +# -------------------------------------------------------------------------------- +# Macro interface + +def multilambda(tree, *, syntax, expander, **kw): + """[syntax, block] Supercharge your lambdas: multiple expressions, local variables. + + For all ``lambda`` lexically inside the ``with multilambda`` block, + ``[...]`` denotes a multiple-expression body with an implicit ``do``:: + + lambda ...: [expr0, ...] --> lambda ...: do[expr0, ...] + + Only the outermost set of brackets around the body of a ``lambda`` denotes + a multi-expression body; the rest are interpreted as lists, as usual. + + Examples:: + + with multilambda: + echo = lambda x: [print(x), x] + assert echo("hi there") == "hi there" + + count = let[(x, 0)][ + lambda: [x << x + 1, + x]] + assert count() == 1 + assert count() == 2 + + mk12 = lambda: [[1, 2]] + assert mk12() == [1, 2] + + For local variables, see ``do``. + """ + if syntax != "block": + raise SyntaxError("multilambda is a block macro only") + + # Expand outside in. + # multilambda should expand first before any let[], do[] et al. that happen + # to be inside the block, to avoid misinterpreting implicit lambdas + # generated by those constructs. + with dyn.let(_macro_expander=expander): # implicit do (extra bracket notation) needs this. + return _multilambda(block_body=tree) + +def namedlambda(tree, *, syntax, expander, **kw): + """[syntax, block] Name lambdas implicitly. + + Lexically inside a ``with namedlambda`` block, any literal ``lambda`` + that is assigned to a name using one of the supported assignment forms + is named to have the name of the LHS of the assignment. The name is + captured at macro expansion time. + + Naming modifies the original function object. + + We support: + + - Single-item assignments to a local name, ``f = lambda ...: ...`` + + - Named expressions (a.k.a. walrus operator, Python 3.8+), + ``f := lambda ...: ...`` + + - Assignments to unpythonic environments, ``f << (lambda ...: ...)`` + + - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any + let syntax supported by unpythonic (here using the haskelly let-in + just as an example). + + Support for other forms of assignment might or might not be added in a + future version. + + Example:: + + with namedlambda: + f = lambda x: x**3 # assignment: name as "f" + + let[(x, 42), (g, None), (h, None)][[ + g << (lambda x: x**2), # env-assignment: name as "g" + h << f, # still "f" (no literal lambda on RHS) + (g(x), h(x))]] + + foo = let[(f7, lambda x: x) in f7] # let-binding: name as "f7" + + The naming is performed using the function ``unpythonic.misc.namelambda``, + which will update ``__name__``, ``__qualname__`` and ``__code__.co_name``. + """ + if syntax != "block": + raise SyntaxError("namedlambda is a block macro only") + + # Two-pass macro. We pass in the expander to allow the macro to decide when to recurse. + with dyn.let(_macro_expander=expander): + return _namedlambda(block_body=tree) + +def f(tree, *, syntax, expander, **kw): + """[syntax, expr] Underscore notation (quick lambdas) for Python. + + Usage:: + + f[body] + + The ``f[]`` macro creates a lambda. Each underscore in ``body`` + introduces a new parameter. + + Example:: + + func = f[_ * _] + + expands to:: + + func = lambda a0, a1: a0 * a1 + + The underscore is interpreted magically by ``f[]``; but ``_`` itself + is not a macro, and has no special meaning outside ``f[]``. The underscore + does **not** need to be imported for ``f[]`` to recognize it. + + The macro does not descend into any nested ``f[]``. + """ + if syntax != "expr": + raise SyntaxError("f is an expr macro only") + + # What's my name in the current expander? (There may be several names.) + # https://github.com/Technologicat/mcpyrate/blob/master/doc/quasiquotes.md#hygienic-macro-recursion + bindings = extract_bindings(expander.bindings, f) + mynames = list(bindings.keys()) + + return _f(tree, mynames) + +def quicklambda(tree, *, syntax, expander, **kw): + """[syntax, block] Make ``f`` quick lambdas expand first. + + To be able to transform correctly, the block macros in ``unpythonic.syntax`` + that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all + ``lambda`` definitions written with Python's standard ``lambda``. + + However, the ``f`` macro uses the syntax ``f[...]``, which (to the analyzer) + does not look like a lambda definition. This macro changes the expansion + order, forcing any ``f[...]`` lexically inside the block to expand before + any other macros do. + + Any expression of the form ``f[...]``, where ``f`` is any name bound in the + current macro expander to the macro `unpythonic.syntax.f`, is understood as + a quick lambda. (In plain English, this respects as-imports of the macro ``f``.) + + Example - a quick multilambda:: + + from unpythonic.syntax import macros, multilambda, quicklambda, f, local + + with quicklambda, multilambda: + func = f[[local[x << _], + local[y << _], + x + y]] + assert func(1, 2) == 3 + + (This is of course rather silly, as an unnamed argument can only be mentioned + once. If we're giving names to them, a regular ``lambda`` is shorter to write. + The point is, this combo is now possible.) + """ + if syntax != "block": + raise SyntaxError("quicklambda is a block macro only") + + # This macro expands outside in. + # + # In `mcpyrate`, expander instances are cheap - so we create a second expander + # to which we register only the `f` macro, under whatever names it appears in + # the original expander. Thus it leaves all other macros alone. This is the + # official `mcpyrate` way to immediately expand only some particular macros + # inside the current macro invocation. + bindings = extract_bindings(expander.bindings, f) + return MacroExpander(bindings, expander.filename).visit(tree) + +def envify(tree, *, syntax, expander, **kw): + """[syntax, block] Make formal parameters live in an unpythonic env. + + The purpose is to allow overwriting formals using unpythonic's + expression-assignment ``name << value``. The price is that the references + to the arguments are copied into an env whenever an envified function is + entered. + + Example - PG's accumulator puzzle (http://paulgraham.com/icad.html):: + + with envify: + def foo(n): + return lambda i: n << n + i + + Or even shorter:: + + with autoreturn, envify: + def foo(n): + lambda i: n << n + i + """ + if syntax != "block": + raise SyntaxError("envify is a block macro only") + + # Two-pass macro. + with dyn.let(_macro_expander=expander): + return _envify(block_body=tree) + +# -------------------------------------------------------------------------------- +# Syntax transformers + +def _multilambda(block_body): class MultilambdaTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -45,14 +243,14 @@ def transform(self, tree): # - but recurse manually into each *do item*; these are explicit # user-provided code so we should transform them bodys = self.visit(bodys) - tree.body = do(bodys) # insert the do, with the implicit lambdas + tree.body = _do(bodys) # insert the do, with the implicit lambdas return tree # multilambda should expand first before any let[], do[] et al. that happen # to be inside the block, to avoid misinterpreting implicit lambdas # generated by those constructs. return MultilambdaTransformer().visit(block_body) -def namedlambda(block_body): +def _namedlambda(block_body): def issingleassign(tree): return type(tree) is Assign and len(tree.targets) == 1 and type(tree.targets[0]) is Name @@ -184,13 +382,7 @@ def transform(self, tree): # # Used under the MIT license. # Copyright (c) 2013-2018, Li Haoyi, Justin Holmgren, Alberto Berti and all the other contributors. -def f(tree): - # What's my name in the current expander? (There may be several names.) - # https://github.com/Technologicat/mcpyrate/blob/master/doc/quasiquotes.md#hygienic-macro-recursion - # TODO: doesn't currently work because this `f` is the syntax transformer, not the `f[]` macro. - bindings = extract_bindings(dyn._macro_expander.bindings, f) - mynames = list(bindings.keys()) - +def _f(tree, mynames=()): class UnderscoreTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -212,7 +404,7 @@ def transform(self, tree): tree.args.args = [arg(arg=x) for x in used_names] return tree -def envify(block_body): +def _envify(block_body): # first pass, outside-in userlambdas = detect_lambda(block_body) @@ -274,8 +466,8 @@ def isourupdate(thecall): # the name should revert to mean the formal parameter. # # inject a do[] and reuse its env - tree.body = do(List(elts=[q[n["_here_"]], - tree.body])) + tree.body = _do(List(elts=[q[n["_here_"]], + tree.body])) view = ExpandedDoView(tree.body) # view.body: [(lambda e14: ...), ...] ename = view.body[0].args.args[0].arg # do[] environment name theupdate = Attribute(value=q[n[ename]], attr="update") diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index ff6e6b9b..9b1e623c 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -19,8 +19,404 @@ # ----------------------------------------------------------------------------- +# The `lazy` macro comes from `demo/promise.py` in `mcpyrate`. +def lazy(tree, *, syntax, **kw): + """[syntax, expr] Delay an expression (lazy evaluation). + + This macro injects a lambda to delay evaluation, and encapsulates + the result into a *promise* (an `unpythonic.lazyutil.Lazy` object). + + In Racket, this operation is known as `delay`. + """ + if syntax != "expr": + raise SyntaxError("lazy is an expr macro only") + + # Expand outside in. Ordering shouldn't matter here. + return _lazy(tree) + +def lazyrec(tree, *, syntax, **kw): + """[syntax, expr] Delay items in a container literal, recursively. + + Essentially, this distributes ``lazy[]`` into the items inside a literal + ``list``, ``tuple``, ``set``, ``frozenset``, ``unpythonic.collections.box`` + or ``unpythonic.llist.cons``, and into the values of a literal ``dict`` or + ``unpythonic.collections.frozendict``. + + Because this is a macro and must work by names only, only this fixed set of + container types is supported. + + The container itself is not lazified, only the items inside it are, to keep + the lazification from interfering with unpacking. This allows things such as + ``f(*lazyrec[(1*2*3, 4*5*6)])`` to work as expected. + + See also ``lazy[]`` (the effect on each item) and ``unpythonic.syntax.force`` + (the inverse of ``lazyrec[]``). + + For an atom, ``lazyrec[]`` has the same effect as ``lazy[]``:: + + lazyrec[dostuff()] --> lazy[dostuff()] + + For a container literal, ``lazyrec[]`` descends into it:: + + lazyrec[(2*21, 1/0)] --> (lazy[2*21], lazy[1/0]) + lazyrec[{'a': 2*21, 'b': 1/0}] --> {'a': lazy[2*21], 'b': lazy[1/0]} + + Constructor call syntax for container literals is also supported:: + + lazyrec[list(2*21, 1/0)] --> [lazy[2*21], lazy[1/0]] + + Nested container literals (with any combination of known types) are + processed recursively, for example:: + + lazyrec[((2*21, 1/0), (1+2+3, 4+5+6))] --> ((lazy[2*21], lazy[1/0]), + (lazy[1+2+3], lazy[4+5+6])) + """ + if syntax != "expr": + raise SyntaxError("lazyrec is an expr macro only") + + # Expand outside in. Ordering shouldn't matter here. + return _lazyrec(tree) + +def lazify(tree, *, syntax, expander, **kw): + """[syntax, block] Call-by-need for Python. + + In a ``with lazify`` block, function arguments are evaluated only when + actually used, at most once each, and in the order in which they are + actually used. Promises are automatically forced on access. + + Automatic lazification applies to arguments in function calls and to + let-bindings, since they play a similar role. **No other binding forms + are auto-lazified.** + + Automatic lazification uses the ``lazyrec[]`` macro, which recurses into + certain types of container literals, so that the lazification will not + interfere with unpacking. See its docstring for details. + + Comboing with other block macros in ``unpythonic.syntax`` is supported, + including ``curry`` and ``continuations``. + + Silly contrived example:: + + with lazify: + def my_if(p, a, b): + if p: + return a # b never evaluated in this code path... + else: + return b # a never evaluated in this code path... + + # ...hence the divisions by zero here are never performed. + assert my_if(True, 23, 1/0) == 23 + assert my_if(False, 1/0, 42) == 42 + + Note ``my_if`` is a run-of-the-mill runtime function, not a macro. Only the + ``with lazify`` is imbued with any magic. + + Like ``with continuations``, no state or context is associated with a + ``with lazify`` block, so lazy functions defined in one block may call + those defined in another. Calls between lazy and strict code are also + supported (in both directions), without requiring any extra effort. + + Evaluation of each lazified argument is guaranteed to occur at most once; + the value is cached. Order of evaluation of lazy arguments is determined + by the (dynamic) order in which the lazy code actually uses them. + + Essentially, the above code expands into:: + + from unpythonic.syntax import macros, lazy + from unpythonic.syntax import force + + def my_if(p, a, b): + if force(p): + return force(a) + else: + return force(b) + assert my_if(lazy[True], lazy[23], lazy[1/0]) == 23 + assert my_if(lazy[False], lazy[1/0], lazy[42]) == 42 + + plus some clerical details to allow lazy and strict code to be mixed. + + Just passing through a lazy argument to another lazy function will + not trigger evaluation, even when it appears in a computation inlined + to the argument list:: + + with lazify: + def g(a, b): + return a + def f(a, b): + return g(2*a, 3*b) + assert f(21, 1/0) == 42 + + The division by zero is never performed, because the value of ``b`` is + not needed to compute the result (worded less magically, that promise is + never forced in the code path that produces the result). Essentially, + the above code expands into:: + + from unpythonic.syntax import macros, lazy + from unpythonic.syntax import force + + def g(a, b): + return force(a) + def f(a, b): + return g(lazy[2*force(a)], lazy[3*force(b)]) + assert f(lazy[21], lazy[1/0]) == 42 + + This relies on the magic of closures to capture f's ``a`` and ``b`` into + the promises. + + But be careful; **assignments are not auto-lazified**, so the following does + **not** work:: + + with lazify: + def g(a, b): + return a + def f(a, b): + c = 3*b # not in an arglist, b gets evaluated! + return g(2*a, c) + assert f(21, 1/0) == 42 + + To avoid that, explicitly wrap the computation into a ``lazy[]``. For why + assignment RHSs are not auto-lazified, see the section on pitfalls below. + + In calls, bare references (name, subscript, attribute) are detected and for + them, re-thunking is skipped. For example:: + + def g(a): + return a + def f(a): + return g(a) + assert f(42) == 42 + + expands into:: + + def g(a): + return force(a) + def f(a): + return g(a) # <-- no lazy[force(a)] since "a" is just a name + assert f(lazy[42]) == 42 + + When resolving references, subscripts and attributes are forced just enough + to obtain the containing object from a promise, if any; for example, the + elements of a list ``lst`` will not be evaluated just because the user code + happens to use ``lst.append(...)``; this only forces the object ``lst`` + itself. + + A ``lst`` appearing by itself evaluates the whole list. Similarly, ``lst[0]`` + by itself evaluates only the first element, and ``lst[:-1]`` by itself + evaluates all but the last element. The index expression in a subscript is + fully forced, because its value is needed to determine which elements of the + subscripted container are to be accessed. + + **Mixing lazy and strict code** + + Lazy code is allowed to call strict functions and vice versa, without + requiring any additional effort. + + Keep in mind what this implies: when calling a strict function, any arguments + given to it will be evaluated! + + In the other direction, when calling a lazy function from strict code, the + arguments are evaluated by the caller before the lazy code gets control. + The lazy code gets just the evaluated values. + + If you have, in strict code, an argument expression you want to pass lazily, + use syntax like ``f(lazy[...], ...)``. If you accidentally do this in lazy + code, it shouldn't break anything; ``with lazify`` detects any argument + expressions that are already promises, and just passes them through. + + **Forcing promises manually** + + This is mainly useful if you ``lazy[]`` or ``lazyrec[]`` something explicitly, + and want to compute its value outside a ``with lazify`` block. + + We provide the functions ``force1`` and ``force``. + + Using ``force1``, if ``x`` is a ``lazy[]`` promise, it will be forced, + and the resulting value is returned. If ``x`` is not a promise, + ``x`` itself is returned, à la Racket. + + The function ``force``, in addition, descends into containers (recursively). + When an atom ``x`` (i.e. anything that is not a container) is encountered, + it is processed using ``force1``. + + Mutable containers are updated in-place; for immutables, a new instance is + created. Any container with a compatible ``collections.abc`` is supported. + (See ``unpythonic.collections.mogrify`` for details.) In addition, as + special cases ``unpythonic.collections.box`` and ``unpythonic.llist.cons`` + are supported. + + **Tips, tricks and pitfalls** + + You can mix and match bare data values and promises, since ``force(x)`` + evaluates to ``x`` when ``x`` is not a promise. + + So this is just fine:: + + with lazify: + def f(x): + x = 2*21 # assign a bare data value + print(x) # the implicit force(x) evaluates to x + f(17) + + If you want to manually introduce a promise, use ``lazy[]``:: + + from unpythonic.syntax import macros, lazify, lazy + + with lazify: + def f(x): + x = lazy[2*21] # assign a promise + print(x) # the implicit force(x) evaluates the promise + f(17) + + If you have a container literal and want to lazify it recursively in a + position that does not auto-lazify, use ``lazyrec[]`` (see its docstring + for details):: + + from unpythonic.syntax import macros, lazify, lazyrec + + with lazify: + def f(x): + return x[:-1] + lst = lazyrec[[1, 2, 3/0]] + assert f(lst) == [1, 2] + + For non-literal containers, use ``lazy[]`` for each item as appropriate:: + + def f(lst): + lst.append(lazy["I'm lazy"]) + lst.append(lazy["Don't call me lazy, I'm just evaluated later!"]) + + Keep in mind, though, that ``lazy[]`` will introduce a lambda, so there's + the usual pitfall:: + + from unpythonic.syntax import macros, lazify, lazy + + with lazify: + lst = [] + for x in range(3): # DANGER: only one "x", mutated imperatively + lst.append(lazy[x]) # all these closures capture the same "x" + print(lst[0]) # 2 + print(lst[1]) # 2 + print(lst[2]) # 2 + + So to capture the value instead of the name, use the usual workaround, + the wrapper lambda (here written more readably as a let, which it really is):: + + from unpythonic.syntax import macros, lazify, lazy, let + + with lazify: + lst = [] + for x in range(3): + lst.append(let[(y, x) in lazy[y]]) + print(lst[0]) # 0 + print(lst[1]) # 1 + print(lst[2]) # 2 + + Be careful not to ``lazy[]`` or ``lazyrec[]`` too much:: + + with lazify: + a = 10 + a = lazy[2*a] # 20, right? + print(a) # crash! + + Why does this example crash? The expanded code is:: + + with lazify: + a = 10 + a = lazy[2*force(a)] + print(force(a)) + + The ``lazy[]`` sets up a promise, which will force ``a`` *at the time when + the containing promise is forced*, but at that time the name ``a`` points + to a promise, which will force... + + The fundamental issue is that ``a = 2*a`` is an imperative update; if you + need to do that, just let Python evaluate the RHS normally (i.e. use the + value the name ``a`` points to *at the time when the RHS runs*). + + Assigning a lazy value to a new name evaluates it, because any read access + triggers evaluation:: + + with lazify: + def g(x): + y = x # the "x" on the RHS triggers the implicit force + print(y) # bare data value + f(2*21) + + Inspired by Haskell, Racket's (delay) and (force), and lazy/racket. + + **Combos** + + Introducing the *HasThon* programming language (it has 100% more Thon than + popular brands):: + + with autocurry, lazify: # or continuations, autocurry, lazify if you want those + def add2first(a, b, c): + return a + b + assert add2first(2)(3)(1/0) == 5 + + def f(a, b): + return a + assert let[((c, 42), + (d, 1/0)) in f(c)(d)] == 42 + assert letrec[((c, 42), + (d, 1/0), + (e, 2*c)) in f(e)(d)] == 84 + + assert letrec[((c, 42), + (d, 1/0), + (e, 2*c)) in [local[x << f(e)(d)], + x/4]] == 21 + + Works also with continuations. Rules: + + - Also continuations are transformed into lazy functions. + + - ``cc`` built by chain_conts is treated as lazy, **itself**; then it's + up to the continuations chained by it to decide whether to force their + arguments. + + - The default continuation ``identity`` is strict, so that return values + from a continuation-enabled computation will be forced. + + Example:: + + with continuations, lazify: + k = None + def setk(*args, cc): + nonlocal k + k = cc + return args[0] + def doit(): + lst = ['the call returned'] + *more, = call_cc[setk('A', 1/0)] + return lst + [more[0]] + assert doit() == ['the call returned', 'A'] + assert k('again') == ['the call returned', 'again'] + assert k('thrice', 1/0) == ['the call returned', 'thrice'] + + For a version with comments, see ``unpythonic/syntax/test/test_lazify.py``. + + **CAUTION**: Call-by-need is a low-level language feature that is difficult + to bolt on after the fact. Some things might not work. + + **CAUTION**: The functions in ``unpythonic.fun`` are lazify-aware (so that + e.g. curry and compose work with lazy functions), as are ``call`` and + ``callwith`` in ``unpythonic.misc``, but the rest of ``unpythonic`` is not. + + **CAUTION**: Argument passing by function call, and let-bindings are + currently the only binding constructs to which auto-lazification is applied. + """ + if syntax != "block": + raise SyntaxError("lazify is a block macro only") + + # Two-pass macro. + with dyn.let(_macro_expander=expander): + return _lazify(body=tree) + +# ----------------------------------------------------------------------------- + # lazy: syntax transformer, lazify a single expression -def lazy(tree): +def _lazy(tree): return q[h[Lazy](lambda: a[tree])] # lazyrec: syntax transformer, recursively lazify elements in container literals @@ -91,7 +487,7 @@ def lazy(tree): unexpanded_lazy_name = "lazy" expanded_lazy_name = "Lazy" -def lazyrec(tree): +def _lazyrec(tree): # This helper doesn't need to recurse, so we don't need `ASTTransformer` here. def transform(tree): if type(tree) in (Tuple, List, Set): @@ -111,7 +507,7 @@ def transform(tree): # TODO: Doing so renames the macro, so detection needs to be adjusted. # TODO: It must also be bound in the current expander for hygienic macro capture to work. # tree = q[h[lazy][a[tree]]] - tree = lazy(tree) + tree = _lazy(tree) return tree def lazify_ctorcall(tree, positionals="all", keywords="all"): @@ -179,7 +575,7 @@ def is_literal_container(tree, maps_only=False): # - don't lazify "for", the loop counter changes value imperatively (and usually rather rapidly) # full list: see unpythonic.syntax.scopeanalyzer.get_names_in_store_context (and the link therein) -def lazify(body): +def _lazify(body): # first pass, outside-in userlambdas = detect_lambda(body) @@ -262,7 +658,7 @@ def transform_arg(tree): self.withstate(tree, forcing_mode=("off" if isref else "full")) tree = self.visit(tree) if not isref: # (re-)thunkify expr; a reference can be passed as-is. - tree = lazyrec(tree) + tree = _lazyrec(tree) return tree def transform_starred(tree, dstarred=False): @@ -272,7 +668,7 @@ def transform_starred(tree, dstarred=False): # lazify items if we have a literal container # we must avoid lazifying any other exprs, since a Lazy cannot be unpacked. if is_literal_container(tree, maps_only=dstarred): - tree = lazyrec(tree) + tree = _lazyrec(tree) return tree # let bindings have a role similar to function arguments, so auto-lazify there diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 038afeb9..cc8ef97c 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -31,7 +31,7 @@ from mcpyrate.quotes import macros, q, u, n, a, t, h # noqa: F401 -from mcpyrate import gensym +from mcpyrate import gensym, parametricmacro from mcpyrate.markers import ASTMarker from mcpyrate.quotes import is_captured_value from mcpyrate.utils import NestingLevelTracker @@ -39,25 +39,306 @@ from ..dynassign import dyn from ..lispylet import _let as letf, _dlet as dletf, _blet as bletf -from ..seq import do as dof from ..misc import namelambda +from ..seq import do as dof +from .letdoutil import (isenvassign, UnexpandedEnvAssignView, + UnexpandedLetView, canonize_bindings) from .scopeanalyzer import scoped_transform -from .letdoutil import isenvassign, UnexpandedEnvAssignView -def let(bindings, body): +# -------------------------------------------------------------------------------- +# Macro interface internal helper + +# NOTE: At the macro interface, the invocations `let()[...]` (empty args) +# and `let[...]` (no args) were indistinguishable in MacroPy. This was a +# problem, because it might be that the user wrote the body but simply +# forgot to put anything in the parentheses. (There's `do[]` if you need +# a `let` without making any bindings.) +# +# In `mcpyrate`, `let()[...]` is a syntax error. The preferred syntax, +# when using macro arguments, is `let[...][...]`. When this is not +# possible (in decorator position up to Python 3.8), then `let(...)[...]` +# is acceptable. But empty brackets/parentheses are not accepted. Thus, +# we will have an empty `args` list only when there are no brackets/parentheses +# for the macro arguments part. +# +# So when `args` is empty, this function assumes haskelly let syntax +# `let[(...) in ...]` or `let[..., where(...)]`. In these cases, +# both the bindings and the body reside inside the brackets (i.e., +# in the AST contained in the `tree` argument). +# +# Since the brackets/parentheses must be deleted when no macro arguments +# are given, this is now the correct assumption to make. +# +# But note that if needed elsewhere, `mcpyrate` has the `invocation` kwarg +# in the macro interface that gives a copy of the whole macro invocation +# node (so we could see the exact original syntax). +# +# allow_call_in_name_position: used by let_syntax to allow template definitions. +def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, allow_call_in_name_position=False): + with dyn.let(_macro_expander=macro_expander): # implicit do (extra bracket notation) needs this. + if args: + bs = canonize_bindings(args, allow_call_in_name_position=allow_call_in_name_position) + return let_transformer(bindings=bs, body=tree) + # haskelly syntax, let[(...) in ...], let[..., where(...)] + view = UnexpandedLetView(tree) # note "tree" here is only the part inside the brackets + return let_transformer(bindings=view.bindings, body=view.body) + +# -------------------------------------------------------------------------------- +# Macro interface - expr macros + +@parametricmacro +def let(tree, *, args, syntax, expander, **kw): + """[syntax, expr] Introduce expression-local variables. + + This is sugar on top of ``unpythonic.lispylet.let``. + + Usage:: + + let[(k0, v0), ...][body] + let[(k0, v0), ...][[body0, ...]] + + where ``body`` is an expression. The names bound by ``let`` are local; + they are available in ``body``, and do not exist outside ``body``. + + Alternative haskelly syntax is also available:: + + let[((k0, v0), ...) in body] + let[((k0, v0), ...) in [body0, ...]] + let[body, where((k0, v0), ...)] + let[[body0, ...], where((k0, v0), ...)] + + For a body with multiple expressions, use an extra set of brackets, + as shown above. This inserts a ``do``. Only the outermost extra brackets + are interpreted specially; all others in the bodies are interpreted + as usual, as lists. + + Note that in the haskelly syntax, the extra brackets for a multi-expression + body should enclose only the ``body`` part. + + Each ``name`` in the same ``let`` must be unique. + + Assignment to let-bound variables is supported with syntax such as ``x << 42``. + This is an expression, performing the assignment, and returning the new value. + + In a multiple-expression body, also an internal definition context exists + for local variables that are not part of the ``let``; see ``do`` for details. + + Technical points: + + - In reality, the let-bound variables live in an ``unpythonic.env``. + This macro performs the magic to make them look (and pretty much behave) + like lexical variables. + + - Compared to ``unpythonic.lispylet.let``, the macro version needs no quotes + around variable names in bindings. + + - The body is automatically wrapped in a ``lambda e: ...``. + + - For all ``x`` in bindings, the macro transforms lookups ``x --> e.x``. + + - Lexical scoping is respected (so ``let`` constructs can be nested) + by actually using a unique name (gensym) instead of just ``e``. + + - In the case of a multiple-expression body, the ``do`` transformation + is applied first to ``[body0, ...]``, and the result becomes ``body``. + """ + if syntax != "expr": + raise SyntaxError("let is an expr macro only") + + # The `let[]` family of macros expands inside out. + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _let) + +@parametricmacro +def letseq(tree, *, args, syntax, expander, **kw): + """[syntax, expr] Let with sequential binding (like Scheme/Racket let*). + + Like ``let``, but bindings take effect sequentially. Later bindings + shadow earlier ones if the same name is used multiple times. + + Expands to nested ``let`` expressions. + """ + if syntax != "expr": + raise SyntaxError("letseq is an expr macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _letseq) + +@parametricmacro +def letrec(tree, *, args, syntax, expander, **kw): + """[syntax, expr] Let with mutually recursive binding. + + Like ``let``, but bindings can see other bindings in the same ``letrec``. + + Each ``name`` in the same ``letrec`` must be unique. + + The definitions are processed sequentially, left to right. A definition + may refer to any previous definition. If ``value`` is callable (lambda), + it may refer to any definition, including later ones. + + This is useful for locally defining mutually recursive functions. + """ + if syntax != "expr": + raise SyntaxError("letrec is an expr macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _letrec) + +# ----------------------------------------------------------------------------- +# Macro interface - decorator versions, for "let over def". + +@parametricmacro +def dlet(tree, *, args, syntax, expander, **kw): + """[syntax, decorator] Decorator version of let, for 'let over def'. + + Example:: + + @dlet[(x, 0)] + def count(): + x << x + 1 + return x + assert count() == 1 + assert count() == 2 + + **CAUTION**: function arguments, local variables, and names declared as + ``global`` or ``nonlocal`` in a given lexical scope shadow names from the + ``let`` environment *for the entirety of that lexical scope*. (This is + modeled after Python's standard scoping rules.) + + **CAUTION**: assignment to the let environment is ``name << value``; + the regular syntax ``name = value`` creates a local variable in the + lexical scope of the ``def``. + """ + if syntax != "decorator": + raise SyntaxError("dlet is a decorator macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _dlet) + +@parametricmacro +def dletseq(tree, *, args, syntax, expander, **kw): + """[syntax, decorator] Decorator version of letseq, for 'letseq over def'. + + Expands to nested function definitions, each with one ``dlet`` decorator. + + Example:: + + @dletseq[(x, 1), + (x, x+1), + (x, x+2)] + def g(a): + return a + x + assert g(10) == 14 + """ + if syntax != "decorator": + raise SyntaxError("dletseq is a decorator macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _dletseq) + +@parametricmacro +def dletrec(tree, *, args, syntax, expander, **kw): + """[syntax, decorator] Decorator version of letrec, for 'letrec over def'. + + Example:: + + @dletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), + (oddp, lambda x: (x != 0) and evenp(x - 1))] + def f(x): + return evenp(x) + assert f(42) is True + assert f(23) is False + + Same cautions apply as to ``dlet``. + """ + if syntax != "decorator": + raise SyntaxError("dletrec is a decorator macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _dletrec) + +@parametricmacro +def blet(tree, *, args, syntax, expander, **kw): + """[syntax, decorator] def --> let block. + + Example:: + + @blet[(x, 21)] + def result(): + return 2*x + assert result == 42 + """ + if syntax != "decorator": + raise SyntaxError("blet is a decorator macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _blet) + +@parametricmacro +def bletseq(tree, *, args, syntax, expander, **kw): + """[syntax, decorator] def --> letseq block. + + Example:: + + @bletseq[(x, 1), + (x, x+1), + (x, x+2)] + def result(): + return x + assert result == 4 + """ + if syntax != "decorator": + raise SyntaxError("bletseq is a decorator macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _bletseq) + +@parametricmacro +def bletrec(tree, *, args, syntax, expander, **kw): + """[syntax, decorator] def --> letrec block. + + Example:: + + @bletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), + (oddp, lambda x: (x != 0) and evenp(x - 1))] + def result(): + return evenp(42) + assert result is True + + Because names inside a ``def`` have mutually recursive scope, + an almost equivalent pure Python solution (no macros) is:: + + from unpythonic.misc import call + + @call + def result(): + evenp = lambda x: (x == 0) or oddp(x - 1) + oddp = lambda x: (x != 0) and evenp(x - 1) + return evenp(42) + assert result is True + """ + if syntax != "decorator": + raise SyntaxError("bletrec is a decorator macro only") + + with dyn.let(_macro_expander=expander): + return _destructure_and_apply_let(tree, args, expander, _bletrec) + +# -------------------------------------------------------------------------------- +# Syntax transformers + +def _let(bindings, body): return _letimpl(bindings, body, "let") -def letseq(bindings, body): +def _letseq(bindings, body): if not bindings: return body first, *rest = bindings - # TODO: Could just return hygienic macro invocations, but that needs to be done - # TODO: where the macro interfaces are visible. See `unpythonic.syntax.simplelet` - # TODO: for how to do it. - return let([first], letseq(rest, body)) + # TODO: Could just return hygienic macro invocations. + # TODO: See `unpythonic.syntax.simplelet` for how to do it. + return _let([first], _letseq(rest, body)) -def letrec(bindings, body): +def _letrec(bindings, body): return _letimpl(bindings, body, "letrec") def _letimpl(bindings, body, mode): @@ -177,24 +458,24 @@ def envwrap(tree, envname): return lam # ----------------------------------------------------------------------------- -# Decorator versions, for "let over def". +# Syntax transformers for decorator versions, for "let over def". -def dlet(bindings, body): +def _dlet(bindings, body): return _dletimpl(bindings, body, "let", "decorate") -def dletseq(bindings, body): +def _dletseq(bindings, body): return _dletseqimpl(bindings, body, "decorate") -def dletrec(bindings, body): +def _dletrec(bindings, body): return _dletimpl(bindings, body, "letrec", "decorate") -def blet(bindings, body): +def _blet(bindings, body): return _dletimpl(bindings, body, "let", "call") -def bletseq(bindings, body): +def _bletseq(bindings, body): return _dletseqimpl(bindings, body, "call") -def bletrec(bindings, body): +def _bletrec(bindings, body): return _dletimpl(bindings, body, "letrec", "call") # Very similar to _letimpl, but perhaps more readable to keep these separate. @@ -284,7 +565,7 @@ def _dletseqimpl(bindings, body, kind): body.name = iname *rest, last = bindings - dletter = dlet if kind == "decorate" else blet + dletter = _dlet if kind == "decorate" else _blet innerdef = dletter([last], body) # optimization: in the final step, no need to generate a wrapper function @@ -310,6 +591,202 @@ def _dletseqimpl(bindings, body, kind): # ----------------------------------------------------------------------------- # Imperative code in expression position. Uses the "let" machinery. +# +# Macro interface + +def local(tree, *, syntax, **kw): + """[syntax] Declare a local name in a "do". + + Usage:: + + local[name << value] + + Only meaningful in a ``do[...]``, ``do0[...]``, or an implicit ``do`` + (extra bracket syntax). + + The declaration takes effect starting from next item in the ``do``, i.e. + the item that comes after the ``local[]``. It will not shadow nonlocal + variables of the same name in any earlier items of the same ``do``, and + in the item making the definition, the old bindings are still in effect + on the RHS. + + This means that if you want, you can declare a local ``x`` that takes its + initial value from a nonlocal ``x``, by ``local[x << x]``. Here the ``x`` + on the RHS is the nonlocal one (since the declaration has not yet taken + effect), and the ``x`` on the LHS is the name given to the new local variable + that only exists inside the ``do``. Any references to ``x`` in any further + items in the same ``do`` will point to the local ``x``. + """ + if syntax != "expr": + raise SyntaxError("local is an expr macro only") # pragma: no cover + return _local(tree) + +def delete(tree, *, syntax, **kw): + """[syntax] Delete a previously declared local name in a "do". + + Usage:: + + delete[name] + + Only meaningful in a ``do[...]``, ``do0[...]``, or an implicit ``do`` + (extra bracket syntax). + + The deletion takes effect starting from the next item; hence, the + deleted local variable will no longer shadow nonlocal variables of + the same name in any later items of the same `do`. + + Note ``do[]`` supports local variable deletion, but the ``let[]`` + constructs don't, by design. + """ + if syntax != "expr": + raise SyntaxError("delete is an expr macro only") # pragma: no cover + return _delete(tree) + +def do(tree, *, syntax, expander, **kw): + """[syntax, expr] Stuff imperative code into an expression position. + + Return value is the value of the last expression inside the ``do``. + See also ``do0``. + + Usage:: + + do[body0, ...] + + Example:: + + do[local[x << 42], + print(x), + x << 23, + x] + + This is sugar on top of ``unpythonic.seq.do``, but with some extra features. + + - To declare and initialize a local name, use ``local[name << value]``. + + The operator ``local`` is syntax, not really a function, and it + only exists inside a ``do``. There is also an operator ``delete`` + to delete a previously declared local name in the ``do``. + + Both ``local`` and ``delete``, if used, should be imported as macros. + + - By design, there is no way to create an uninitialized variable; + a value must be given at declaration time. Just use ``None`` + as an explicit "no value" if needed. + + - Names declared within the same ``do`` must be unique. Re-declaring + the same name is an expansion-time error. + + - To assign to an already declared local name, use ``name << value``. + + **local name declarations** + + A ``local`` declaration comes into effect in the expression following + the one where it appears. Thus:: + + result = [] + let((lst, []))[do[result.append(lst), # the let "lst" + local[lst << lst + [1]], # LHS: do "lst", RHS: let "lst" + result.append(lst)]] # the do "lst" + assert result == [[], [1]] + + **Syntactic ambiguity** + + These two cases cannot be syntactically distinguished: + + - Just one body expression, which is a literal tuple or list, + + - Multiple body expressions, represented as a literal tuple or list. + + ``do`` always uses the latter interpretation. + + Whenever there are multiple expressions in the body, the ambiguity does not + arise, because then the distinction between the sequence of expressions itself + and its items is clear. + + Examples:: + + do[1, 2, 3] # --> tuple, 3 + do[(1, 2, 3)] # --> tuple, 3 (since in Python, the comma creates tuples; + # parentheses are only used for disambiguation) + do[[1, 2, 3]] # --> list, 3 + do[[[1, 2, 3]]] # --> list containing a list, [1, 2, 3] + do[([1, 2, 3],)] # --> tuple containing a list, [1, 2, 3] + do[[1, 2, 3],] # --> tuple containing a list, [1, 2, 3] + do[[(1, 2, 3)]] # --> list containing a tuple, (1, 2, 3) + do[((1, 2, 3),)] # --> tuple containing a tuple, (1, 2, 3) + do[(1, 2, 3),] # --> tuple containing a tuple, (1, 2, 3) + + It is possible to use ``unpythonic.misc.pack`` to create a tuple from + given elements: ``do[pack(1, 2, 3)]`` is interpreted as a single-item body + that creates a tuple (by calling a function). + + Note the outermost brackets belong to the ``do``; they don't yet create a list. + + In the *use brackets to denote a multi-expr body* syntax (e.g. ``multilambda``, + ``let`` constructs), the extra brackets already create a list, so in those + uses, the ambiguity does not arise. The transformation inserts not only the + word ``do``, but also the outermost brackets. For example:: + + let[(x, 1), + (y, 2)][[ + [x, y]]] + + transforms to:: + + let[(x, 1), + (y, 2)][do[[ # "do[" is inserted between the two opening brackets + [x, y]]]] # and its closing "]" is inserted here + + which already gets rid of the ambiguity. + + **Notes** + + Macros are expanded in an inside-out order, so a nested ``let`` shadows + names, if the same names appear in the ``do``:: + + do[local[x << 17], + let[(x, 23)][ + print(x)], # 23, the "x" of the "let" + print(x)] # 17, the "x" of the "do" + + The reason we require local names to be declared is to allow write access + to lexically outer environments from inside a ``do``:: + + let[(x, 17)][ + do[x << 23, # no "local[...]"; update the "x" of the "let" + local[y << 42], # "y" is local to the "do" + print(x, y)]] + + With the extra bracket syntax, the latter example can be written as:: + + let[(x, 17)][[ + x << 23, + local[y << 42], + print(x, y)]] + + It's subtly different in that the first version has the do-items in a tuple, + whereas this one has them in a list, but the behavior is exactly the same. + + Python does it the other way around, requiring a ``nonlocal`` statement + to re-bind a name owned by an outer scope. + + The ``let`` constructs solve this problem by having the local bindings + declared in a separate block, which plays the role of ``local``. + """ + if syntax != "expr": + raise SyntaxError("do is an expr macro only") + with dyn.let(_macro_expander=expander): + return _do(tree) + +def do0(tree, *, syntax, expander, **kw): + """[syntax, expr] Like do, but return the value of the first expression.""" + if syntax != "expr": + raise SyntaxError("do0 is an expr macro only") + with dyn.let(_macro_expander=expander): + return _do0(tree) + +# -------------------------------------------------------------------------------- +# Syntax transformers _do_level = NestingLevelTracker() # for checking validity of local[] and delete[] @@ -325,17 +802,17 @@ class UnpythonicDoDeleteMarker(UnpythonicLetDoMarker): # TODO: fail-fast: promote `local[]`/`delete[]` usage errors to compile-time errors # TODO: (doesn't currently work e.g. for `let` with an implicit do (extra bracket notation)) -def local(tree): # syntax transformer +def _local(tree): # syntax transformer if _do_level.value < 1: raise SyntaxError("local[] is only valid within a do[] or do0[]") # pragma: no cover return UnpythonicDoLocalMarker(tree) -def delete(tree): # syntax transformer +def _delete(tree): # syntax transformer if _do_level.value < 1: raise SyntaxError("delete[] is only valid within a do[] or do0[]") # pragma: no cover return UnpythonicDoDeleteMarker(tree) -def do(tree): +def _do(tree): if type(tree) not in (Tuple, List): raise SyntaxError("do body: expected a sequence of comma-separated expressions") # pragma: no cover, let's not test the macro expansion errors. @@ -412,7 +889,7 @@ def transform(self, tree): thecall.args = lines return thecall -def do0(tree): +def _do0(tree): if type(tree) not in (Tuple, List): raise SyntaxError("do0 body: expected a sequence of comma-separated expressions") # pragma: no cover elts = tree.elts @@ -423,12 +900,12 @@ def do0(tree): firstexpr = elts[0] firstexpr = dyn._macro_expander.visit(firstexpr) thelocalexpr = q[_do0_result << a[firstexpr]] # noqa: F821, the local[] defines it inside the do[]. - newelts.append(q[a[local(thelocalexpr)]]) + newelts.append(q[a[_local(thelocalexpr)]]) newelts.extend(elts[1:]) newelts.append(q[_do0_result]) # noqa: F821 newtree = q[t[newelts]] # TODO: Would be cleaner to use `do[]` as a hygienically captured macro. - return do(newtree) # do0[] is also just a do[] + return _do(newtree) # do0[] is also just a do[] def implicit_do(tree): """Allow a sequence of expressions in expression position. @@ -447,4 +924,4 @@ def implicit_do(tree): The outer brackets enable multiple-expression mode, and the inner brackets are then interpreted as a list. """ - return do(tree) if type(tree) is List else tree + return _do(tree) if type(tree) is List else tree diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index cabc848b..0853b0f4 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -4,7 +4,7 @@ # at macro expansion time. If you're looking for regular run-time let et al. macros, # see letdo.py. -__all__ = ["let_syntax_expr", "let_syntax_block"] +__all__ = ["let_syntax", "abbrev"] from mcpyrate.quotes import macros, q, a # noqa: F401 @@ -12,12 +12,164 @@ FunctionDef, AsyncFunctionDef, ClassDef, Attribute) from copy import deepcopy +from mcpyrate import parametricmacro from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer -from .letdo import implicit_do +from .letdo import implicit_do, _destructure_and_apply_let from .util import eliminate_ifones +# -------------------------------------------------------------------------------- +# Macro interface + +# TODO: change the block() construct to block[], for syntactic consistency +@parametricmacro +def let_syntax(tree, *, args, syntax, expander, **kw): + """[syntax, expr/block] Introduce local **syntactic** bindings. + + **Expression variant**:: + + let_syntax[(lhs, rhs), ...][body] + let_syntax[(lhs, rhs), ...][[body0, ...]] + + Alternative haskelly syntax:: + + let_syntax[((lhs, rhs), ...) in body] + let_syntax[((lhs, rhs), ...) in [body0, ...]] + + let_syntax[body, where((lhs, rhs), ...)] + let_syntax[[body0, ...], where((lhs, rhs), ...)] + + **Block variant**:: + + with let_syntax: + with block as xs: # capture a block of statements - bare name + ... + with block(a, ...) as xs: # capture a block of statements - template + ... + with expr as x: # capture a single expression - bare name + ... + with expr(a, ...) as x: # capture a single expression - template + ... + body0 + ... + + A single expression can be a ``do[]`` if multiple expressions are needed. + + The bindings are applied **at macro expansion time**, substituting + the expression on the RHS for each instance of the corresponding LHS. + Each substitution gets a fresh copy. + + This is useful to e.g. locally abbreviate long function names at macro + expansion time (with zero run-time overhead), or to splice in several + (possibly parametric) instances of a common pattern. + + In the expression variant, ``lhs`` may be: + + - A bare name (e.g. ``x``), or + + - A simple template of the form ``f(x, ...)``. The names inside the + parentheses declare the formal parameters of the template (that can + then be used in the body). + + In the block variant: + + - The **as-part** specifies the name of the LHS. + + - If a template, the formal parameters are declared on the ``block`` + or ``expr``, not on the as-part (due to syntactic limitations). + + **Templates** + + To make parametric substitutions, use templates. + + Templates support only positional arguments, with no default values. + + Even in block templates, parameters are always expressions (because they + use the function-call syntax at the use site). + + In the body of the ``let_syntax``, a template is used like a function call. + Just like in an actual function call, when the template is substituted, + any instances of its formal parameters on its RHS get replaced by the + argument values from the "call" site; but ``let_syntax`` performs this + at macro-expansion time. + + Note each instance of the same formal parameter gets a fresh copy of the + corresponding argument value. + + **Substitution order** + + This is a two-step process. In the first step, we apply template substitutions. + In the second step, we apply bare name substitutions to the result of the + first step. (So RHSs of templates may use any of the bare-name definitions.) + + Within each step, the substitutions are applied **in the order specified**. + So if the bindings are ``((x, y), (y, z))``, then ``x`` transforms to ``z``. + But if the bindings are ``((y, z), (x, y))``, then ``x`` transforms to ``y``, + and only an explicit ``y`` at the use site transforms to ``z``. + + **Notes** + + Inspired by Racket's ``let-syntax`` and ``with-syntax``, see: + https://docs.racket-lang.org/reference/let.html + https://docs.racket-lang.org/reference/stx-patterns.html + + **CAUTION**: This is essentially a toy macro system inside the real + macro system, implemented with the real macro system. + + The usual caveats of macro systems apply. Especially, we support absolutely + no form of hygiene. Be very, very careful to avoid name conflicts. + + ``let_syntax`` is meant only for simple local substitutions where the + elimination of repetition can shorten the code and improve readability. + + If you need to do something complex, prefer writing a real macro directly + in `mcpyrate`. + """ + if syntax not in ("expr", "block"): + raise SyntaxError("let_syntax is an expr and block macro only") + + tree = expander.visit(tree) + + if syntax == "expr": + return _destructure_and_apply_let(tree, args, expander, let_syntax_expr, allow_call_in_name_position=True) + else: # syntax == "block": + return let_syntax_block(block_body=tree) + +@parametricmacro +def abbrev(tree, *, args, syntax, expander, **kw): + """[syntax, expr/block] Exactly like ``let_syntax``, but expands outside in. + + Because this variant expands before any macros in the body, it can locally + rename other macros, e.g.:: + + abbrev[(m, macrowithverylongname)][ + m[tree1] if m[tree2] else m[tree3]] + + **CAUTION**: Because ``abbrev`` expands outside-in, and does not respect + boundaries of any nested ``abbrev`` invocations, it will not lexically scope + the substitutions. Instead, the outermost ``abbrev`` expands first, and then + any inner ones expand with whatever substitutions they have remaining. + + If the same name is used on the LHS in two or more nested ``abbrev``, + any inner ones will likely raise an error (unless the outer substitution + just replaces a name with another), because also the names on the LHS + in the inner ``abbrev`` will undergo substitution when the outer + ``abbrev`` expands. + """ + if syntax not in ("expr", "block"): + raise SyntaxError("abbrev is an expr and block macro only") + + # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. + + if syntax == "expr": + return _destructure_and_apply_let(tree, args, expander, let_syntax_expr, allow_call_in_name_position=True) + else: + return let_syntax_block(block_body=tree) + +# -------------------------------------------------------------------------------- +# Syntax transformers + def let_syntax_expr(bindings, body): # bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) body = implicit_do(body) # support the extra bracket syntax if not bindings: @@ -61,8 +213,6 @@ def register_bindings(): body = _substitute_barenames(barenames, body) return body -# ----------------------------------------------------------------------------- - # block version: # # with let_syntax: diff --git a/unpythonic/syntax/nb.py b/unpythonic/syntax/nb.py index 8c53d4d2..c2fb2a4d 100644 --- a/unpythonic/syntax/nb.py +++ b/unpythonic/syntax/nb.py @@ -12,9 +12,38 @@ from mcpyrate.quotes import macros, q, u, a, h # noqa: F401 +from mcpyrate import parametricmacro + from .testingtools import istestmacro -def nb(body, args): +@parametricmacro +def nb(tree, *, args, syntax, **kw): + """[syntax, block] Ultralight math notebook. + + Auto-print top-level expressions, auto-assign last result as _. + + A custom print function can be supplied as an argument. + + Example:: + + with nb: + 2 + 3 + 42 * _ + + from sympy import * + with nb[pprint]: + x, y = symbols("x, y") + x * y + 3 * _ + """ + if syntax != "block": + raise SyntaxError("nb is a block macro only") + + # Expand outside in. This macro is so simple and orthogonal the + # ordering doesn't matter. This is cleaner. + return _nb(body=tree, args=args) + +def _nb(body, args): p = args[0] if args else q[h[print]] # custom print function hook with q as newbody: # pragma: no cover, quoted only. _ = None diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index d0aed4a3..17dcb082 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -18,7 +18,112 @@ from ..it import flatmap, rev, uniqify -def prefix(block_body): +# -------------------------------------------------------------------------------- + +def prefix(tree, *, syntax, **kw): # noqa: F811 + """[syntax, block] Write Python like Lisp: the first item is the operator. + + Example:: + + with prefix: + (print, "hello world") + t1 = (q, 1, 2, (3, 4), 5) + x = 42 + t2 = (q, 17, 23, x) + (print, t1, t2) + + Lexically inside a ``with prefix``: + + - A bare ``q`` at the head of a tuple is the quote operator. It increases + the quote level by one. + + It actually just tells the macro that this tuple (and everything in it, + recursively) is not a function call. + + Variables can be used as usual, there is no need to unquote them. + + - A bare ``u`` at the head of a tuple is the unquote operator, which + decreases the quote level by one. In other words, in:: + + with prefix: + t = (q, 1, 2, (u, print, 3), (print, 4), 5) + (print, t) + + the third item will call ``print(3)`` and evaluate to its return value + (in this case ``None``, since it's ``print``), whereas the fourth item + is a tuple with the two items ``(, 4)``. + + - Quote/unquote operators are parsed from the start of the tuple until + no more remain. Then any remaining items are either returned quoted + (if quote level > 0), or evaluated as a function call and replaced + by the return value. + + - How to pass named args:: + + from unpythonic.misc import call + + with prefix: + (f, kw(myarg=3)) # ``kw(...)`` (syntax, not really a function!) + call(f, myarg=3) # in a call(), kwargs are ok + f(myarg=3) # or just use Python's usual function call syntax + + One ``kw`` operator may include any number of named args (and **only** + named args). The tuple may have any number of ``kw`` operators. + + All named args are collected from ``kw`` operators in the tuple + when writing the final function call. If the same kwarg has been + specified by multiple ``kw`` operators, the rightmost definition wins. + + **Note**: Python itself prohibits having repeated named args in the **same** + ``kw`` operator, because it uses the function call syntax. If you get a + `SyntaxError: keyword argument repeated` with no useful traceback, + check any recent ``kw`` operators you have added in prefix blocks. + + A ``kw(...)`` operator in a quoted tuple (not a function call) is an error. + + Current limitations: + + - passing ``*args`` and ``**kwargs`` not supported. + + Workarounds: ``call(...)``; Python's usual function call syntax. + + - For ``*args``, to keep it lispy, maybe you want ``unpythonic.fun.apply``; + this allows syntax such as ``(apply, f, 1, 2, lst)``. + + **CAUTION**: This macro is experimental, not intended for production use. + """ + if syntax != "block": + raise SyntaxError("prefix is a block macro only") + + # Expand outside in. Any nested macros should get clean standard Python, + # not having to worry about tuples possibly denoting function calls. + return _prefix(block_body=tree) + +# Note the exported "q" and "u" are ours, but the "q" and "u" we use in this +# module are macros. The "q" and "u" we define here are regular run-time objects, +# namely the stubs for the "q" and "u" markers used within a `prefix` block. +class q: # noqa: F811 + """[syntax] Quote operator. Only meaningful in a tuple in a prefix block.""" + def __repr__(self): # in case one of these ends up somewhere at runtime # pragma: no cover + return "" +q = q() + +class u: # noqa: F811 + """[syntax] Unquote operator. Only meaningful in a tuple in a prefix block.""" + def __repr__(self): # in case one of these ends up somewhere at runtime # pragma: no cover + return "" +u = u() + +# TODO: Think of promoting this error to compile macro expansion time. +# TODO: Difficult to do, because we shouldn't probably hijack the name "kw" (so no name macro), +# TODO: and it can't be invoked like an expr macro, because the whole point is to pass arguments by name. +def kw(**kwargs): + """[syntax] Pass-named-args operator. Only meaningful in a tuple in a prefix block.""" + raise RuntimeError("kw(...) only meaningful inside a tuple in a prefix block") # pragma: no cover + +# -------------------------------------------------------------------------------- + +def _prefix(block_body): isquote = lambda tree: type(tree) is Name and tree.id == "q" isunquote = lambda tree: type(tree) is Name and tree.id == "u" iskwargs = lambda tree: type(tree) is Call and type(tree.func) is Name and tree.func.id == "kw" @@ -117,25 +222,3 @@ def transform(self, tree): # This is a first-pass macro. Any nested macros should get clean standard Python, # not having to worry about tuples possibly denoting function calls. return PrefixTransformer(quotelevel=0).visit(block_body) - -# Note the exported "q" and "u" are ours, but the "q" and "u" we use in this -# module are macros. The "q" and "u" we define here are regular run-time objects, -# namely the stubs for the "q" and "u" markers used within a `prefix` block. -class q: # noqa: F811 - """[syntax] Quote operator. Only meaningful in a tuple in a prefix block.""" - def __repr__(self): # in case one of these ends up somewhere at runtime # pragma: no cover - return "" -q = q() - -class u: # noqa: F811 - """[syntax] Unquote operator. Only meaningful in a tuple in a prefix block.""" - def __repr__(self): # in case one of these ends up somewhere at runtime # pragma: no cover - return "" -u = u() - -# TODO: Think of promoting this error to compile macro expansion time. -# TODO: Difficult to do, because we shouldn't probably hijack the name "kw" (so no name macro), -# TODO: and it can't be invoked like an expr macro, because the whole point is to pass arguments by name. -def kw(**kwargs): - """[syntax] Pass-named-args operator. Only meaningful in a tuple in a prefix block.""" - raise RuntimeError("kw(...) only meaningful inside a tuple in a prefix block") # pragma: no cover diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 49923282..a222d3aa 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -5,7 +5,7 @@ __all__ = ["autoreturn", "tco", - "call_cc", "continuations"] + "continuations", "call_cc"] from functools import partial @@ -32,7 +32,7 @@ has_tco, sort_lambda_decorators, suggest_decorator_index, ExpandedContinuationsMarker, wrapwith, isexpandedmacromarker) from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView -from .ifexprs import aif +from .ifexprs import _aif from ..dynassign import dyn from ..it import uniqify @@ -40,10 +40,611 @@ from ..tco import trampolined, jump from ..lazyutil import passthrough_lazy_args -# ----------------------------------------------------------------------------- -# Implicit return statement. This performs a tail-position analysis of function bodies. +# -------------------------------------------------------------------------------- +# Macro interface + +def autoreturn(tree, *, syntax, **kw): + """[syntax, block] Implicit "return" in tail position, like in Lisps. + + Each ``def`` function definition lexically within the ``with autoreturn`` + block is examined, and if the last item within the body is an expression + ``expr``, it is transformed into ``return expr``. + + If the last item is an if/elif/else block, the transformation is applied + to the last item in each of its branches. + + If the last item is a ``with`` or ``async with`` block, the transformation + is applied to the last item in its body. + + If the last item is a try/except/else/finally block, the rules are as follows. + If an ``else`` clause is present, the transformation is applied to the last + item in it; otherwise, to the last item in the ``try`` clause. Additionally, + in both cases, the transformation is applied to the last item in each of the + ``except`` clauses. The ``finally`` clause is not transformed; the intention + is it is usually a finalizer (e.g. to release resources) that runs after the + interesting value is already being returned by ``try``, ``else`` or ``except``. + + Example:: + + with autoreturn: + def f(): + "I'll just return this" + assert f() == "I'll just return this" + + def g(x): + if x == 1: + "one" + elif x == 2: + "two" + else: + "something else" + assert g(1) == "one" + assert g(2) == "two" + assert g(42) == "something else" + + **CAUTION**: If the final ``else`` is omitted, as often in Python, then + only the ``else`` item is in tail position with respect to the function + definition - likely not what you want. + + So with ``autoreturn``, the final ``else`` should be written out explicitly, + to make the ``else`` branch part of the same if/elif/else block. + + **CAUTION**: ``for``, ``async for``, ``while`` are currently not analyzed; + effectively, these are defined as always returning ``None``. If the last item + in your function body is a loop, use an explicit return. + + **CAUTION**: With ``autoreturn`` enabled, functions no longer return ``None`` + by default; the whole point of this macro is to change the default return + value. + + The default return value is ``None`` only if the tail position contains + a statement (because in a sense, a statement always returns ``None``). + """ + if syntax != "block": + raise SyntaxError("autoreturn is a block macro only") + + # Expand outside in. Any nested macros should get clean standard Python, + # not having to worry about implicit "return" statements. + return _autoreturn(block_body=tree) + +def tco(tree, *, syntax, expander, **kw): + """[syntax, block] Implicit tail-call optimization (TCO). + + Examples:: + + with tco: + evenp = lambda x: (x == 0) or oddp(x - 1) + oddp = lambda x: (x != 0) and evenp(x - 1) + assert evenp(10000) is True + + with tco: + def evenp(x): + if x == 0: + return True + return oddp(x - 1) + def oddp(x): + if x != 0: + return evenp(x - 1) + return False + assert evenp(10000) is True + + This is based on a strategy similar to MacroPy's tco macro, but using + the TCO machinery from ``unpythonic.tco``. + + This recursively handles also builtins ``a if p else b``, ``and``, ``or``; + and from ``unpythonic.syntax``, ``do[]``, ``let[]``, ``letseq[]``, ``letrec[]``, + when used in computing a return value. (``aif[]`` and ``cond[]`` also work.) + + Note only calls **in tail position** will be TCO'd. Any other calls + are left as-is. Tail positions are: + + - The whole return value, if it is just a single call. + + - Both ``a`` and ``b`` branches of ``a if p else b`` (but not ``p``). + + - The last item in an ``and``/``or``. If these are nested, only the + last item in the whole expression involving ``and``/``or``. E.g. in:: + + (a and b) or c + a and (b or c) + + in either case, only ``c`` is in tail position, regardless of the + values of ``a``, ``b``. + + - The last item in a ``do[]``. + + - In a ``do0[]``, this is the implicit item that just returns the + stored return value. + + - The argument of a call to an escape continuation. The ``ec(...)`` call + itself does not need to be in tail position; escaping early is the + whole point of an ec. + + All function definitions (``def`` and ``lambda``) lexically inside the block + undergo TCO transformation. The functions are automatically ``@trampolined``, + and any tail calls in their return values are converted to ``jump(...)`` + for the TCO machinery. + + Note in a ``def`` you still need the ``return``; it marks a return value. + But see ``autoreturn``:: + + with autoreturn, tco: + def evenp(x): + if x == 0: + True + else: + oddp(x - 1) + def oddp(x): + if x != 0: + evenp(x - 1) + else: + False + assert evenp(10000) is True + + **CAUTION**: regarding escape continuations, only basic uses of ecs created + via ``call_ec`` are currently detected as being in tail position. Any other + custom escape mechanisms are not supported. (This is mainly of interest for + lambdas, which have no ``return``, and for "multi-return" from a nested + function.) + + *Basic use* is defined as either of these two cases:: + + # use as decorator + @call_ec + def result(ec): + ... + + # use directly on a literal lambda + result = call_ec(lambda ec: ...) + + When macro expansion of the ``with tco`` block starts, names of escape + continuations created **anywhere lexically within** the ``with tco`` block + are captured. Lexically within the block, any call to a function having + any of the captured names, or as a fallback, one of the literal names + ``ec``, ``brk``, ``throw`` is interpreted as invoking an escape + continuation. + """ + if syntax != "block": + raise SyntaxError("tco is a block macro only") + + # Two-pass macro. + with dyn.let(_macro_expander=expander): + return _tco(block_body=tree) + +def continuations(tree, *, syntax, expander, **kw): + """[syntax, block] call/cc for Python. + + This allows saving the control state and then jumping back later + (in principle, any time later). Some possible use cases: + + - Tree traversal (possibly a cartesian product of multiple trees, with the + current position in each tracked automatically). + + - McCarthy's amb operator. + + - Generators. (Python already has those, so only for teaching.) + + This is a very loose pythonification of Paul Graham's continuation-passing + macros, which implement continuations by chaining closures and passing the + continuation semi-implicitly. For details, see chapter 20 in On Lisp: + + http://paulgraham.com/onlisp.html + + Continuations are most readily implemented when the program is written in + continuation-passing style (CPS), but that is unreadable for humans. + The purpose of this macro is to partly automate the CPS transformation, so + that at the use site, we can write CPS code in a much more readable fashion. + + A ``with continuations`` block implies TCO; the same rules apply as in a + ``with tco`` block. Furthermore, ``with continuations`` introduces the + following additional rules: + + - Functions which make use of continuations, or call other functions that do, + must be defined within a ``with continuations`` block, using the usual + ``def`` or ``lambda`` forms. + + - All function definitions in a ``with continuations`` block, including + any nested definitions, have an implicit formal parameter ``cc``, + **even if not explicitly declared** in the formal parameter list. + + If declared explicitly, ``cc`` must be in a position that can accept a + default value. + + This means ``cc`` must be declared either as by-name-only:: + + with continuations: + def myfunc(a, b, *, cc): + ... + + f = lambda *, cc: ... + + or as the last parameter that has no default:: + + with continuations: + def myfunc(a, b, cc): + ... + + f = lambda cc: ... + + Then the continuation machinery will automatically set the default value + of ``cc`` to the default continuation (``identity``), which just returns + its arguments. + + The most common use case for explicitly declaring ``cc`` is that the + function is the target of a ``call_cc[]``; then it helps readability + to make the ``cc`` parameter explicit. + + - A ``with continuations`` block will automatically transform all + function definitions and ``return`` statements lexically contained + within the block to use the continuation machinery. + + - ``return somevalue`` actually means a tail-call to ``cc`` with the + given ``somevalue``. + + Multiple values can be returned as a ``tuple``. Tupleness is tested + at run-time. + + Any tuple return value is automatically unpacked to the positional + args of ``cc``. To return multiple things as one without the implicit + unpacking, use a ``list``. + + - An explicit ``return somefunc(arg0, ..., k0=v0, ...)`` actually means + a tail-call to ``somefunc``, with its ``cc`` automatically set to our + ``cc``. Hence this inserts a call to ``somefunc`` before proceeding + with our current continuation. (This is most often what we want when + making a tail-call from a continuation-enabled function.) + + Here ``somefunc`` **must** be a continuation-enabled function; + otherwise the TCO chain will break and the result is immediately + returned to the top-level caller. + + (If the call succeeds at all; the ``cc`` argument is implicitly + filled in and passed by name. Regular functions usually do not + accept a named parameter ``cc``, let alone know what to do with it.) + + - Just like in ``with tco``, a lambda body is analyzed as one big + return-value expression. This uses the exact same analyzer; for example, + ``do[]`` (including any implicit ``do[]``) and the ``let[]`` expression + family are supported. + + - Calls from functions defined in one ``with continuations`` block to those + defined in another are ok; there is no state or context associated with + the block. + + - Much of the language works as usual. + + Any non-tail calls can be made normally. Regular functions can be called + normally in any non-tail position. + + Continuation-enabled functions behave as regular functions when + called normally; only tail calls implicitly set ``cc``. A normal call + uses ``identity`` as the default ``cc``. + + - For technical reasons, the ``return`` statement is not allowed at the + top level of the ``with continuations:`` block. (Because a continuation + is essentially a function, ``return`` would behave differently based on + whether it is placed lexically before or after a ``call_cc[]``.) + + If you absolutely need to terminate the function surrounding the + ``with continuations:`` block from inside the block, use an exception + to escape; see ``call_ec``, ``catch``, ``throw``. + + **Capturing the continuation**: + + Inside a ``with continuations:`` block, the ``call_cc[]`` statement + captures a continuation. (It is actually a macro, for technical reasons.) + + For various possible program topologies that continuations may introduce, see + the clarifying pictures under ``doc/`` in the source distribution. + + Syntax:: + + x = call_cc[func(...)] + *xs = call_cc[func(...)] + x0, ... = call_cc[func(...)] + x0, ..., *xs = call_cc[func(...)] + call_cc[func(...)] + + Conditional variant:: + + x = call_cc[f(...) if p else g(...)] + *xs = call_cc[f(...) if p else g(...)] + x0, ... = call_cc[f(...) if p else g(...)] + x0, ..., *xs = call_cc[f(...) if p else g(...)] + call_cc[f(...) if p else g(...)] + + Assignment targets: + + - To destructure a multiple-values (from a tuple return value), + use a tuple assignment target (comma-separated names, as usual). + + - The last assignment target may be starred. It is transformed into + the vararg (a.k.a. ``*args``) of the continuation function. + (It will capture a whole tuple, or any excess items, as usual.) + + - To ignore the return value (useful if ``func`` was called only to + perform its side-effects), just omit the assignment part. + + Conditional variant: + + - ``p`` is any expression. If truthy, ``f(...)`` is called, and if falsey, + ``g(...)`` is called. + + - Each of ``f(...)``, ``g(...)`` may be ``None``. A ``None`` skips the + function call, proceeding directly to the continuation. Upon skipping, + all assignment targets (if any are present) are set to ``None``. + The starred assignment target (if present) gets the empty tuple. + + - The main use case of the conditional variant is for things like:: + + with continuations: + k = None + def setk(cc): + global k + k = cc + def dostuff(x): + call_cc[setk() if x > 10 else None] # capture only if x > 10 + ... + + To keep things relatively straightforward, a ``call_cc[]`` is only + allowed to appear **at the top level** of: + + - the ``with continuations:`` block itself + - a ``def`` or ``async def`` + + Nested defs are ok; here *top level* only means the top level of the + *currently innermost* ``def``. + + If you need to place ``call_cc[]`` inside a loop, use ``@looped`` et al. + from ``unpythonic.fploop``; this has the loop body represented as the + top level of a ``def``. + + Multiple ``call_cc[]`` statements in the same function body are allowed. + These essentially create nested closures. + + **Main differences to Scheme and Racket**: + + Compared to Scheme/Racket, where ``call/cc`` will capture also expressions + occurring further up in the call stack, our ``call_cc`` may be need to be + placed differently (further out, depending on what needs to be captured) + due to the delimited nature of the continuations implemented here. + + Scheme and Racket implicitly capture the continuation at every position, + whereas we do it explicitly, only at the use sites of the ``call_cc`` macro. + + Also, since there are limitations to where a ``call_cc[]`` may appear, some + code may need to be structured differently to do some particular thing, if + porting code examples originally written in Scheme or Racket. + + Unlike ``call/cc`` in Scheme/Racket, ``call_cc`` takes **a function call** + as its argument, not just a function reference. Also, there's no need for + it to be a one-argument function; any other args can be passed in the call. + The ``cc`` argument is filled implicitly and passed by name; any others are + passed exactly as written in the client code. + + **Technical notes**: + + The ``call_cc[]`` statement essentially splits its use site into *before* + and *after* parts, where the *after* part (the continuation) can be run + a second and further times, by later calling the callable that represents + the continuation. This makes a computation resumable from a desired point. + + The return value of the continuation is whatever the original function + returns, for any ``return`` statement that appears lexically after the + ``call_cc[]``. -def autoreturn(block_body): + The effect of ``call_cc[]`` is that the function call ``func(...)`` in + the brackets is performed, with its ``cc`` argument set to the lexically + remaining statements of the current ``def`` (at the top level, the rest + of the ``with continuations`` block), represented as a callable. + + The continuation itself ends there (it is *delimited* in this particular + sense), but it will chain to the ``cc`` of the function it appears in. + This is termed the *parent continuation* (**pcc**), stored in the internal + variable ``_pcc`` (which defaults to ``None``). + + Via the use of the pcc, here ``f`` will maintain the illusion of being + just one function, even though a ``call_cc`` appears there:: + + def f(*, cc): + ... + call_cc[g(1, 2, 3)] + ... + + The continuation is a closure. For its pcc, it will use the value the + original function's ``cc`` had when the definition of the continuation + was executed (for that particular instance of the closure). Hence, calling + the original function again with its ``cc`` set to something else will + produce a new continuation instance that chains into that new ``cc``. + + The continuation's own ``cc`` will be ``identity``, to allow its use just + like any other function (also as argument of a ``call_cc`` or target of a + tail call). + + When the pcc is set (not ``None``), the effect is to run the pcc first, + and ``cc`` only after that. This preserves the whole captured tail of a + computation also in the presence of nested ``call_cc`` invocations (in the + above example, this would occur if also ``g`` used ``call_cc``). + + Continuations are not accessible by name (their definitions are named by + gensym). To get a reference to a continuation instance, stash the value + of the ``cc`` argument somewhere while inside the ``call_cc``. + + The function ``func`` called by a ``call_cc[func(...)]`` is (almost) the + only place where the ``cc`` argument is actually set. There it is the + captured continuation. Roughly everywhere else, ``cc`` is just ``identity``. + + Tail calls are an exception to this rule; a tail call passes along the current + value of ``cc``, unless overridden manually (by setting the ``cc=...`` kwarg + in the tail call). + + When the pcc is set (not ``None``) at the site of the tail call, the + machinery will create a composed continuation that runs the pcc first, + and ``cc`` (whether current or manually overridden) after that. This + composed continuation is then passed to the tail call as its ``cc``. + + **Tips**: + + - Once you have a captured continuation, one way to use it is to set + ``cc=...`` manually in a tail call, as was mentioned. Example:: + + def main(): + call_cc[myfunc()] # call myfunc, capturing the current cont... + ... # ...which is the rest of "main" + + def myfunc(cc): + ourcc = cc # save the captured continuation (sent by call_cc[]) + def somefunc(): + return dostuff(..., cc=ourcc) # and use it here + somestack.append(somefunc) + + In this example, when ``somefunc`` is eventually called, it will tail-call + ``dostuff`` and then proceed with the continuation ``myfunc`` had + at the time when that instance of the ``somefunc`` closure was created. + (This pattern is essentially how to build the ``amb`` operator.) + + - Instead of setting ``cc``, you can also overwrite ``cc`` with a captured + continuation inside a function body. That overrides the continuation + for the rest of the dynamic extent of the function, not only for a + particular tail call:: + + def myfunc(cc): + ourcc = cc + def somefunc(): + cc = ourcc + return dostuff(...) + somestack.append(somefunc) + + - A captured continuation can also be called manually; it's just a callable. + + The assignment targets, at the ``call_cc[]`` use site that spawned this + particular continuation, specify its call signature. All args are + positional, except the implicit ``cc``, which is by-name-only. + + - Just like in Scheme/Racket's ``call/cc``, the values that get bound + to the ``call_cc[]`` assignment targets on second and further calls + (when the continuation runs) are the arguments given to the continuation + when it is called (whether implicitly or manually). + + - Setting ``cc`` to ``unpythonic.fun.identity``, while inside a ``call_cc``, + will short-circuit the rest of the computation. In such a case, the + continuation will not be invoked automatically. A useful pattern for + suspend/resume. + + - However, it is currently not possible to prevent the rest of the tail + of a captured continuation (the pcc) from running, apart from manually + setting ``_pcc`` to ``None`` before executing a ``return``. Note that + doing that is not strictly speaking supported (and may be subject to + change in a future version). + + - When ``call_cc[]`` appears inside a function definition: + + - It tail-calls ``func``, with its ``cc`` set to the captured + continuation. + + - The return value of the function containing one or more ``call_cc[]`` + statements is the return value of the continuation. + + - When ``call_cc[]`` appears at the top level of ``with continuations``: + + - A normal call to ``func`` is made, with its ``cc`` set to the captured + continuation. + + - In this case, if the continuation is called later, it always + returns ``None``, because the use site of ``call_cc[]`` is not + inside a function definition. + + - If you need to insert just a tail call (no further statements) before + proceeding with the current continuation, no need for ``call_cc[]``; + use ``return func(...)`` instead. + + The purpose of ``call_cc[func(...)]`` is to capture the current + continuation (the remaining statements), and hand it to ``func`` + as a first-class value. + + - To combo with ``multilambda``, use this ordering:: + + with multilambda, continuations: + ... + + - Some very limited comboability with ``call_ec``. May be better to plan + ahead, using ``call_cc[]`` at the appropriate outer level, and then + short-circuit (when needed) by setting ``cc`` to ``identity``. + This avoids the need to have both ``call_cc`` and ``call_ec`` at the + same time. + + - ``unpythonic.ec.call_ec`` can be used normally **lexically before any** + ``call_cc[]``, but (in a given function) after at least one ``call_cc[]`` + has run, the ``ec`` ceases to be valid. This is because our ``call_cc[]`` + actually splits the function into *before* and *after* parts, and + **tail-calls** the *after* part. + + (Wrapping the ``def`` in another ``def``, and placing the ``call_ec`` + on the outer ``def``, does not help either, because even the outer + function has exited by the time *the continuation* is later called + the second and further times.) + + Usage of ``call_ec`` while inside a ``with continuations`` block is:: + + with continuations: + @call_ec + def result(ec): + print("hi") + ec(42) + print("not reached") + assert result == 42 + + result = call_ec(lambda ec: do[print("hi"), + ec(42), + print("not reached")]) + + Note the signature of ``result``. Essentially, ``ec`` is a function + that raises an exception (to escape to a dynamically outer context), + whereas the implicit ``cc`` is the closure-based continuation handled + by the continuation machinery. + + See the ``tco`` macro for details on the ``call_ec`` combo. + """ + if syntax != "block": + raise SyntaxError("continuations is a block macro only") + + # Two-pass macro. + with dyn.let(_macro_expander=expander): + return _continuations(block_body=tree) + +def call_cc(tree, **kw): + """[syntax] Only meaningful in a "with continuations" block. + + Syntax cheat sheet:: + + x = call_cc[func(...)] + *xs = call_cc[func(...)] + x0, ... = call_cc[func(...)] + x0, ..., *xs = call_cc[func(...)] + call_cc[func(...)] + + Conditional variant:: + + x = call_cc[f(...) if p else g(...)] + *xs = call_cc[f(...) if p else g(...)] + x0, ... = call_cc[f(...) if p else g(...)] + x0, ..., *xs = call_cc[f(...) if p else g(...)] + call_cc[f(...) if p else g(...)] + + where ``f()`` or ``g()`` may be ``None`` instead of a function call. + + For more, see the docstring of ``continuations``. + """ + if _continuations_level.value < 1: + raise SyntaxError("call_cc[] is only meaningful in a `with continuations` block.") # pragma: no cover, not meant to hit the expander (expanded away by `with continuations`) + return UnpythonicCallCcMarker(tree) + + +# -------------------------------------------------------------------------------- +# Syntax transformers + +# Implicit return statement. This performs a tail-position analysis of function bodies. +def _autoreturn(block_body): class AutoreturnTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -75,10 +676,9 @@ def transform_tailstmt(tree): # not having to worry about implicit "return" statements. return AutoreturnTransformer().visit(block_body) -# ----------------------------------------------------------------------------- -# Automatic TCO. This is the same framework as in "continuations", in its simplest form. -def tco(block_body): +# Automatic TCO. This is the same framework as in "continuations", in its simplest form. +def _tco(block_body): # first pass, outside-in userlambdas = detect_lambda(block_body) known_ecs = list(uniqify(detect_callec(block_body))) @@ -106,41 +706,9 @@ def tco(block_body): new_block_body.append(stmt) return new_block_body -# ----------------------------------------------------------------------------- - -_continuations_level = NestingLevelTracker() # for checking validity of call_cc[] - -class UnpythonicContinuationsMarker(ASTMarker): - """AST marker related to the unpythonic's continuations (call_cc) subsystem.""" -class UnpythonicCallCcMarker(UnpythonicContinuationsMarker): - """AST marker denoting a `call_cc[]` invocation.""" - -def call_cc(tree, **kw): - """[syntax] Only meaningful in a "with continuations" block. - - Syntax cheat sheet:: - - x = call_cc[func(...)] - *xs = call_cc[func(...)] - x0, ... = call_cc[func(...)] - x0, ..., *xs = call_cc[func(...)] - call_cc[func(...)] - - Conditional variant:: - - x = call_cc[f(...) if p else g(...)] - *xs = call_cc[f(...) if p else g(...)] - x0, ... = call_cc[f(...) if p else g(...)] - x0, ..., *xs = call_cc[f(...) if p else g(...)] - call_cc[f(...) if p else g(...)] - - where ``f()`` or ``g()`` may be ``None`` instead of a function call. - For more, see the docstring of ``continuations``. - """ - if _continuations_level.value < 1: - raise SyntaxError("call_cc[] is only meaningful in a `with continuations` block.") # pragma: no cover, not meant to hit the expander (expanded away by `with continuations`) - return UnpythonicCallCcMarker(tree) +# ----------------------------------------------------------------------------- +# True multi-shot continuations for Python, based on a CPS transformation. # _pcc/cc chaining handler, to be exported to client code via q[h[]]. # @@ -178,7 +746,16 @@ def cc(value): return jump(cc2, value) return cc -def continuations(block_body): + +_continuations_level = NestingLevelTracker() # for checking validity of call_cc[] + +class UnpythonicContinuationsMarker(ASTMarker): + """AST marker related to the unpythonic's continuations (call_cc) subsystem.""" +class UnpythonicCallCcMarker(UnpythonicContinuationsMarker): + """AST marker denoting a `call_cc[]` invocation.""" + + +def _continuations(block_body): # This is a very loose pythonification of Paul Graham's continuation-passing # macros in On Lisp, chapter 20. # @@ -678,9 +1255,9 @@ def transform(tree): op_of_others = tree.values[0] if type(tree.op) is Or: # or(data1, ..., datan, tail) --> it if any(others) else tail - tree = aif(Tuple(elts=[op_of_others, - transform_data(Name(id="it")), - transform(tree.values[-1])])) # tail-call item + tree = _aif(Tuple(elts=[op_of_others, + transform_data(Name(id="it")), + transform(tree.values[-1])])) # tail-call item elif type(tree.op) is And: # and(data1, ..., datan, tail) --> tail if all(others) else False fal = q[False] diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 21a5d494..e61fd501 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -5,14 +5,13 @@ """ __all__ = ["isunexpandedtestmacro", "isexpandedtestmacro", "istestmacro", - "fail_expr", "error_expr", "warn_expr", - "the", - "test_expr", "test_expr_signals", "test_expr_raises", - "test_block", "test_block_signals", "test_block_raises"] + "the", "test", + "test_signals", "test_raises", + "fail", "error", "warn"] from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 -from mcpyrate import gensym, unparse +from mcpyrate import gensym, parametricmacro, unparse from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer @@ -30,8 +29,396 @@ from ..test import fixtures # unpythonic.test.fixtures, regular (non-macro) code belonging to the framework +# -------------------------------------------------------------------------------- +# Macro interface + +def the(tree, **kw): + """[syntax, expr] In a test, mark a subexpression as the interesting one. + + Only meaningful inside a `test[]`, or inside a `with test` block. + + What `test[expr]` captures for reporting for human inspection upon + test failure: + + - If any `the[...]` are present, the subexpressions marked as `the[...]`. + + - Else if `expr` is a comparison, the LHS (leftmost term in case of + a chained comparison). So e.g. `test[x < 3]` needs no annotation + to do the right thing. This is a common use case, hence automatic. + + - Else nothing is captured; the value of the whole `expr` is reported. + + So the `the[...]` mark is useful in tests involving comparisons:: + + test[lower_limit < the[computeitem(...)]] + test[lower_limit < the[computeitem(...)] < upper_limit] + test[myconstant in the[computeset(...)]] + + especially if you need to capture several subexpressions:: + + test[the[counter()] < the[counter()]] + + Note the above rules mean that if there is just one interesting + subexpression, and it is the leftmost term of a comparison, `the[...]` + is optional, although allowed (to explicitly document intent). + These have the same effect:: + + test[the[computeitem(...)] in myitems] + test[computeitem(...) in myitems] + + The `the[...]` mark passes the value through, and does not affect the + evaluation order of user code. + + A `test[...]` may have multiple `the[...]`; the captured values are + gathered in a list that is shown upon test failure. + + In case of nested tests, each `the[...]` is understood as belonging to + the lexically innermost surrounding test. + + For `test_raises` and `test_signals`, the `the[...]` mark is not supported. + """ + raise SyntaxError("the[] is only meaningful inside a `test[]` or in a `with test` block") # pragma: no cover, not meant to hit the expander + +@parametricmacro +def test(tree, *, args, syntax, expander, **kw): # noqa: F811 + """[syntax, expr/block] Make a test assertion. For writing automated tests. + + **Testing overview**: + + Use the `test[]`, `test_raises[]`, `test_signals[]`, `fail[]`, `error[]` + and `warn[]` macros inside a `with testset()`, as appropriate. + + See `testset` and `session` in the module `unpythonic.test.fixtures`, + as well as the docstrings of any constructs exported from that module. + + See below for tips and tricks. + + Finally, see the unit tests of `unpythonic` itself for examples. + + **Expression variant**: + + Syntax:: + + test[expr] + test[expr, message] + + The test succeeds if `expr` evaluates to truthy. The `message` + is used in forming the error message if the test fails or errors. + + If you want to assert just that an expression runs to completion + normally, and don't care about the return value:: + + from unpythonic.test.fixtures import returns_normally + + test[returns_normally(expr)] + test[returns_normally(expr), message] + + This can be useful for testing functions with side effects; sometimes + what is important is that the function completes normally. + + What `test[expr]` captures for reporting as "result" in the failure + message, if the test fails: + + - If a `the[...]` mark is present, the subexpression marked as `the[...]`. + At most one `the[]` may appear in a single `test[...]`. + - Else if `expr` is a comparison, the LHS (leftmost term in case of + a chained comparison). So e.g. `test[x < 3]` needs no annotation + to do the right thing. This is a common use case, hence automatic. + - Else the whole `expr`. + + The `the[...]` mark is useful in tests involving comparisons:: + + test[lower_limit < the[computeitem(...)]] + test[lower_limit < the[computeitem(...)] < upper_limit] + test[myconstant in the[computeset(...)]] + + If your interesting part is on the LHS, `the[]` is optional, although + allowed (to explicitly document intent). These have the same effect:: + + test[the[computeitem(...)] in myitems] + test[computeitem(...) in myitems] + + The `the[...]` mark passes the value through, and does not affect the + evaluation order of user code. + + The `the[]` mark can be imported as a macro from this module, so that + its appearance in your source code won't confuse `flake8`, and you'll + get a nice macro-expansion-time error if it accidentally appears outside + a `test[]` or `with test:`. + + **Block variant**: + + A test that requires statements (e.g. assignments) can be written as a + `with test` block:: + + with test: + body0 + ... + return expr # optional + + with test[message]: + body0 + ... + return expr # optional + + The test block is automatically lifted into a function, so it introduces + **a local scope**. Use the `nonlocal` or `global` declarations if you need + to mutate something defined on the outside. + + If there is a `return` at the top level of the block, that is the return + value from the test; it is what will be asserted. + + If there is no `return`, the test asserts that the block completes normally, + just like a `test[returns_normally(...)]` does for an expression. + + The asymmetry in syntax reflects the asymmetry between expressions and + statements in Python. Likewise, the fact that `with test` requires `return` + to return a value, but `test[...]` doesn't, is similar to the difference + between `def` and `lambda`. + + In the block variant, the "result" capture rules apply to the return value + designated by `return`. To override, the `the[]` mark can be used for + capturing the value of any one expression inside the block. The mark + doesn't have to be in the `return`. + + At most one `the[]` may appear in the same `with test` block. + + **Failure and error signaling**: + + Upon a test failure, `test[]` will *signal* a `TestFailure` using the + *cerror* (correctable error) protocol, via unpythonic's condition + system, which is a pythonification of Common Lisp's condition system. + See `unpythonic.conditions`. + + If a test fails to run to completion due to an uncaught exception or an + unhandled signal (e.g. an `error` or `cerror` condition), `TestError` + is signaled instead, so the caller can easily tell apart which case + occurred. + + Finally, when a `warn[]` runs, `TestWarning` is signaled. + + These condition types are defined in `unpythonic.test.fixtures`. + They inherit from `TestingException`, defined in the same module. + Beside the human-readable message, these exception types contain + attributes with programmatically inspectable information about + what happened. See the docstring of `TestingException`. + + *Signaling* a condition, instead of *raising* an exception, allows the + surrounding code (inside the test framework) to install a handler that + invokes the `proceed` restart (if there is such in scope), so upon a test + failure or error, the test suite resumes. + + **Disabling the signal barrier**: + + As implied above, `test[]` (likewise `with test:`) forms a barrier that + alerts the user about uncaught signals, and stops those signals from + propagating further. If your `with handlers` block that needs to see + the signal is outside the `test` invocation, or if allowing a signal to + go uncaught is part of normal operation (e.g. `warn` signals are often + not caught, because the only reason to do so is to muffle the warning), + use a `with catch_signals(False):` block (from the module + `unpythonic.test.fixtures`) to disable the signal barrier:: + + from unpythonic.test.fixtures import catch_signals + + with catch_signals(False): + test[...] + + Another way to avoid catching signals that should not be caught by the + test framework is to rearrange the `test[]` so that the expression being + asserted cannot result in an uncaught signal. For example, save the result + of a computation into a variable first, and then use it in the `test[]`, + instead of invoking that computation inside the `test[]`. See + `unpythonic.test.test_conditions` for examples. + + Exceptions are always caught by `test[]`, because exceptions do not support + resumption; unlike with signals, the inner level of the call stack is already + destroyed by the time the exception is caught by the test construct. + """ + if syntax not in ("expr", "block"): + raise SyntaxError("test is an expr and block macro only") + + # Two-pass macros. + with dyn.let(_macro_expander=expander): + if syntax == "expr": + if args: + raise SyntaxError("test[] in expression mode does not take macro arguments") + return test_expr(tree) + else: # syntax == "block": + return test_block(block_body=tree, args=args) + +@parametricmacro +def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 + """[syntax, expr/block] Like `test`, but expect the expression to signal a condition. + + "Signal" as in `unpythonic.conditions.signal` and its sisters. + + Syntax:: + + test_signals[exctype, expr] + test_signals[exctype, expr, message] + + with test_signals[exctype]: + body0 + ... + + with test_signals[exctype, message]: + body0 + ... + + Example:: + + test_signals[ValueError, myfunc()] + test_signals[ValueError, myfunc(), "failure message"] + + The test succeeds, if `expr` signals a condition of type `exctype`, and the + signal propagates into the (implicit) handler inside the `test_signals[]` + construct. + + If `expr` returns normally, the test fails. + + If `expr` signals some other type of condition, or raises an exception, the + test errors. + + **Differences to `test[]`, `with test`**: + + As the focus of this construct is on signaling vs. returning normally, the + `the[]` mark is not supported. The block variant does not support `return`. + """ + if syntax not in ("expr", "block"): + raise SyntaxError("test_signals is an expr and block macro only") + + # Two-pass macros. + with dyn.let(_macro_expander=expander): + if syntax == "expr": + if args: + raise SyntaxError("test_signals[] in expression mode does not take macro arguments") + return test_expr_signals(tree) + else: # syntax == "block": + return test_block_signals(block_body=tree, args=args) + +@parametricmacro +def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 + """[syntax, expr/block] Like `test`, but expect the expression to raise an exception. + + Syntax:: + + test_raises[exctype, expr] + test_raises[exctype, expr, message] + + with test_raises[exctype]: + body0 + ... + + with test_raises[exctype, message]: + body0 + ... + + Example:: + + test_raises[TypeError, issubclass(1, int)] + test_raises[ValueError, myfunc()] + test_raises[ValueError, myfunc(), "failure message"] + + The test succeeds, if `expr` raises an exception of type `exctype`, and the + exception propagates into the (implicit) handler inside the `test_raises[]` + construct. + + If `expr` returns normally, the test fails. + + If `expr` signals a condition, or raises some other type of exception, the + test errors. + + **Differences to `test[]`, `with test`**: + + As the focus of this construct is on raising vs. returning normally, the + `the[]` mark is not supported. The block variant does not support `return`. + """ + if syntax not in ("expr", "block"): + raise SyntaxError("test_raises is an expr and block macro only") + + with dyn.let(_macro_expander=expander): + if syntax == "expr": + if args: + raise SyntaxError("test_raises[] in expression mode does not take macro arguments") + return test_expr_raises(tree) + else: # syntax == "block": + return test_block_raises(block_body=tree, args=args) + +def fail(tree, *, syntax, expander, **kw): # noqa: F811 + """[syntax, expr] Produce a test failure, unconditionally. + + Useful to e.g. mark a line of code that should not be reached in automated + tests, reaching which is therefore a test failure. + + Usage:: + + fail["human-readable reason"] + + which has the same effect as:: + + test[False, "human-readable reason"] + + except in the case of `fail[]`, the error message generating machinery is + special-cased to omit the source code expression, because it explicitly + states that the intent of the "test" is not actually to perform a test. + + See also `error[]`, `warn[]`. + """ + if syntax != "expr": + raise SyntaxError("fail is an expr macro only") + + # Expand outside in. The ordering shouldn't matter here. + # The underlying `test` machinery needs to access the expander. + with dyn.let(_macro_expander=expander): + return fail_expr(tree) + +def error(tree, *, syntax, expander, **kw): # noqa: F811 + """[syntax, expr] Produce a test error, unconditionally. + + Useful to e.g. indicate to the user that an optional dependency that could + be used to run some integration test is not installed. + + Usage:: + + error["human-readable reason"] + + See also `warn[]`, `fail[]`. + """ + if syntax != "expr": + raise SyntaxError("error is an expr macro only") + + # Expand outside in. The ordering shouldn't matter here. + # The underlying `test` machinery needs to access the expander. + with dyn.let(_macro_expander=expander): + return error_expr(tree) + +def warn(tree, *, syntax, expander, **kw): # noqa: F811 + """[syntax, expr] Produce a test warning, unconditionally. + + Useful to e.g. indicate that the Python interpreter or version the + tests are running on does not support a particular test, or to alert + about a non-essential TODO. + + A warning does not increase the failure count, so it will not cause + your CI workflow to break. + + Usage:: + + warn["human-readable reason"] + + See also `error[]`, `fail[]`. + """ + if syntax != "expr": + raise SyntaxError("warn is an expr macro only") + + # Expand outside in. The ordering shouldn't matter here. + # The underlying `test` machinery needs to access the expander. + with dyn.let(_macro_expander=expander): + return warn_expr(tree) + # ----------------------------------------------------------------------------- -# Helper for other macros to detect uses of the ones we define here. +# Helpers for other macros to detect uses of the ones we defined here. # Note the unexpanded `error[]` macro is distinguishable from a call to # the function `unpythonic.conditions.error`, because a macro invocation @@ -59,7 +446,7 @@ def istestmacro(tree): return isunexpandedtestmacro(tree) or isexpandedtestmacro(tree) # ----------------------------------------------------------------------------- -# Regular code, no macros yet. +# Run-time helpers. _fail = sym("_fail") # used by the fail[] macro _error = sym("_error") # used by the error[] macro @@ -344,8 +731,9 @@ def unpythonic_assert_raises(exctype, sourcecode, thunk, *, filename, lineno, me # ----------------------------------------------------------------------------- -# Syntax transformers for the macros. +# Syntax transformers +# fail/error/warn def _unconditional_error_expr(tree, syntaxname, marker): thetuple = q[(a[marker], a[tree])] # consider `test[tree, message]` thetuple = copy_location(thetuple, tree) @@ -359,60 +747,63 @@ def error_expr(tree): def warn_expr(tree): return _unconditional_error_expr(tree, "warn", q[h[_warn]]) -# ----------------------------------------------------------------------------- +# -------------------------------------------------------------------------------- # Expr variants. -def the(tree, **kw): - """[syntax, expr] In a test, mark a subexpression as the interesting one. - - Only meaningful inside a `test[]`, or inside a `with test` block. - - What `test[expr]` captures for reporting for human inspection upon - test failure: - - - If any `the[...]` are present, the subexpressions marked as `the[...]`. - - - Else if `expr` is a comparison, the LHS (leftmost term in case of - a chained comparison). So e.g. `test[x < 3]` needs no annotation - to do the right thing. This is a common use case, hence automatic. - - - Else nothing is captured; the value of the whole `expr` is reported. - - So the `the[...]` mark is useful in tests involving comparisons:: - - test[lower_limit < the[computeitem(...)]] - test[lower_limit < the[computeitem(...)] < upper_limit] - test[myconstant in the[computeset(...)]] - - especially if you need to capture several subexpressions:: +def test_expr(tree): + # Note we want the line number *before macro expansion*, so we capture it now. + ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] + filename = q[h[callsite_filename]()] + asserter = q[h[unpythonic_assert]] - test[the[counter()] < the[counter()]] + # test[expr, message] (like assert expr, message) + if type(tree) is Tuple and len(tree.elts) == 2: + tree, message = tree.elts + # test[expr] (like assert expr) + else: + message = q[None] - Note the above rules mean that if there is just one interesting - subexpression, and it is the leftmost term of a comparison, `the[...]` - is optional, although allowed (to explicitly document intent). - These have the same effect:: + # Before we edit the tree, get the source code in its pre-transformation + # state, so we can include that into the test failure message. + # + # We capture the source in the first pass, so that no macros in tree are + # expanded yet. For the same reason, we process the `the[]` marks in the + # first pass. + sourcecode = unparse(tree) - test[the[computeitem(...)] in myitems] - test[computeitem(...) in myitems] + envname = gensym("e") # for injecting the captured value - The `the[...]` mark passes the value through, and does not affect the - evaluation order of user code. + # Handle the `the[...]` marks, if any. + tree, the_exprs = _transform_important_subexpr(tree, envname=envname) + if not the_exprs and type(tree) is Compare: # inject the implicit the[] on the LHS + tree.left = _inject_value_recorder(envname, tree.left) - A `test[...]` may have multiple `the[...]`; the captured values are - gathered in a list that is shown upon test failure. + # End of first pass. + tree = dyn._macro_expander.visit(tree) - In case of nested tests, each `the[...]` is understood as belonging to - the lexically innermost surrounding test. + # We delay the execution of the test expr using a lambda, so + # `unpythonic_assert` can get control first before the expr runs. + # + # Also, we need the lambda for passing in the value capture environment + # for the `the[]` mark, anyway. + # + # We name it `testexpr` to make the stack trace more understandable. + # If you change the name, change it also in `unpythonic_assert`. + thelambda = q[lambda _: a[tree]] + thelambda.args.args[0] = arg(arg=envname) # inject the gensymmed parameter name + func_tree = q[h[namelambda]("testexpr")(a[thelambda])] # create the function that takes in the env - For `test_raises` and `test_signals`, the `the[...]` mark is not supported. - """ - raise SyntaxError("the[] is only meaningful inside a `test[]` or in a `with test` block") # pragma: no cover, not meant to hit the expander + return q[(a[asserter])(u[sourcecode], + a[func_tree], + filename=a[filename], + lineno=a[ln], + message=a[message])] # Destructuring utilities for marking a custom part of the expr # to be displayed upon test failure, using `the[...]`: # test[myconstant in the[computeset(...)]] # test[the[computeitem(...)] in expected_results_plus_uninteresting_items] +# These are used by `test_expr` and `test_block`. def _is_important_subexpr_mark(tree): return type(tree) is Subscript and type(tree.value) is Name and tree.value.id == "the" def _record_value(envname, sourcecode, value): @@ -450,54 +841,10 @@ def transform(self, tree): return tree, transformer.collected -def test_expr(tree): - # Note we want the line number *before macro expansion*, so we capture it now. - ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] - filename = q[h[callsite_filename]()] - asserter = q[h[unpythonic_assert]] - - # test[expr, message] (like assert expr, message) - if type(tree) is Tuple and len(tree.elts) == 2: - tree, message = tree.elts - # test[expr] (like assert expr) - else: - message = q[None] - - # Before we edit the tree, get the source code in its pre-transformation - # state, so we can include that into the test failure message. - # - # We capture the source in the first pass, so that no macros in tree are - # expanded yet. For the same reason, we process the `the[]` marks in the - # first pass. - sourcecode = unparse(tree) - - envname = gensym("e") # for injecting the captured value - - # Handle the `the[...]` marks, if any. - tree, the_exprs = _transform_important_subexpr(tree, envname=envname) - if not the_exprs and type(tree) is Compare: # inject the implicit the[] on the LHS - tree.left = _inject_value_recorder(envname, tree.left) - - # End of first pass. - tree = dyn._macro_expander.visit(tree) - - # We delay the execution of the test expr using a lambda, so - # `unpythonic_assert` can get control first before the expr runs. - # - # Also, we need the lambda for passing in the value capture environment - # for the `the[]` mark, anyway. - # - # We name it `testexpr` to make the stack trace more understandable. - # If you change the name, change it also in `unpythonic_assert`. - thelambda = q[lambda _: a[tree]] - thelambda.args.args[0] = arg(arg=envname) # inject the gensymmed parameter name - func_tree = q[h[namelambda]("testexpr")(a[thelambda])] # create the function that takes in the env - - return q[(a[asserter])(u[sourcecode], - a[func_tree], - filename=a[filename], - lineno=a[ln], - message=a[message])] +def test_expr_signals(tree): + return _test_expr_signals_or_raises(tree, "test_signals", q[h[unpythonic_assert_signals]]) +def test_expr_raises(tree): + return _test_expr_signals_or_raises(tree, "test_raises", q[h[unpythonic_assert_raises]]) def _test_expr_signals_or_raises(tree, syntaxname, asserter): ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] @@ -530,11 +877,6 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): lineno=a[ln], message=a[message])] -def test_expr_signals(tree): - return _test_expr_signals_or_raises(tree, "test_signals", q[h[unpythonic_assert_signals]]) -def test_expr_raises(tree): - return _test_expr_signals_or_raises(tree, "test_raises", q[h[unpythonic_assert_raises]]) - # ----------------------------------------------------------------------------- # Block variants. @@ -583,8 +925,8 @@ def test_block(block_body, args): message=a[message])] with q as newbody: def _insert_funcname_here_(_insert_envname_here_): - ... - a[thetest] + ... # to be filled in below + a[thetest] # call the asserter thefunc = newbody[0] thefunc.name = testblock_function_name thefunc.args.args[0] = arg(arg=envname) # inject the gensymmed parameter name @@ -611,6 +953,12 @@ def _insert_funcname_here_(_insert_envname_here_): return newbody + +def test_block_signals(block_body, args): + return _test_block_signals_or_raises(block_body, args, "test_signals", q[h[unpythonic_assert_signals]]) +def test_block_raises(block_body, args): + return _test_block_signals_or_raises(block_body, args, "test_raises", q[h[unpythonic_assert_raises]]) + def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): if not block_body: return [] # pragma: no cover, cannot happen through the public API. @@ -655,8 +1003,3 @@ def _insert_funcname_here_(): # no env needed, since `the[]` is not meaningful thefunc.name = testblock_function_name thefunc.body = block_body return newbody - -def test_block_signals(block_body, args): - return _test_block_signals_or_raises(block_body, args, "test_signals", q[h[unpythonic_assert_signals]]) -def test_block_raises(block_body, args): - return _test_block_signals_or_raises(block_body, args, "test_raises", q[h[unpythonic_assert_raises]]) From f7d1c4f1e0c366d1c73e4532d96ee96a3a224801 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 03:57:01 +0300 Subject: [PATCH 047/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index eb287ad2..2ddc4ee8 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -85,7 +85,9 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: have a common base class for all `unpythonic` `ASTMarker`s? +# TODO: Consistent naming for syntax transformers? `_macroname_transform`? `_macroname_stx`? + +# TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) From ddead42b8ea40a4bf8244ae5df40f3a4bbd483a7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 11:04:30 +0300 Subject: [PATCH 048/832] update module comments --- unpythonic/syntax/__init__.py | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 2ddc4ee8..8e2583bd 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -7,9 +7,10 @@ from ..dynassign import make_dynvar # -------------------------------------------------------------------------------- -# This module contains the macro interface and docstrings; the submodules -# contain the actual syntax transformers (regular functions that process ASTs) -# that implement the macros. +# This module only re-exports the macro interfaces so the macros can be imported +# by `from unpythonic.syntax import macros, ...`. The submodules contain the actual +# macro interfaces (and their docstrings), as well as the syntax transformers +# (i.e. regular functions that process ASTs) that implement the macros. # -------------------------------------------------------------------------------- # -------------------------------------------------------------------------------- @@ -74,10 +75,6 @@ # capture and just return an unexpanded `let` and another `letseq` (with one fewer binding), # similarly to how Racket implements `let*`. See `unpythonic.syntax.simplelet` for a demo. # -# - The macro interfaces and their docstrings could live inside the implementation modules, and this -# module could just re-export them. (A function being a macro is a feature of its *use site* where -# it is imported, by `from xxx import macros, ...`; `mcpyrate` has no macro registry.) -# # - Many macros could perhaps run in the outside-in pass. Some need a redesign for their AST analysis, # but much of that has been sufficiently abstracted (e.g. `unpythonic.syntax.letdoutil`) so that this # is mainly a case of carefully changing the analysis mode at all appropriate use sites. From 19f552900075a7c685a39e494db52dcb27ad1573 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 18:04:31 +0300 Subject: [PATCH 049/832] update contribution guidelines --- CONTRIBUTING.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c406b47e..69a0a881 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -201,14 +201,10 @@ As of the first half of 2021, the main target platforms are **CPython 3.8** and - *Macros are the nuclear option of software engineering.* - Only make a macro when a regular function can't do what is needed. - Sometimes a regular code core with a thin macro layer on top, to improve the user experience, is the appropriate solution for [minimizing magic](https://macropy3.readthedocs.io/en/latest/discussion.html#minimize-macro-magic). See `do`, `let`, `autocurry`, `forall` for examples. - - `unpythonic/syntax/__init__.py` is very long (> 2000 lines), because: - - For technical reasons, as of MacroPy 1.1.0b2, it's not possible to re-export macros defined in another module. (As of `unpythonic` 0.15, this is no longer relevant, since we use `mcpyrate`, which **can** re-export macros. So `unpythonic.syntax` may be revised in a future version of `unpythonic`.) - - Therefore, all macro entry points must reside in `unpythonic/syntax/__init__.py`, so that user code can `from unpythonic.syntax import macros, something`, without caring about how the `unpythonic.syntax` package is internally organized. - - The docstring must be placed on the macro entry point, so that the REPL will find it. This forces all macro docstrings into that one module. (That's less magic than injecting them dynamically when `unpythonic` boots up.) - A macro entry point can be just a thin wrapper around the relevant [*syntax transformer*](http://www.greghendershott.com/fear-of-macros/): a regular function, which takes and returns an AST. - - You can have an expr, block and decorator macro with the same name, in the same module, by making the macro interface into a dispatcher. See the `syntax` kw in `mcpyrate`. - - Syntax transformers can and should be sensibly organized into modules, just like any other regular (non-macro) code. - - But they don't need docstrings, since the macro entry point already has the docstring. + - You can have an expr, block and decorator macro with the same name, in the same module, by making the macro interface into a dispatcher. See the `syntax` kwarg in `mcpyrate`. + - Macros and syntax transformers should be sensibly organized into modules, just like any other regular (non-macro) code. + - The docstring should usually be placed on the macro entry point, so the syntax transformer typically does not need one. - If your syntax transformer (or another one it internally uses) needs `mcpyrate` `**kw` arguments: - Declare the relevant `**kw`s as parameters for the macro entry point, therefore requesting `mcpyrate` to provide them. Stuff them into `dyn` using `with dyn.let(...)`, and call your syntax transformer, which can then get the `**kw`s from `dyn`. See the existing macros for examples. - Using `dyn` keeps the syntax transformer call signatures clean, while limiting the dynamic extent of what is effectively a global assignment. If we used only function parameters, some of the high-level syntax transformers would have to declare `expander` just to pass it through, possibly through several layers, until it reaches the low-level syntax transformer that actually needs it. Avoiding such a parameter definition cascade is exactly the use case `dyn` was designed for. From 0af25416ad636dc35f4bd4382e85ec522bd3afb7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 18:23:58 +0300 Subject: [PATCH 050/832] port the `lispython` dialect example from pydialect --- unpythonic/dialects/__init__.py | 4 + unpythonic/dialects/lispython.py | 43 +++++++ unpythonic/dialects/tests/test_lispython.py | 134 ++++++++++++++++++++ 3 files changed, 181 insertions(+) create mode 100644 unpythonic/dialects/__init__.py create mode 100644 unpythonic/dialects/lispython.py create mode 100644 unpythonic/dialects/tests/test_lispython.py diff --git a/unpythonic/dialects/__init__.py b/unpythonic/dialects/__init__.py new file mode 100644 index 00000000..47111b58 --- /dev/null +++ b/unpythonic/dialects/__init__.py @@ -0,0 +1,4 @@ +# -*- coding: utf-8 -*- + +# re-exports +from .lispython import Lispython # noqa: F401 diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py new file mode 100644 index 00000000..c9157339 --- /dev/null +++ b/unpythonic/dialects/lispython.py @@ -0,0 +1,43 @@ +# -*- coding: utf-8 -*- +"""Lispython: the love child of Python and Scheme. + +Powered by `mcpyrate` and `unpythonic`. +""" + +__all__ = ["Lispython"] + +__version__ = '2.0.0' + +from mcpyrate.quotes import macros, q # noqa: F401 + +from mcpyrate.dialects import Dialect +from mcpyrate.splicing import splice_dialect + +class Lispython(Dialect): + """**Schemers rejoice!** + + Multiple musings mix in a lambda, + Lament no longer the lack of let. + Languish no longer labelless, lambda, + Linked lists cons and fold. + Tail-call into recursion divine, + The final value always provide. + """ + + def transform_ast(self, tree): # tree is an ast.Module + with q as template: + from unpythonic.syntax import (macros, tco, autoreturn, # noqa: F401, F811 + multilambda, quicklambda, namedlambda, f, + let, letseq, letrec, + dlet, dletseq, dletrec, + blet, bletseq, bletrec, + local, delete, do, do0, + let_syntax, abbrev, + cond) + # auxiliary syntax elements for the macros + from unpythonic.syntax import where, block, expr # noqa: F401, F811 + from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn # noqa: F401, F811 + with namedlambda, autoreturn, quicklambda, multilambda, tco: + __paste_here__ # noqa: F821, just a splicing marker. + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py new file mode 100644 index 00000000..04af9721 --- /dev/null +++ b/unpythonic/dialects/tests/test_lispython.py @@ -0,0 +1,134 @@ +# -*- coding: utf-8 -*- +"""Test the Lispython dialect.""" + +# See the `mcpyrate` dialects user manual: +# https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md + +from ...dialects import dialects, Lispython # noqa: F401 + +# Can use macros, too. +from ...syntax import macros, continuations, call_cc # noqa: F401 + +# `unpythonic` is effectively `lispython`'s stdlib; not everything gets imported by default. +from ...fold import foldl + +# Of course, all of Python's stdlib is available too. +# +# So is **any** Python library; the ability to use arbitrary Python libraries in +# a customized Python-based language is pretty much the whole point of dialects. +# +from operator import mul + +# TODO: use the test framework + +def main(): + assert prod((2, 3, 4)) == 24 # noqa: F821, bye missing battery, hello new dialect builtin + assert foldl(mul, 1, (2, 3, 4)) == 24 + + # cons, car, cdr, ll, llist are builtins (for more linked list utils, import them from unpythonic) + c = cons(1, 2) # noqa: F821 + assert tuple(c) == (1, 2) + assert car(c) == 1 # noqa: F821 + assert cdr(c) == 2 # noqa: F821 + assert ll(1, 2, 3) == llist((1, 2, 3)) # noqa: F821 + + # all unpythonic.syntax let[], letseq[], letrec[] constructs are builtins + # (including the decorator versions, let_syntax and abbrev) + x = let[(a, 21) in 2 * a] # noqa: F821 + assert x == 42 + + x = letseq[((a, 1), # noqa: F821 + (a, 2 * a), # noqa: F821 + (a, 2 * a)) in # noqa: F821 + a] # noqa: F821 + assert x == 4 + + # rackety cond + a = lambda x: cond[x < 0, "nope", # noqa: F821 + x % 2 == 0, "even", + "odd"] + assert a(-1) == "nope" + assert a(2) == "even" + assert a(3) == "odd" + + # auto-TCO (both in defs and lambdas), implicit return in tail position + def fact(n): + def f(k, acc): + if k == 1: + return acc # "return" still available for early return + f(k - 1, k * acc) + f(n, acc=1) + assert fact(4) == 24 + fact(5000) # no crash (and correct result, since Python uses bignums transparently) + + t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 + evenp(10000)] # no crash # noqa: F821 + assert t is True + + # lambdas are named automatically + square = lambda x: x**2 + assert square(3) == 9 + assert square.__name__ == "square" + + # the underscore (NOTE: due to this, "f" is a reserved name in lispython) + cube = f[_**3] # noqa: F821 + assert cube(3) == 27 + assert cube.__name__ == "cube" + + # lambdas can have multiple expressions and local variables + # + # If you need to return a literal list from a lambda, use an extra set of + # brackets; the outermost brackets always enable multiple-expression mode. + # + test = lambda x: [local[y << 2 * x], # noqa: F821, local[name << value] makes a local variable + y + 1] # noqa: F821 + assert test(10) == 21 + + a = lambda x: [local[t << x % 2], # noqa: F821 + cond[t == 0, "even", # noqa: F821 + t == 1, "odd", + None]] # cond[] requires an else branch + assert a(2) == "even" + assert a(3) == "odd" + + # actually the multiple-expression environment is an unpythonic.syntax.do[], + # which can be used in any expression position. + x = do[local[z << 2], # noqa: F821 + 3 * z] # noqa: F821 + assert x == 6 + + # do0[] is the same, but returns the value of the first expression instead of the last one. + x = do0[local[z << 3], # noqa: F821 + print("hi from do0, z is {}".format(z))] # noqa: F821 + assert x == 3 + + # MacroPy #21; namedlambda must be in its own with block in the + # dialect implementation or this particular combination will fail + # (uncaught jump, __name__ not set). + t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 + [local[x << evenp(100)], # noqa: F821, multi-expression let body is a do[] environment + (x, evenp.__name__, oddp.__name__)]] # noqa: F821 + assert t == (True, "evenp", "oddp") + + with continuations: # should be skipped by the implicit tco inserted by the dialect + k = None # kontinuation + def setk(*args, cc): + nonlocal k + k = cc # current continuation, i.e. where to go after setk() finishes + args # tuple means multiple-return-values + def doit(): + lst = ['the call returned'] + *more, = call_cc[setk('A')] + lst + list(more) + assert doit() == ['the call returned', 'A'] + # We can now send stuff into k, as long as it conforms to the + # signature of the assignment targets of the "call_cc". + assert k('again') == ['the call returned', 'again'] + assert k('thrice', '!') == ['the call returned', 'thrice', '!'] + + print("All tests PASSED") + +if __name__ == '__main__': + main() From 0689ade8f75d2f6df5f0b3c53439e7a4eb522f9b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 19:09:38 +0300 Subject: [PATCH 051/832] invoke dialect tests in runtests --- runtests.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/runtests.py b/runtests.py index f1276cd4..e928983e 100644 --- a/runtests.py +++ b/runtests.py @@ -38,7 +38,8 @@ def main(): # `test` (singular). testsets = (("regular code", (listtestmodules(os.path.join("unpythonic", "tests")) + listtestmodules(os.path.join("unpythonic", "net", "tests")))), - ("macros", listtestmodules(os.path.join("unpythonic", "syntax", "tests")))) + ("macros", listtestmodules(os.path.join("unpythonic", "syntax", "tests"))), + ("dialects", listtestmodules(os.path.join("unpythonic", "dialects", "tests")))) for tsname, modnames in testsets: with testset(tsname): for m in modnames: From f93339ec2fcd3c6ef1b063d6330cc908e926dba5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 19:09:56 +0300 Subject: [PATCH 052/832] add dialects module docstring --- unpythonic/dialects/__init__.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/unpythonic/dialects/__init__.py b/unpythonic/dialects/__init__.py index 47111b58..08a6e171 100644 --- a/unpythonic/dialects/__init__.py +++ b/unpythonic/dialects/__init__.py @@ -1,4 +1,15 @@ # -*- coding: utf-8 -*- +"""Dialects: Python the way you want it. + +These dialects, i.e. whole-module syntax transformations, are powered by +`mcpyrate`'s dialect subsystem. The user manual is at: + https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md + +We provide these dialects mainly to demonstrate how to use that subsystem +to customize Python beyond what a local macro expander can do. + +For examples of how to use the dialects, see the unit tests. +""" # re-exports from .lispython import Lispython # noqa: F401 From 5e3d40eab4555d0a03ab5986ca9c7f5bee7e6e03 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 19:10:14 +0300 Subject: [PATCH 053/832] improve comments --- unpythonic/syntax/letdoutil.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 3e4438b0..a8ed5e51 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -130,13 +130,16 @@ def islet(tree, expanded=True): # otherwise we should have an expr macro invocation if not type(tree) is Subscript: return False + # let[(k0, v0), ...][body] + # ^^^^^^^^^^^^^^^^^^ macro = tree.value + # let[(k0, v0), ...][body] + # ^^^^^ if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. expr = tree.slice else: expr = tree.slice.value exprnames = ("let", "letseq", "letrec", "let_syntax", "abbrev") - # let[(k0, v0), ...][body] if type(macro) is Subscript and type(macro.value) is Name: s = macro.value.id if any(s == x for x in exprnames): From e310be3dae7d210c8af8631dfe1b12a40e9009c6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 19:11:45 +0300 Subject: [PATCH 054/832] update CHANGELOG --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 203b094b..124085f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,8 +21,9 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - Migrate to the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander; **MacroPy support dropped**. This change facilitates future development of the macro parts of `unpythonic`. - **Macro arguments are now passed using brackets** `macroname[args]` instead of parentheses. - - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. + - Parentheses are still available **for decorator macros only** as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. + - Other macro kinds do not support parentheses to pass arguments; particularly, the `let` constructs will be confused if you attempt to pass macro arguments using parentheses. - As a result of the new macro expander, macro test coverage should now be reported correctly. - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **Any imports of these constructs in user code should be modified to point to the new locations.** From 1c4b76a3d736c4fd68af27849bebd48b920ccbe1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 19:12:59 +0300 Subject: [PATCH 055/832] add listhell and pytkell dialects --- unpythonic/dialects/__init__.py | 2 + unpythonic/dialects/listhell.py | 28 +++ unpythonic/dialects/pytkell.py | 42 +++++ unpythonic/dialects/tests/test_listhell.py | 102 ++++++++++ unpythonic/dialects/tests/test_pytkell.py | 205 +++++++++++++++++++++ 5 files changed, 379 insertions(+) create mode 100644 unpythonic/dialects/listhell.py create mode 100644 unpythonic/dialects/pytkell.py create mode 100644 unpythonic/dialects/tests/test_listhell.py create mode 100644 unpythonic/dialects/tests/test_pytkell.py diff --git a/unpythonic/dialects/__init__.py b/unpythonic/dialects/__init__.py index 08a6e171..67d6d7df 100644 --- a/unpythonic/dialects/__init__.py +++ b/unpythonic/dialects/__init__.py @@ -13,3 +13,5 @@ # re-exports from .lispython import Lispython # noqa: F401 +from .listhell import Listhell # noqa: F401 +from .pytkell import Pytkell # noqa: F401 diff --git a/unpythonic/dialects/listhell.py b/unpythonic/dialects/listhell.py new file mode 100644 index 00000000..552bdfb1 --- /dev/null +++ b/unpythonic/dialects/listhell.py @@ -0,0 +1,28 @@ +# -*- coding: utf-8 -*- +"""LisThEll: it's not Lisp, it's not Python, it's not Haskell. + +Powered by `mcpyrate` and `unpythonic`. +""" + +__all__ = ["Listhell"] + +__version__ = '2.0.0' + +from mcpyrate.quotes import macros, q # noqa: F401 + +from mcpyrate.dialects import Dialect +from mcpyrate.splicing import splice_dialect + +class Listhell(Dialect): + def transform_ast(self, tree): # tree is an ast.Module + with q as template: + __lang__ = "Listhell" # noqa: F841, just provide it to user code. + from unpythonic.syntax import macros, prefix, autocurry # noqa: F401, F811 + # auxiliary syntax elements for the macros + from unpythonic.syntax import q, u, kw # noqa: F401 + from unpythonic import apply # noqa: F401 + from unpythonic import composerc as compose # compose from Right, Currying # noqa: F401 + with prefix, autocurry: + __paste_here__ # noqa: F821, just a splicing marker. + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree diff --git a/unpythonic/dialects/pytkell.py b/unpythonic/dialects/pytkell.py new file mode 100644 index 00000000..1f27a038 --- /dev/null +++ b/unpythonic/dialects/pytkell.py @@ -0,0 +1,42 @@ +# -*- coding: utf-8 -*- +"""Pytkell: Python with automatic currying and lazy functions. + +Powered by `mcpyrate` and `unpythonic`. +""" + +__all__ = ["Pytkell"] + +__version__ = '2.0.0' + +from mcpyrate.quotes import macros, q # noqa: F401 + +from mcpyrate.dialects import Dialect +from mcpyrate.splicing import splice_dialect + +class Pytkell(Dialect): + def transform_ast(self, tree): # tree is an ast.Module + with q as template: + __lang__ = "Pytkell" # noqa: F841, just provide it to user code. + from unpythonic.syntax import (macros, lazy, lazyrec, lazify, autocurry, # noqa: F401, F811 + let, letseq, letrec, + dlet, dletseq, dletrec, + blet, bletseq, bletrec, + local, delete, do, do0, + cond, forall) + # auxiliary syntax elements for the macros + from unpythonic.syntax import where, insist, deny # noqa: F401 + # functions that have a haskelly feel to them + from unpythonic import (foldl, foldr, scanl, scanr, # noqa: F401 + s, imathify, gmathify, frozendict, + memoize, fupdate, fup, + gmemoize, imemoize, fimemoize, + islice, take, drop, split_at, first, second, nth, last, + flip, rotate) + from unpythonic import composerc as compose # compose from Right, Currying (Haskell's . operator) # noqa: F401 + # this is a bit lispy, but we're not going out of our way to provide + # a haskelly surface syntax for these. + from unpythonic import cons, car, cdr, ll, llist, nil # noqa: F401 + with lazify, autocurry: + __paste_here__ # noqa: F821, just a splicing marker. + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree diff --git a/unpythonic/dialects/tests/test_listhell.py b/unpythonic/dialects/tests/test_listhell.py new file mode 100644 index 00000000..f8cddbd9 --- /dev/null +++ b/unpythonic/dialects/tests/test_listhell.py @@ -0,0 +1,102 @@ +# -*- coding: utf-8 -*- +"""Test the LisThEll dialect.""" + +# from mcpyrate.debug import dialects, StepExpansion +from ...dialects import dialects, Listhell # noqa: F401 + +from ...syntax import macros, let, local, delete, do # noqa: F401 +from ...syntax import where # for let-where # noqa: F401 +from unpythonic import foldr, cons, nil, ll + +# TODO: use the test framework + +def runtests(): + # Function calls can be made in prefix notation, like in Lisps. + # The first element of a literal tuple is the function to call, + # the rest are its arguments. + (print, f"Hello from {__lang__}!") # noqa: F821, the dialect template defines it. + + x = 42 # can write any regular Python, too + + # quote operator q locally turns off the function-call transformation: + t1 = (q, 1, 2, (3, 4), 5) # q takes effect recursively # noqa: F821, the dialect template defines `q`. + t2 = (q, 17, 23, x) # unlike in Lisps, x refers to its value even in a quote # noqa: F821 + (print, t1, t2) + + # unquote operator u locally turns the transformation back on: + t3 = (q, (u, print, 42), (print, 42), "foo", "bar") # noqa: F821 + assert t3 == (q, None, (print, 42), "foo", "bar") # noqa: F821 + + # quotes nest; call transformation made when quote level == 0 + t4 = (q, (print, 42), (q, (u, u, print, 42)), "foo", "bar") # noqa: F821 + assert t4 == (q, (print, 42), (None,), "foo", "bar") # noqa: F821 + + # Be careful: + # + # In LisThEll, this means "call the 0-arg function `x`". + # But if `x` is not callable, `currycall` will return + # the value as-is (needed for interaction with `call_ec` + # and some other replace-def-with-value decorators). + assert (x,) == 42 + + # This means "the tuple where the first element is `x`" + (q, x) # noqa: F821 + + # give named args with kw(...) [it's syntax, not really a function!]: + def f(*, a, b): + return (q, a, b) # noqa: F821 + # in one kw(...), or... + assert (f, kw(a="hi there", b="foo")) == (q, "hi there", "foo") # noqa: F821 + # in several kw(...), doesn't matter + assert (f, kw(a="hi there"), kw(b="foo")) == (q, "hi there", "foo") # noqa: F821 + # in case of duplicate name across kws, rightmost wins + assert (f, kw(a="hi there"), kw(b="foo"), kw(b="bar")) == (q, "hi there", "bar") # noqa: F821 + + # give *args with unpythonic.fun.apply, like in Lisps: + lst = [1, 2, 3] + def g(*args, **kwargs): + return args + tuple(sorted(kwargs.items())) + assert (apply, g, lst) == (q, 1, 2, 3) # noqa: F821 + # lst goes last; may have other args first + assert (apply, g, "hi", "ho", lst) == (q, "hi", "ho", 1, 2, 3) # noqa: F821 + # named args in apply are also fine + assert (apply, g, "hi", "ho", lst, kw(myarg=4)) == (q, "hi", "ho", 1, 2, 3, ('myarg', 4)) # noqa: F821 + + # Function call transformation only applies to tuples in load context + # (i.e. NOT on the LHS of an assignment) + a, b = (q, 100, 200) # noqa: F821 + assert a == 100 and b == 200 + a, b = (q, b, a) # pythonic swap in prefix syntax; must quote RHS # noqa: F821 + assert a == 200 and b == 100 + + # the prefix syntax leaves alone the let binding syntax ((name0, value0), ...) + a = let[(x, 42)][x << x + 1] + assert a == 43 + + # but the RHSs of the bindings are transformed normally: + def double(x): + return 2 * x + a = let[(x, (double, 21))][x << x + 1] + assert a == 43 + + # similarly, the prefix syntax leaves the "body tuple" of a do alone + # (syntax, not semantically a tuple), but recurses into it: + a = do[1, 2, 3] + assert a == 3 + a = do[1, 2, (double, 3)] + assert a == 6 + + # the extra bracket syntax (implicit do) has no danger of confusion, as it's a list, not tuple + a = let[(x, 3)][[ + 1, + 2, + (double, x)]] + assert a == 6 + + my_map = lambda f: (foldr, (compose, cons, f), nil) # noqa: F821 + assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) # noqa: F821 + + (print, "All tests PASSED") + +if __name__ == '__main__': + (runtests,) diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py new file mode 100644 index 00000000..58db887b --- /dev/null +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -0,0 +1,205 @@ +# -*- coding: utf-8 -*- +"""Test the Pytkell dialect.""" + +# from mcpyrate.debug import dialects, StepExpansion +from ...dialects import dialects, Pytkell # noqa: F401 + +from ...syntax import macros, continuations, call_cc, tco # noqa: F401 +from ...misc import timer + +from types import FunctionType +from operator import add, mul + +# TODO: use the test framework + +def runtests(): + print(f"Hello from {__lang__}!") # noqa: F821, the dialect template defines it. + + # function definitions (both def and lambda) and calls are auto-curried + def add3(a, b, c): + return a + b + c + + a = add3(1) + assert isinstance(a, FunctionType) + a = a(2) + assert isinstance(a, FunctionType) + a = a(3) + assert isinstance(a, int) + + # actually partial evaluation so any of these works + assert add3(1)(2)(3) == 6 + assert add3(1, 2)(3) == 6 + assert add3(1)(2, 3) == 6 + assert add3(1, 2, 3) == 6 + + # arguments of a function call are auto-lazified (converted to promises, MacroPy lazy[]) + def addfirst2(a, b, c): + # a and b are read, so their promises are forced + # c is not used, so not evaluated either + return a + b + assert addfirst2(1)(2)(1 / 0) == 3 + + # let-bindings are auto-lazified + x = let[((x, 42), # noqa: F821 + (y, 1 / 0)) in x] # noqa: F821 + assert x == 42 + + # assignments are not (because they can imperatively update existing names) + try: + a = 1 / 0 + except ZeroDivisionError: + pass + else: + assert False, "expected a zero division error" + + # so if you want that, use lazy[] manually (it's a builtin in Pytkell) + a = lazy[1 / 0] # this blows up only when the value is read (name 'a' in Load context) # noqa: F821 + + # manually lazify items in a data structure literal, recursively (see unpythonic.syntax.lazyrec): + a = lazyrec[(1, 2, 3 / 0)] # noqa: F821 + assert a[:-1] == (1, 2) # reading a slice forces only that slice + + # laziness passes through + def g(a, b): + return a # b not used + def f(a, b): + return g(a, b) # b is passed along, but its value is not used + assert f(42, 1 / 0) == 42 + + def f(a, b): + return (a, b) + assert f(1, 2) == (1, 2) + assert (flip(f))(1, 2) == (2, 1) # NOTE flip reverses all (doesn't just flip the first two) # noqa: F821 + +# # TODO: this doesn't work, because curry sees f's arities as (2, 2) (kwarg handling!) +# assert (flip(f))(1, b=2) == (1, 2) # b -> kwargs + + # http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html + my_sum = foldl(add, 0) # noqa: F821 + my_prod = foldl(mul, 1) # noqa: F821 + my_map = lambda f: foldr(compose(cons, f), nil) # compose is unpythonic.fun.composerc # noqa: F821 + + assert my_sum(range(1, 5)) == 10 + assert my_prod(range(1, 5)) == 24 + assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) + + assert tuple(scanl(add, 0, (1, 2, 3))) == (0, 1, 3, 6) # noqa: F821 + assert tuple(scanr(add, 0, (1, 2, 3))) == (0, 3, 5, 6) # NOTE output ordering different from Haskell # noqa: F821 + + # let-in + x = let[(a, 21) in 2 * a] # noqa: F821 + assert x == 42 + + x = let[((a, 21), # noqa: F821 + (b, 17)) in # noqa: F821 + 2 * a + b] # noqa: F821 + assert x == 59 + + # let-where + x = let[2 * a, where(a, 21)] # noqa: F821 + assert x == 42 + + x = let[2 * a + b, # noqa: F821 + where((a, 21), # noqa: F821 + (b, 17))] # noqa: F821 + assert x == 59 + + # nondeterministic evaluation (essentially do-notation in the List monad) + # + # pythagorean triples + pt = forall[z << range(1, 21), # hypotenuse # noqa: F821 + x << range(1, z + 1), # shorter leg # noqa: F821 + y << range(x, z + 1), # longer leg # noqa: F821 + insist(x * x + y * y == z * z), # see also deny() # noqa: F821 + (x, y, z)] # noqa: F821 + assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), + (8, 15, 17), (9, 12, 15), (12, 16, 20)) + + # functional update for sequences + # + tup1 = (1, 2, 3, 4, 5) + tup2 = fup(tup1)[2:] << (10, 20, 30) # fup(sequence)[idx_or_slice] << sequence_of_values # noqa: F821 + assert tup2 == (1, 2, 10, 20, 30) + assert tup1 == (1, 2, 3, 4, 5) + + # immutable dict, with functional update + # + d1 = frozendict(foo='bar', bar='tavern') # noqa: F821 + d2 = frozendict(d1, bar='pub') # noqa: F821 + assert tuple(sorted(d1.items())) == (('bar', 'tavern'), ('foo', 'bar')) + assert tuple(sorted(d2.items())) == (('bar', 'pub'), ('foo', 'bar')) + + # s = mathematical Sequence (const, arithmetic, geometric, power) + # + assert last(take(10000, s(1, ...))) == 1 # noqa: F821 + assert last(take(5, s(0, 1, ...))) == 4 # noqa: F821 + assert last(take(5, s(1, 2, 4, ...))) == (1 * 2 * 2 * 2 * 2) # 16 # noqa: F821 + assert last(take(5, s(2, 4, 16, ...))) == (((((2)**2)**2)**2)**2) # 65536 # noqa: F821 + + # s() takes care to avoid roundoff + assert last(take(1001, s(0, 0.001, ...))) == 1 # noqa: F821 + + # iterables returned by s() support infix math + # (to add infix math support to some other iterable, m(iterable)) + c = s(1, 3, ...) + s(2, 4, ...) # noqa: F821 + assert tuple(take(5, c)) == (3, 7, 11, 15, 19) # noqa: F821 + assert tuple(take(5, c)) == (23, 27, 31, 35, 39) # consumed! # noqa: F821 + + # imemoize = memoize Iterable (makes a gfunc, drops math support) + # gmathify returns a new gfunc that adds infix math support + # to generators the original gfunc makes. + # + # see also gmemoize, fimemoize in unpythonic + # + mi = lambda x: gmathify(imemoize(x)) # noqa: F821 + a = mi(s(1, 3, ...)) # noqa: F821 + b = mi(s(2, 4, ...)) # noqa: F821 + c = lambda: a() + b() + assert tuple(take(5, c())) == (3, 7, 11, 15, 19) # noqa: F821 + assert tuple(take(5, c())) == (3, 7, 11, 15, 19) # now it's a new instance; no recomputation # noqa: F821 + + factorials = mi(scanl(mul, 1, s(1, 2, ...))) # 0!, 1!, 2!, ... # noqa: F821 + assert last(take(6, factorials())) == 120 # noqa: F821 + assert first(drop(5, factorials())) == 120 # noqa: F821 + + squares = s(1, 2, ...)**2 # noqa: F821 + assert last(take(10, squares)) == 100 # noqa: F821 + + harmonic = 1 / s(1, 2, ...) # noqa: F821 + assert last(take(10, harmonic)) == 1 / 10 # noqa: F821 + + # unpythonic's continuations are supported + with continuations: + k = None # kontinuation + def setk(*args, cc): + nonlocal k + k = cc # current continuation, i.e. where to go after setk() finishes + return args # tuple means multiple-return-values + def doit(): + lst = ['the call returned'] + *more, = call_cc[setk('A')] + return lst + list(more) + assert doit() == ['the call returned', 'A'] + # We can now send stuff into k, as long as it conforms to the + # signature of the assignment targets of the "call_cc". + assert k('again') == ['the call returned', 'again'] + assert k('thrice', '!') == ['the call returned', 'thrice', '!'] + + # as is unpythonic's tco + with tco: + def fact(n): + def f(k, acc): + if k == 1: + return acc + return f(k - 1, k * acc) + return f(n, 1) # TODO: doesn't work as f(n, acc=1) due to curry's kwarg handling + assert fact(4) == 24 + print("Performance...") + with timer() as tictoc: + fact(5000) # no crash, but Pytkell is a bit slow + print(" Time taken for factorial of 5000: {:g}s".format(tictoc.dt)) + + print("All tests PASSED") + +if __name__ == '__main__': + runtests() From 77a6002d4135bb38dc8d0d8d1989895e3f4ee869 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 19:13:44 +0300 Subject: [PATCH 056/832] make lispython dialect set __lang__; name the test function `runtests` --- unpythonic/dialects/lispython.py | 1 + unpythonic/dialects/tests/test_lispython.py | 10 ++++------ 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index c9157339..ab1d8269 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -26,6 +26,7 @@ class Lispython(Dialect): def transform_ast(self, tree): # tree is an ast.Module with q as template: + __lang__ = "Lispython" # noqa: F841, just provide it to user code. from unpythonic.syntax import (macros, tco, autoreturn, # noqa: F401, F811 multilambda, quicklambda, namedlambda, f, let, letseq, letrec, diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index 04af9721..826de1fa 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -1,12 +1,8 @@ # -*- coding: utf-8 -*- """Test the Lispython dialect.""" -# See the `mcpyrate` dialects user manual: -# https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md - from ...dialects import dialects, Lispython # noqa: F401 -# Can use macros, too. from ...syntax import macros, continuations, call_cc # noqa: F401 # `unpythonic` is effectively `lispython`'s stdlib; not everything gets imported by default. @@ -21,7 +17,9 @@ # TODO: use the test framework -def main(): +def runtests(): + print(f"Hello from {__lang__}!") # noqa: F821, the dialect template defines it. + assert prod((2, 3, 4)) == 24 # noqa: F821, bye missing battery, hello new dialect builtin assert foldl(mul, 1, (2, 3, 4)) == 24 @@ -131,4 +129,4 @@ def doit(): print("All tests PASSED") if __name__ == '__main__': - main() + runtests() From 6fed79eaf1b0a57745a60b82aa82eeb2056320f1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 23:40:08 +0300 Subject: [PATCH 057/832] allow parentheses to pass macro arguments in letdoutil --- CHANGELOG.md | 3 +-- unpythonic/syntax/letdoutil.py | 43 +++++++++++++++++++++++++--------- 2 files changed, 33 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 124085f8..203b094b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,9 +21,8 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - Migrate to the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander; **MacroPy support dropped**. This change facilitates future development of the macro parts of `unpythonic`. - **Macro arguments are now passed using brackets** `macroname[args]` instead of parentheses. - - Parentheses are still available **for decorator macros only** as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. + - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. - - Other macro kinds do not support parentheses to pass arguments; particularly, the `let` constructs will be confused if you attempt to pass macro arguments using parentheses. - As a result of the new macro expander, macro test coverage should now be reported correctly. - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **Any imports of these constructs in user code should be modified to point to the new locations.** diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index a8ed5e51..97752649 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -131,10 +131,12 @@ def islet(tree, expanded=True): if not type(tree) is Subscript: return False # let[(k0, v0), ...][body] + # let((k0, v0), ...)[body] # ^^^^^^^^^^^^^^^^^^ macro = tree.value # let[(k0, v0), ...][body] - # ^^^^^ + # let((k0, v0), ...)[body] + # ^^^^ if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. expr = tree.slice else: @@ -144,6 +146,10 @@ def islet(tree, expanded=True): s = macro.value.id if any(s == x for x in exprnames): return ("lispy_expr", s) + elif type(macro) is Call and type(macro.func) is Name: # alternative parenthesis syntax to pass macro arguments + s = macro.func.id + if any(s == x for x in exprnames): + return ("lispy_expr", s) # The haskelly syntaxes are only available as a let expression (no decorator form). elif type(macro) is Name: s = macro.id @@ -288,6 +294,11 @@ class UnexpandedLetView: let[((k0, v0), ...) in body] # haskelly expression let[body, where((k0, v0), ...)] # haskelly expression, inverted + Lispy expressions are supported also using the old parenthesis syntax + to pass macro parameters:: + + let((k0, v0), ...)[body] # lispy expression + In addition, we also support *just the bracketed part* of the haskelly formats. This is to make it easier for the macro interface to destructure these forms (for sending into the ``let`` syntax transformer). So these @@ -365,12 +376,17 @@ def _getbindings(self): else: theargs = self._tree.slice.value return canonize_bindings(theargs.elts) - elif t == "lispy_expr": # Subscript inside a Subscript, (let[...])[...] - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - theargs = self._tree.value.slice - else: - theargs = self._tree.value.slice.value - return canonize_bindings(theargs.elts) + elif t == "lispy_expr": + # Subscript inside a Subscript, (let[...])[...] + if type(self._tree.value) is Subscript: + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + theargs = self._tree.value.slice.elts + else: + theargs = self._tree.value.slice.value.elts + # Call inside a Subscript, (let(...))[...], parenthesis syntax to pass macro arguments + else: # type(self._tree.value) is Call: + theargs = self._tree.value.args + return canonize_bindings(theargs) else: # haskelly let, let[(...) in ...], let[..., where(...)] theexpr = self._theexpr_ref() if t == "in_expr": @@ -389,10 +405,15 @@ def _setbindings(self, newbindings): else: self._tree.slice.value.elts = newbindings elif t == "lispy_expr": - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - self._tree.value.slice.elts = newbindings - else: - self._tree.value.slice.value.elts = newbindings + # Subscript inside a Subscript, (let[...])[...] + if type(self._tree.value) is Subscript: + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + self._tree.value.slice.elts = newbindings + else: + self._tree.value.slice.value.elts = newbindings + # Call inside a Subscript, (let(...))[...], parenthesis syntax to pass macro arguments + else: # type(self._tree.value) is Call: + self._tree.value.args = newbindings else: theexpr = self._theexpr_ref() if t == "in_expr": From efbf4cb5a62343de930e47ab6a3a2b7b5e4771cd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 1 May 2021 23:40:21 +0300 Subject: [PATCH 058/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 8e2583bd..e0342bdc 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -82,6 +82,12 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. +# TODO: debugging: +# TODO: investigate error in pytkell test when run by the runner (works fine when run as a single module) +# +# TODO: The HasThon test (grep for it), when putting the macros in the wrong order on purpose, +# TODO: confuses the call site filename detector of the test framework. Investigate. + # TODO: Consistent naming for syntax transformers? `_macroname_transform`? `_macroname_stx`? # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? @@ -130,9 +136,6 @@ # TODO: with mcpyrate, do we really need to set `ctx` in our macros? (does our macro code need it?) -# TODO: The HasThon test (grep for it), when putting the macros in the wrong order on purpose, -# TODO: confuses the call site filename detector of the test framework. Investigate. - # TODO: Move dialect examples from `pydialect` into a new package, `unpythonic.dialects`. # TODO: `mcpyrate` now provides the necessary infrastructure, while `unpythonic` has the macros # TODO: needed to make interesting things happen. Update docs accordingly for both projects. From cec29ffb881242cdd7d72b2d348949af719bb249 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 00:16:39 +0300 Subject: [PATCH 059/832] Use `unpythonic.test.fixtures` to test dialect examples, too --- unpythonic/dialects/tests/test_lispython.py | 206 +++++++------ unpythonic/dialects/tests/test_listhell.py | 146 ++++----- unpythonic/dialects/tests/test_pytkell.py | 323 ++++++++++---------- 3 files changed, 353 insertions(+), 322 deletions(-) diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index 826de1fa..cf6ac336 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -3,7 +3,10 @@ from ...dialects import dialects, Lispython # noqa: F401 -from ...syntax import macros, continuations, call_cc # noqa: F401 +from ...syntax import macros, test, the # noqa: F401 +from ...test.fixtures import session, testset + +from ...syntax import macros, continuations, call_cc # noqa: F401, F811 # `unpythonic` is effectively `lispython`'s stdlib; not everything gets imported by default. from ...fold import foldl @@ -15,118 +18,129 @@ # from operator import mul -# TODO: use the test framework - def runtests(): print(f"Hello from {__lang__}!") # noqa: F821, the dialect template defines it. - assert prod((2, 3, 4)) == 24 # noqa: F821, bye missing battery, hello new dialect builtin - assert foldl(mul, 1, (2, 3, 4)) == 24 - - # cons, car, cdr, ll, llist are builtins (for more linked list utils, import them from unpythonic) - c = cons(1, 2) # noqa: F821 - assert tuple(c) == (1, 2) - assert car(c) == 1 # noqa: F821 - assert cdr(c) == 2 # noqa: F821 - assert ll(1, 2, 3) == llist((1, 2, 3)) # noqa: F821 - - # all unpythonic.syntax let[], letseq[], letrec[] constructs are builtins - # (including the decorator versions, let_syntax and abbrev) - x = let[(a, 21) in 2 * a] # noqa: F821 - assert x == 42 - - x = letseq[((a, 1), # noqa: F821 - (a, 2 * a), # noqa: F821 - (a, 2 * a)) in # noqa: F821 - a] # noqa: F821 - assert x == 4 - - # rackety cond - a = lambda x: cond[x < 0, "nope", # noqa: F821 - x % 2 == 0, "even", - "odd"] - assert a(-1) == "nope" - assert a(2) == "even" - assert a(3) == "odd" + with testset("dialect builtins"): + test[prod((2, 3, 4)) == 24] # noqa: F821, bye missing battery, hello new dialect builtin + test[foldl(mul, 1, (2, 3, 4)) == 24] + + # cons, car, cdr, ll, llist are builtins (for more linked list utils, import them from unpythonic) + c = cons(1, 2) # noqa: F821 + test[tuple(c) == (1, 2)] + test[car(c) == 1] # noqa: F821 + test[cdr(c) == 2] # noqa: F821 + test[ll(1, 2, 3) == llist((1, 2, 3))] # noqa: F821 + + # all unpythonic.syntax let[], letseq[], letrec[] constructs are builtins + # (including the decorator versions, let_syntax and abbrev) + x = let[(a, 21) in 2 * a] # noqa: F821 + test[x == 42] + + x = letseq[((a, 1), # noqa: F821 + (a, 2 * a), # noqa: F821 + (a, 2 * a)) in # noqa: F821 + a] # noqa: F821 + test[x == 4] + + # rackety cond + a = lambda x: cond[x < 0, "nope", # noqa: F821 + x % 2 == 0, "even", + "odd"] + test[a(-1) == "nope"] + test[a(2) == "even"] + test[a(3) == "odd"] # auto-TCO (both in defs and lambdas), implicit return in tail position - def fact(n): - def f(k, acc): - if k == 1: - return acc # "return" still available for early return - f(k - 1, k * acc) - f(n, acc=1) - assert fact(4) == 24 - fact(5000) # no crash (and correct result, since Python uses bignums transparently) - - t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 - evenp(10000)] # no crash # noqa: F821 - assert t is True + with testset("implicit tco, implicit autoreturn"): + def fact(n): + def f(k, acc): + if k == 1: + return acc # "return" still available for early return + f(k - 1, k * acc) + f(n, acc=1) + test[fact(4) == 24] + fact(5000) # no crash (and correct result, since Python uses bignums transparently) + + t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 + evenp(10000)] # no crash # noqa: F821 + test[t is True] # lambdas are named automatically - square = lambda x: x**2 - assert square(3) == 9 - assert square.__name__ == "square" + with testset("implicit namedlambda"): + square = lambda x: x**2 + test[square(3) == 9] + test[square.__name__ == "square"] - # the underscore (NOTE: due to this, "f" is a reserved name in lispython) - cube = f[_**3] # noqa: F821 - assert cube(3) == 27 - assert cube.__name__ == "cube" + # the underscore (NOTE: due to this, "f" is a reserved name in lispython) + cube = f[_**3] # noqa: F821 + test[cube(3) == 27] + test[cube.__name__ == "cube"] # lambdas can have multiple expressions and local variables # # If you need to return a literal list from a lambda, use an extra set of # brackets; the outermost brackets always enable multiple-expression mode. # - test = lambda x: [local[y << 2 * x], # noqa: F821, local[name << value] makes a local variable - y + 1] # noqa: F821 - assert test(10) == 21 - - a = lambda x: [local[t << x % 2], # noqa: F821 - cond[t == 0, "even", # noqa: F821 - t == 1, "odd", - None]] # cond[] requires an else branch - assert a(2) == "even" - assert a(3) == "odd" - - # actually the multiple-expression environment is an unpythonic.syntax.do[], - # which can be used in any expression position. - x = do[local[z << 2], # noqa: F821 - 3 * z] # noqa: F821 - assert x == 6 - - # do0[] is the same, but returns the value of the first expression instead of the last one. - x = do0[local[z << 3], # noqa: F821 - print("hi from do0, z is {}".format(z))] # noqa: F821 - assert x == 3 + with testset("implicit multilambda"): + mylam = lambda x: [local[y << 2 * x], # noqa: F821, local[name << value] makes a local variable + y + 1] # noqa: F821 + test[mylam(10) == 21] + + a = lambda x: [local[t << x % 2], # noqa: F821 + cond[t == 0, "even", # noqa: F821 + t == 1, "odd", + None]] # cond[] requires an else branch + test[a(2) == "even"] + test[a(3) == "odd"] # MacroPy #21; namedlambda must be in its own with block in the # dialect implementation or this particular combination will fail # (uncaught jump, __name__ not set). - t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 - [local[x << evenp(100)], # noqa: F821, multi-expression let body is a do[] environment - (x, evenp.__name__, oddp.__name__)]] # noqa: F821 - assert t == (True, "evenp", "oddp") - - with continuations: # should be skipped by the implicit tco inserted by the dialect - k = None # kontinuation - def setk(*args, cc): - nonlocal k - k = cc # current continuation, i.e. where to go after setk() finishes - args # tuple means multiple-return-values - def doit(): - lst = ['the call returned'] - *more, = call_cc[setk('A')] - lst + list(more) - assert doit() == ['the call returned', 'A'] - # We can now send stuff into k, as long as it conforms to the - # signature of the assignment targets of the "call_cc". - assert k('again') == ['the call returned', 'again'] - assert k('thrice', '!') == ['the call returned', 'thrice', '!'] - - print("All tests PASSED") + # + # With `mcpyrate` this shouldn't matter, but we're keeping the example. + with testset("autonamed letrec lambdas, multiple-expression let body"): + t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 + [local[x << evenp(100)], # noqa: F821, multi-expression let body is a do[] environment + (x, evenp.__name__, oddp.__name__)]] # noqa: F821 + test[t == (True, "evenp", "oddp")] + + # actually the multiple-expression environment is an unpythonic.syntax.do[], + # which can be used in any expression position. + with testset("do and do0"): + x = do[local[z << 2], # noqa: F821 + 3 * z] # noqa: F821 + test[x == 6] + + # do0[] is the same, but returns the value of the first expression instead of the last one. + x = do0[local[z << 3], # noqa: F821 + print("hi from do0, z is {}".format(z))] # noqa: F821 + test[x == 3] + + with testset("integration with continuations"): + with continuations: # should be skipped by the implicit tco inserted by the dialect + k = None # kontinuation + def setk(*args, cc): + nonlocal k + k = cc # current continuation, i.e. where to go after setk() finishes + args # tuple means multiple-return-values + def doit(): + lst = ['the call returned'] + *more, = call_cc[setk('A')] + lst + list(more) + test[doit() == ['the call returned', 'A']] + # We can now send stuff into k, as long as it conforms to the + # signature of the assignment targets of the "call_cc". + test[k('again') == ['the call returned', 'again']] + test[k('thrice', '!') == ['the call returned', 'thrice', '!']] + + # We must have some statement here to make the implicit autoreturn happy, + # because the continuations testset is the last one, and the top level of + # a `with continuations` block is not allowed to have a `return`. + pass if __name__ == '__main__': - runtests() + with session(__file__): + runtests() diff --git a/unpythonic/dialects/tests/test_listhell.py b/unpythonic/dialects/tests/test_listhell.py index f8cddbd9..d469f1ae 100644 --- a/unpythonic/dialects/tests/test_listhell.py +++ b/unpythonic/dialects/tests/test_listhell.py @@ -4,12 +4,13 @@ # from mcpyrate.debug import dialects, StepExpansion from ...dialects import dialects, Listhell # noqa: F401 -from ...syntax import macros, let, local, delete, do # noqa: F401 +from ...syntax import macros, test # noqa: F401 +from ...test.fixtures import session, testset + +from ...syntax import macros, let, local, delete, do # noqa: F401, F811 from ...syntax import where # for let-where # noqa: F401 from unpythonic import foldr, cons, nil, ll -# TODO: use the test framework - def runtests(): # Function calls can be made in prefix notation, like in Lisps. # The first element of a literal tuple is the function to call, @@ -23,80 +24,89 @@ def runtests(): t2 = (q, 17, 23, x) # unlike in Lisps, x refers to its value even in a quote # noqa: F821 (print, t1, t2) - # unquote operator u locally turns the transformation back on: - t3 = (q, (u, print, 42), (print, 42), "foo", "bar") # noqa: F821 - assert t3 == (q, None, (print, 42), "foo", "bar") # noqa: F821 - - # quotes nest; call transformation made when quote level == 0 - t4 = (q, (print, 42), (q, (u, u, print, 42)), "foo", "bar") # noqa: F821 - assert t4 == (q, (print, 42), (None,), "foo", "bar") # noqa: F821 - - # Be careful: - # - # In LisThEll, this means "call the 0-arg function `x`". - # But if `x` is not callable, `currycall` will return - # the value as-is (needed for interaction with `call_ec` - # and some other replace-def-with-value decorators). - assert (x,) == 42 - - # This means "the tuple where the first element is `x`" - (q, x) # noqa: F821 + # Calls to the test framework are written with pythonic function call notation, + # because if the `prefix` macro isn't working, then writing them in prefix notation + # could cause a crash while testing. + with testset("quoting"): + # unquote operator u locally turns the transformation back on: + t3 = (q, (u, print, 42), (print, 42), "foo", "bar") # noqa: F821 + test[t3 == (q, None, (print, 42), "foo", "bar")] # noqa: F821 + + # quotes nest; call transformation made when quote level == 0 + t4 = (q, (print, 42), (q, (u, u, print, 42)), "foo", "bar") # noqa: F821 + test[t4 == (q, (print, 42), (None,), "foo", "bar")] # noqa: F821 + + # Be careful: + # + # In LisThEll, `(x,)` means "call the 0-arg function `x`". + # But if `x` is not callable, `currycall` will return + # the value as-is (needed for interaction with `call_ec` + # and some other replace-def-with-value decorators). + # + # `(q, x)` means "the tuple where the first element is `x`". + test[(x,) == 42] + test[(q, x) == (tuple, [x])] # noqa: F821 # give named args with kw(...) [it's syntax, not really a function!]: - def f(*, a, b): - return (q, a, b) # noqa: F821 - # in one kw(...), or... - assert (f, kw(a="hi there", b="foo")) == (q, "hi there", "foo") # noqa: F821 - # in several kw(...), doesn't matter - assert (f, kw(a="hi there"), kw(b="foo")) == (q, "hi there", "foo") # noqa: F821 - # in case of duplicate name across kws, rightmost wins - assert (f, kw(a="hi there"), kw(b="foo"), kw(b="bar")) == (q, "hi there", "bar") # noqa: F821 + with testset("named arguments with kw()"): + def f(*, a, b): + return (q, a, b) # noqa: F821 + # in one kw(...), or... + test[(f, kw(a="hi there", b="foo")) == (q, "hi there", "foo")] # noqa: F821 + # in several kw(...), doesn't matter + test[(f, kw(a="hi there"), kw(b="foo")) == (q, "hi there", "foo")] # noqa: F821 + # in case of duplicate name across kws, rightmost wins + test[(f, kw(a="hi there"), kw(b="foo"), kw(b="bar")) == (q, "hi there", "bar")] # noqa: F821 # give *args with unpythonic.fun.apply, like in Lisps: - lst = [1, 2, 3] - def g(*args, **kwargs): - return args + tuple(sorted(kwargs.items())) - assert (apply, g, lst) == (q, 1, 2, 3) # noqa: F821 - # lst goes last; may have other args first - assert (apply, g, "hi", "ho", lst) == (q, "hi", "ho", 1, 2, 3) # noqa: F821 - # named args in apply are also fine - assert (apply, g, "hi", "ho", lst, kw(myarg=4)) == (q, "hi", "ho", 1, 2, 3, ('myarg', 4)) # noqa: F821 + with testset("starargs with apply()"): + lst = [1, 2, 3] + def g(*args, **kwargs): + return args + (tuple, (sorted, (kwargs.items,))) + test[(apply, g, lst) == (q, 1, 2, 3)] # noqa: F821 + # lst goes last; may have other args first + test[(apply, g, "hi", "ho", lst) == (q, "hi", "ho", 1, 2, 3)] # noqa: F821 + # named args in apply are also fine + test[(apply, g, "hi", "ho", lst, kw(myarg=4)) == (q, "hi", "ho", 1, 2, 3, ('myarg', 4))] # noqa: F821 # Function call transformation only applies to tuples in load context # (i.e. NOT on the LHS of an assignment) - a, b = (q, 100, 200) # noqa: F821 - assert a == 100 and b == 200 - a, b = (q, b, a) # pythonic swap in prefix syntax; must quote RHS # noqa: F821 - assert a == 200 and b == 100 - - # the prefix syntax leaves alone the let binding syntax ((name0, value0), ...) - a = let[(x, 42)][x << x + 1] - assert a == 43 - - # but the RHSs of the bindings are transformed normally: - def double(x): - return 2 * x - a = let[(x, (double, 21))][x << x + 1] - assert a == 43 + with testset("no transform on LHS of assignment"): + a, b = (q, 100, 200) # noqa: F821 + test[a == 100 and b == 200] + a, b = (q, b, a) # pythonic swap in prefix syntax; must quote RHS # noqa: F821 + test[a == 200 and b == 100] + + with testset("transform of let bindings"): + # the prefix syntax leaves alone the let binding syntax ((name0, value0), ...) + a = let[(x, 42)][x << x + 1] + test[a == 43] + + # but the RHSs of the bindings are transformed normally: + def double(x): + return 2 * x + a = let[(x, (double, 21))][x << x + 1] + test[a == 43] # similarly, the prefix syntax leaves the "body tuple" of a do alone # (syntax, not semantically a tuple), but recurses into it: - a = do[1, 2, 3] - assert a == 3 - a = do[1, 2, (double, 3)] - assert a == 6 - - # the extra bracket syntax (implicit do) has no danger of confusion, as it's a list, not tuple - a = let[(x, 3)][[ - 1, - 2, - (double, x)]] - assert a == 6 - - my_map = lambda f: (foldr, (compose, cons, f), nil) # noqa: F821 - assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) # noqa: F821 - - (print, "All tests PASSED") + with testset("transform of do body"): + a = do[1, 2, 3] + test[a == 3] + a = do[1, 2, (double, 3)] + test[a == 6] + + # the extra bracket syntax (implicit do) has no danger of confusion, as it's a list, not tuple + a = let[(x, 3)][[ + 1, + 2, + (double, x)]] + test[a == 6] + + with testset("final example"): + my_map = lambda f: (foldr, (compose, cons, f), nil) # noqa: F821 + test[(my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6)] # noqa: F821 if __name__ == '__main__': - (runtests,) + with (session, __file__): + (runtests,) diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index 58db887b..894a3d05 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -4,146 +4,150 @@ # from mcpyrate.debug import dialects, StepExpansion from ...dialects import dialects, Pytkell # noqa: F401 -from ...syntax import macros, continuations, call_cc, tco # noqa: F401 +from ...syntax import macros, test, the, test_raises # noqa: F401 +from ...test.fixtures import session, testset + +from ...syntax import macros, continuations, call_cc, tco # noqa: F401, F811 from ...misc import timer from types import FunctionType from operator import add, mul -# TODO: use the test framework - def runtests(): print(f"Hello from {__lang__}!") # noqa: F821, the dialect template defines it. # function definitions (both def and lambda) and calls are auto-curried - def add3(a, b, c): - return a + b + c - - a = add3(1) - assert isinstance(a, FunctionType) - a = a(2) - assert isinstance(a, FunctionType) - a = a(3) - assert isinstance(a, int) - - # actually partial evaluation so any of these works - assert add3(1)(2)(3) == 6 - assert add3(1, 2)(3) == 6 - assert add3(1)(2, 3) == 6 - assert add3(1, 2, 3) == 6 - - # arguments of a function call are auto-lazified (converted to promises, MacroPy lazy[]) - def addfirst2(a, b, c): - # a and b are read, so their promises are forced - # c is not used, so not evaluated either - return a + b - assert addfirst2(1)(2)(1 / 0) == 3 - - # let-bindings are auto-lazified - x = let[((x, 42), # noqa: F821 - (y, 1 / 0)) in x] # noqa: F821 - assert x == 42 - - # assignments are not (because they can imperatively update existing names) - try: - a = 1 / 0 - except ZeroDivisionError: - pass - else: - assert False, "expected a zero division error" - - # so if you want that, use lazy[] manually (it's a builtin in Pytkell) - a = lazy[1 / 0] # this blows up only when the value is read (name 'a' in Load context) # noqa: F821 - - # manually lazify items in a data structure literal, recursively (see unpythonic.syntax.lazyrec): - a = lazyrec[(1, 2, 3 / 0)] # noqa: F821 - assert a[:-1] == (1, 2) # reading a slice forces only that slice - - # laziness passes through - def g(a, b): - return a # b not used - def f(a, b): - return g(a, b) # b is passed along, but its value is not used - assert f(42, 1 / 0) == 42 - - def f(a, b): - return (a, b) - assert f(1, 2) == (1, 2) - assert (flip(f))(1, 2) == (2, 1) # NOTE flip reverses all (doesn't just flip the first two) # noqa: F821 - -# # TODO: this doesn't work, because curry sees f's arities as (2, 2) (kwarg handling!) -# assert (flip(f))(1, b=2) == (1, 2) # b -> kwargs + with testset("implicit autocurry"): + def add3(a, b, c): + return a + b + c + + a = add3(1) + test[isinstance(the[a], FunctionType)] + a = a(2) + test[isinstance(the[a], FunctionType)] + a = a(3) + test[isinstance(the[a], int)] + + # actually partial evaluation so any of these works + test[add3(1)(2)(3) == 6] + test[add3(1, 2)(3) == 6] + test[add3(1)(2, 3) == 6] + test[add3(1, 2, 3) == 6] + + # arguments of a function call are auto-lazified (converted to promises, lazy[]) + with testset("implicit lazify"): + def addfirst2(a, b, c): + # a and b are read, so their promises are forced + # c is not used, so not evaluated either + return a + b + test[addfirst2(1)(2)(1 / 0) == 3] + + # let-bindings are auto-lazified + with test["y is unused, so it should not be evaluated"]: + x = let[((x, 42), # noqa: F821 + (y, 1 / 0)) in x] # noqa: F821 + return x == 42 # access `x`, to force the promise + + # assignments are not (because they can imperatively update existing names) + with test_raises[ZeroDivisionError]: + a = 1 / 0 + + # so if you want that, use lazy[] manually (it's a builtin in Pytkell) + with test: + a = lazy[1 / 0] # this blows up only when the value is read (name 'a' in Load context) # noqa: F821 + + # manually lazify items in a data structure literal, recursively (see unpythonic.syntax.lazyrec): + with test: + a = lazyrec[(1, 2, 3 / 0)] # noqa: F821 + return a[:-1] == (1, 2) # reading a slice forces only that slice + + # laziness passes through + def g(a, b): + return a # b not used + def f(a, b): + return g(a, b) # b is passed along, but its value is not used + test[f(42, 1 / 0) == 42] + + def f(a, b): + return (a, b) + test[f(1, 2) == (1, 2)] + test[(flip(f))(1, 2) == (2, 1)] # NOTE flip reverses all (doesn't just flip the first two) # noqa: F821 + + # # TODO: this doesn't work, because curry sees f's arities as (2, 2) (kwarg handling!) + # test[(flip(f))(1, b=2) == (1, 2)] # b -> kwargs # http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html - my_sum = foldl(add, 0) # noqa: F821 - my_prod = foldl(mul, 1) # noqa: F821 - my_map = lambda f: foldr(compose(cons, f), nil) # compose is unpythonic.fun.composerc # noqa: F821 + with testset("iterables"): + my_sum = foldl(add, 0) # noqa: F821 + my_prod = foldl(mul, 1) # noqa: F821 + my_map = lambda f: foldr(compose(cons, f), nil) # compose is unpythonic.fun.composerc # noqa: F821 - assert my_sum(range(1, 5)) == 10 - assert my_prod(range(1, 5)) == 24 - assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) + test[my_sum(range(1, 5)) == 10] + test[my_prod(range(1, 5)) == 24] + test[tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6)] - assert tuple(scanl(add, 0, (1, 2, 3))) == (0, 1, 3, 6) # noqa: F821 - assert tuple(scanr(add, 0, (1, 2, 3))) == (0, 3, 5, 6) # NOTE output ordering different from Haskell # noqa: F821 + test[tuple(scanl(add, 0, (1, 2, 3))) == (0, 1, 3, 6)] # noqa: F821 + test[tuple(scanr(add, 0, (1, 2, 3))) == (0, 3, 5, 6)] # NOTE output ordering different from Haskell # noqa: F821 - # let-in - x = let[(a, 21) in 2 * a] # noqa: F821 - assert x == 42 + with testset("let constructs"): + # let-in + x = let[(a, 21) in 2 * a] # noqa: F821 + test[x == 42] - x = let[((a, 21), # noqa: F821 - (b, 17)) in # noqa: F821 - 2 * a + b] # noqa: F821 - assert x == 59 + x = let[((a, 21), # noqa: F821 + (b, 17)) in # noqa: F821 + 2 * a + b] # noqa: F821 + test[x == 59] - # let-where - x = let[2 * a, where(a, 21)] # noqa: F821 - assert x == 42 + # let-where + x = let[2 * a, where(a, 21)] # noqa: F821 + test[x == 42] - x = let[2 * a + b, # noqa: F821 - where((a, 21), # noqa: F821 - (b, 17))] # noqa: F821 - assert x == 59 + x = let[2 * a + b, # noqa: F821 + where((a, 21), # noqa: F821 + (b, 17))] # noqa: F821 + test[x == 59] # nondeterministic evaluation (essentially do-notation in the List monad) # # pythagorean triples - pt = forall[z << range(1, 21), # hypotenuse # noqa: F821 - x << range(1, z + 1), # shorter leg # noqa: F821 - y << range(x, z + 1), # longer leg # noqa: F821 - insist(x * x + y * y == z * z), # see also deny() # noqa: F821 - (x, y, z)] # noqa: F821 - assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), - (8, 15, 17), (9, 12, 15), (12, 16, 20)) - - # functional update for sequences - # - tup1 = (1, 2, 3, 4, 5) - tup2 = fup(tup1)[2:] << (10, 20, 30) # fup(sequence)[idx_or_slice] << sequence_of_values # noqa: F821 - assert tup2 == (1, 2, 10, 20, 30) - assert tup1 == (1, 2, 3, 4, 5) - - # immutable dict, with functional update - # - d1 = frozendict(foo='bar', bar='tavern') # noqa: F821 - d2 = frozendict(d1, bar='pub') # noqa: F821 - assert tuple(sorted(d1.items())) == (('bar', 'tavern'), ('foo', 'bar')) - assert tuple(sorted(d2.items())) == (('bar', 'pub'), ('foo', 'bar')) + with testset("nondeterministic evaluation"): + pt = forall[z << range(1, 21), # hypotenuse # noqa: F821 + x << range(1, z + 1), # shorter leg # noqa: F821 + y << range(x, z + 1), # longer leg # noqa: F821 + insist(x * x + y * y == z * z), # see also deny() # noqa: F821 + (x, y, z)] # noqa: F821 + test[tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), + (8, 15, 17), (9, 12, 15), (12, 16, 20))] + + with testset("functional update"): + # functional update for sequences + tup1 = (1, 2, 3, 4, 5) + tup2 = fup(tup1)[2:] << (10, 20, 30) # fup(sequence)[idx_or_slice] << sequence_of_values # noqa: F821 + test[tup2 == (1, 2, 10, 20, 30)] + test[tup1 == (1, 2, 3, 4, 5)] + + # immutable dict, with functional update + d1 = frozendict(foo='bar', bar='tavern') # noqa: F821 + d2 = frozendict(d1, bar='pub') # noqa: F821 + test[tuple(sorted(d1.items())) == (('bar', 'tavern'), ('foo', 'bar'))] + test[tuple(sorted(d2.items())) == (('bar', 'pub'), ('foo', 'bar'))] # s = mathematical Sequence (const, arithmetic, geometric, power) - # - assert last(take(10000, s(1, ...))) == 1 # noqa: F821 - assert last(take(5, s(0, 1, ...))) == 4 # noqa: F821 - assert last(take(5, s(1, 2, 4, ...))) == (1 * 2 * 2 * 2 * 2) # 16 # noqa: F821 - assert last(take(5, s(2, 4, 16, ...))) == (((((2)**2)**2)**2)**2) # 65536 # noqa: F821 + with testset("mathematical sequences with s()"): + test[last(take(10000, s(1, ...))) == 1] # noqa: F821 + test[last(take(5, s(0, 1, ...))) == 4] # noqa: F821 + test[last(take(5, s(1, 2, 4, ...))) == (1 * 2 * 2 * 2 * 2)] # 16 # noqa: F821 + test[last(take(5, s(2, 4, 16, ...))) == (((((2)**2)**2)**2)**2)] # 65536 # noqa: F821 - # s() takes care to avoid roundoff - assert last(take(1001, s(0, 0.001, ...))) == 1 # noqa: F821 + # s() takes care to avoid roundoff + test[last(take(1001, s(0, 0.001, ...))) == 1] # noqa: F821 - # iterables returned by s() support infix math - # (to add infix math support to some other iterable, m(iterable)) - c = s(1, 3, ...) + s(2, 4, ...) # noqa: F821 - assert tuple(take(5, c)) == (3, 7, 11, 15, 19) # noqa: F821 - assert tuple(take(5, c)) == (23, 27, 31, 35, 39) # consumed! # noqa: F821 + # iterables returned by s() support infix math + # (to add infix math support to some other iterable, m(iterable)) + c = s(1, 3, ...) + s(2, 4, ...) # noqa: F821 + test[tuple(take(5, c)) == (3, 7, 11, 15, 19)] # noqa: F821 + test[tuple(take(5, c)) == (23, 27, 31, 35, 39)] # consumed! # noqa: F821 # imemoize = memoize Iterable (makes a gfunc, drops math support) # gmathify returns a new gfunc that adds infix math support @@ -151,55 +155,58 @@ def f(a, b): # # see also gmemoize, fimemoize in unpythonic # - mi = lambda x: gmathify(imemoize(x)) # noqa: F821 - a = mi(s(1, 3, ...)) # noqa: F821 - b = mi(s(2, 4, ...)) # noqa: F821 - c = lambda: a() + b() - assert tuple(take(5, c())) == (3, 7, 11, 15, 19) # noqa: F821 - assert tuple(take(5, c())) == (3, 7, 11, 15, 19) # now it's a new instance; no recomputation # noqa: F821 + with testset("mathematical sequences utilities"): + mi = lambda x: gmathify(imemoize(x)) # noqa: F821 + a = mi(s(1, 3, ...)) # noqa: F821 + b = mi(s(2, 4, ...)) # noqa: F821 + c = lambda: a() + b() + test[tuple(take(5, c())) == (3, 7, 11, 15, 19)] # noqa: F821 + test[tuple(take(5, c())) == (3, 7, 11, 15, 19)] # now it's a new instance; no recomputation # noqa: F821 - factorials = mi(scanl(mul, 1, s(1, 2, ...))) # 0!, 1!, 2!, ... # noqa: F821 - assert last(take(6, factorials())) == 120 # noqa: F821 - assert first(drop(5, factorials())) == 120 # noqa: F821 + factorials = mi(scanl(mul, 1, s(1, 2, ...))) # 0!, 1!, 2!, ... # noqa: F821 + test[last(take(6, factorials())) == 120] # noqa: F821 + test[first(drop(5, factorials())) == 120] # noqa: F821 - squares = s(1, 2, ...)**2 # noqa: F821 - assert last(take(10, squares)) == 100 # noqa: F821 + squares = s(1, 2, ...)**2 # noqa: F821 + test[last(take(10, squares)) == 100] # noqa: F821 - harmonic = 1 / s(1, 2, ...) # noqa: F821 - assert last(take(10, harmonic)) == 1 / 10 # noqa: F821 + harmonic = 1 / s(1, 2, ...) # noqa: F821 + test[last(take(10, harmonic)) == 1 / 10] # noqa: F821 # unpythonic's continuations are supported - with continuations: - k = None # kontinuation - def setk(*args, cc): - nonlocal k - k = cc # current continuation, i.e. where to go after setk() finishes - return args # tuple means multiple-return-values - def doit(): - lst = ['the call returned'] - *more, = call_cc[setk('A')] - return lst + list(more) - assert doit() == ['the call returned', 'A'] - # We can now send stuff into k, as long as it conforms to the - # signature of the assignment targets of the "call_cc". - assert k('again') == ['the call returned', 'again'] - assert k('thrice', '!') == ['the call returned', 'thrice', '!'] + with testset("integration with continuations"): + with continuations: + k = None # kontinuation + def setk(*args, cc): + nonlocal k + k = cc # current continuation, i.e. where to go after setk() finishes + return args # tuple means multiple-return-values + def doit(): + lst = ['the call returned'] + *more, = call_cc[setk('A')] + return lst + list(more) + test[doit() == ['the call returned', 'A']] + # We can now send stuff into k, as long as it conforms to the + # signature of the assignment targets of the "call_cc". + test[k('again') == ['the call returned', 'again']] + test[k('thrice', '!') == ['the call returned', 'thrice', '!']] # as is unpythonic's tco - with tco: - def fact(n): - def f(k, acc): - if k == 1: - return acc - return f(k - 1, k * acc) - return f(n, 1) # TODO: doesn't work as f(n, acc=1) due to curry's kwarg handling - assert fact(4) == 24 - print("Performance...") - with timer() as tictoc: - fact(5000) # no crash, but Pytkell is a bit slow - print(" Time taken for factorial of 5000: {:g}s".format(tictoc.dt)) - - print("All tests PASSED") + with testset("integration with tco"): + with tco: + def fact(n): + def f(k, acc): + if k == 1: + return acc + return f(k - 1, k * acc) + return f(n, 1) # TODO: doesn't work as f(n, acc=1) due to curry's kwarg handling + test[fact(4) == 24] + + print("Performance...") + with timer() as tictoc: + fact(5000) # no crash, but Pytkell is a bit slow + print(" Time taken for factorial of 5000: {:g}s".format(tictoc.dt)) if __name__ == '__main__': - runtests() + with session(__file__): + runtests() From f5e1a78bccfaf1ce361780eefe910cdd3259f940 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 00:17:01 +0300 Subject: [PATCH 060/832] pytkell error disappeared, remove TODO comment. Now that we use `unpythonic.test.fixtures` for testing dialects, too, if the error ever comes back, the test framework will show the incorrect return value so I can actually debug this. --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index e0342bdc..d7444de4 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -83,8 +83,6 @@ # Cleanups can be done in a future release. # TODO: debugging: -# TODO: investigate error in pytkell test when run by the runner (works fine when run as a single module) -# # TODO: The HasThon test (grep for it), when putting the macros in the wrong order on purpose, # TODO: confuses the call site filename detector of the test framework. Investigate. From bbf3a94aaab29bba39a28be17a690c1cf4033423 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:17:01 +0300 Subject: [PATCH 061/832] document dialects Initial version of documentation from Pydialect; then updated. --- doc/dialects.md | 46 ++ doc/dialects/lis.png | Bin 0 -> 36164 bytes doc/dialects/lis.svg | 903 +++++++++++++++++++++ doc/dialects/lispython.md | 184 +++++ doc/dialects/listhell.md | 58 ++ doc/dialects/pytkell.md | 95 +++ unpythonic/dialects/listhell.py | 2 +- unpythonic/dialects/tests/test_listhell.py | 4 +- unpythonic/syntax/tests/test_lazify.py | 1 + unpythonic/syntax/tests/test_prefix.py | 5 +- 10 files changed, 1293 insertions(+), 5 deletions(-) create mode 100644 doc/dialects.md create mode 100644 doc/dialects/lis.png create mode 100644 doc/dialects/lis.svg create mode 100644 doc/dialects/lispython.md create mode 100644 doc/dialects/listhell.md create mode 100644 doc/dialects/pytkell.md diff --git a/doc/dialects.md b/doc/dialects.md new file mode 100644 index 00000000..ae810a81 --- /dev/null +++ b/doc/dialects.md @@ -0,0 +1,46 @@ +# Python dialect examples in ``unpythonic.dialects`` + +What if Python had automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas? Look no further: + +```python +from unpythonic.dialects import dialects, Lispython # noqa: F401 + +def factorial(n): + def f(k, acc): + if k == 1: + return acc + f(k - 1, k * acc) + f(n, acc=1) +assert factorial(4) == 24 +factorial(5000) # no crash + +# - brackets denote a multiple-expression lambda body +# (if you want to have one expression that is a literal list, +# double the brackets: `lambda x: [[5 * x]]`) +# - local[name << value] makes an expression-local variable +lam = lambda x: [local[y << 2 * x], + y + 1] +assert lam(10) == 21 + +t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), + (oddp, lambda x:(x != 0) and evenp(x - 1))) in + [local[x << evenp(100)], + (x, evenp.__name__, oddp.__name__)]] +assert t == (True, "evenp", "oddp") +``` + +The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). +It provides the plumbing that allows to create, in Python, dialects that compile into Python +at import time. It is geared toward creating languages that extend Python +and look almost like Python, but extend or modify its syntax and/or semantics. +Hence *dialects*. + +As examples of what can be done with a dialects system together with a kitchen-sink language extension macro package such as `unpythonic`, we currently provide the following dialects: + + - [**Lispython**: Python with tail-call optimization (TCO), implicit return, multi-expression lambdas](dialects/lispython.md) + - [**Pytkell**: Python with automatic currying and lazy functions](dialects/pytkell.md) + - [**Listhell**: Python with prefix syntax and automatic currying](dialects/listhell.md) + +All three dialects support `unpythonic`'s ``continuations`` block macro, to add ``call/cc`` to the language; but it is not enabled automatically. + +Mostly, these dialects are intended as a cross between teaching material and a (fully functional!) practical joke, but Lispython may occasionally come in handy. diff --git a/doc/dialects/lis.png b/doc/dialects/lis.png new file mode 100644 index 0000000000000000000000000000000000000000..1e722eadd279e55759bfef2b96367ff1bb905369 GIT binary patch literal 36164 zcmXt91yCDZ*T&u5X^XoQcZx%CEAH;@UZ8k!_aepJA-FpPcPZ}f|Gxjs$4n*(*<|nD zd+s@poQ+UYkU~WwLV|#RK$VdeR{@^0ARr()5a57U%YoY7z!QwiHyJeq;LjJq^cV0R z(NS8<1pc0nM0PzPm@J~Wli65@24(6^NM$Tps9v&XdR`xb7CPt2C%nr^LndbsT z5D=drWW+_)JhM)-J<@Q%vx5QaiGBYd+J90VWPO!QLJ3B~$Hg58jIe3ZuQx_p%wmJY zQWM93m0-t$r92?v9vTr`-z4v2=I0k28$V}syf}Cfc=@$nqL_Q+->#5d#;5MM--onj z$Z>2TA(4)La@Y1ngpJ@fR9SfenlJm@#1J8`wWF`XSq+-3=Et}kqq%CEWTBH4N5POO z>z;&^7T2$W$1l^{&gj?17|ah4qoN54B|cC|GT;R4R{vW=vD8?*{x&yjENoHV79+FZDuV!n!kMFna9ge+mu_8=*0qoihd>RSftNeJ?6czyJyM30aW$OP z9_NoWFQ4RPgU+J)s$^^K^k|yE^VAn9aR|Ojjh6zA@~bsg^ItYo3sm@iXio=CbPk=* zKe6zM&p0yWGlRH*BXy4~?brz)N{ZE77$5If$`!Oxhgfb(=)5eN>78eSe&>p9ELV=cJ*=J1ST~Y|h&~7zwyX)45@^mQd1q>r zKOIg+>~h2jc(xaNH&UX&`GxYWH2I;6No{VAjt!X~}V1V#Seh zBIIpR*0l*nCPOP7K(PuAMaR(+1vCbMh%dX7`KVmz5EPs`fR`n2zZR91~D z{0h1EZTaRgNr7%GJ_|MP|z`DzSPO8Dy z_?=g?CeJ5Q=x9e!$~e;}ZfWwxQIl{60XNVd-~DGP#;&vVs~^UsQChyVWc`fd%#eKgwkP`_FV2Np*{R9Pf}vO~Xh?F0!aia@}+;AVXJ z_99#1LB2>~017qG^AlVy(-bTY#L(4|z{X{OzRae4Adhd8IO-O}brAN!%L9}fD%{&b z_<1gY--RBxx^;Zv%&-1|!Mr)EkS490pMKPtiA_wRXrd>U>36T9|s^P|Z zK*dcBafH70UN4|R?waPbQu|0)V`ZrhJ4Ey&eBz%c!ehOfYCUb_gH2-w4Hotehsh8Mr|#Fh z)TII~3Y9XNzxQX(t-PL(1OfKz1J}tXox<<8?bJoySgPlpq1$3FpAnLMn1Ue?ublQ1 z&V!_iWpn+W&e(vEAaT?@4%?1CQ&}&4^rudA+7!M3Rc`OeaO)@dOubxb=J*`xWpG+` z_o09PBlem2**bh6Lv9Yy;#%rbPbXht7c^4{W^#i03`b-~2i$vK!1gmG;x=wc7e7Gy%_Phq&U6S0UMbI+; zlNg)SF1yw23@T(mrBU_o`uSFnl!4)H@tMHOo}Vtiuc7#}p!+LM^Tr*^V(o58oOh@5 zn`Vhnh2Q0dPUv;gol<;a=1`$x$6)h~|Mog}j%a7gpw>Fi(mWUROOBbVsx%TLUyQ5P1Sis9}|3~Q+l`8!_) ze~eRGFO*LHyee!M)3p2LxBn#NnrKhVpyrOm7&c;R`uMwxl7_}{Q*0`K#`17J-x!O~ zx7;I6s@IgGRAG)OgL9{F$~;0sq6xaYN)|=XR9Chad?`C?^Wn9>84R}mTbD^9;2Mp& zxG9sWVE1_Xh940di`0%T3XwP>;+mA4c1SyJj_uuvR;dvsNny-E)N)rmV_c3GQ@({S z(#|!}+NOzr9Wf~Py(yhnk`NHtVdSluD7&*^2E-kk8~Q9>;0o( z;?^=v8ha9ctjhZxJlqgbMPJ?M0!5o|tLcuze4GMH$@3 z?E>eAz+lx6jkQv|9C&@E znHjMjdc6${Jx=);{5DV$2f#_)smsB@(7@&T_`{>|^TVmI=btryU1qm2DPTWbdJEW5 z2sz^ZcPjhE;?}A!_>&p!7>UdMZiVv{v^hD?T-+F8Mb zmM+g8z16z8@-}n6L6V(3y5sA5e8koFy%S@vuM4^VETiACA+qQQpdQDzH8zpE?(#2D za^HB$!LvnXQG-TOsyPB~Df{IMH&ZY6s)8W*H$3;#m8p=24I-5QWewz7wi(T}-cM@Z zTOTyKa(K}tlpWgzaq#qhs8RP1UEiN5UeDT)r>=&tk`gdt6JyI88+SmLrUjG2F(Yo9 zYT1MTY!4)_)OlVtqI(_J`h;NZB~Dp5jmZ~A#6&PRljmmAH1+mmX(Cm%yJI}Ta&g%{ zBL^0I;ZJZCHRg0!`TcxnUGKa9oag6BDRcMerL*|~vyFA?E@gaUj@s8|dDb?;VXGA0 zu)?V}RbpXUZ#_XhsK?D=NuK$>Hkne{F8%$NSatPuwEwc>@oj6MiFtj5QAm~f)vP+z z5KvrSR|KA_eIA_$AE@~BwMu1p0-3W{(#FkIQ)N+jDRq&08|9>bxd4eZ)NSTq4s>v@ zQ-$pP-fU)L2`)f%o3$>>$>sF;Y(n#Wdp^D1C|#xsCQB$$Va^>|tLE?_&}H%VVj!b2 zQ|^6p-}xne*p!{!a|`_1WLs&ynLwFO8`KWJ+F%{|2|>N9`c4I-%5Ka&rcWi~o?PJD z1Q9qQ0!>K$z1E*$y}e0O@L_N@m+dYYJ?-n8-Fexc{3qu;No&qxze}uL`}AHGP}ME} z;={2$36E=Z$18GF*q6)?vV4A;ShI~dIklI2*vc{{k_?k(e(N1%yh?U8-fc$n8*ny% z6MLA;1Uj<+Zkuw*s9n9ygFjz`9j4=JGPm&diNn9I0q0UEA;D=^qO3B^-&G!GB4LjqFXYC!DW10!4mY);m0!PeOIQ{1fE!u*9tem-gk` zv&O(=Dd59~N|ZgMvA%1ccb-rKfF$uWt0Q4|B7*~^rj?h38T1igc1)orr=ZqZBBipp z+vZ`#qBM{>T%O^Gs>iJ8nwhdK!4E)m?enKEnWQCS5-P^NBel7y6rF289y! zl17AXW?IFi!?ANmNIVCwZI;t@EMLt(r1kwF(ZW^L;mZ)5Z>~os7`n|}ey19g(tGK9 znj-;2*t+*2W8qCnND%jK19eUuU6cDX?uldVn5bV6+ip9mS5Jn~mKO!IfG9}RIJeut z)tak>M}}iN(y=@E0gs1>WWKks8~$gAd<`~1RN{3bb>IC(KNF@e<1E{jF)UY`zjp&O z)GsNu?btFr_Ry%AD9xOjS#3VqXYQhVyJ6o9Uoy9Ootv3TZkUEE5nnCjmwZRueD$=r z3HdvVKvBaz)_^}w3Tb*G*e>JV+YGF3`@nqW)TT>X+FR2|Pn9)dNU&0(nObw(tdzHC zg$V;0-WR%h_V4^9cjtLVj6moQO98DhyWVRGaMtk(l2dn@$epjeNvJZ{-klv+lN=C4 zTotO$x+%D)^gsFisIzr0e4`2L11LdxO5ieswu`#oW}&aa1%bN&S$hIfq}OeP=E(Ty zXZlguc){QEW`0H5mDGn_aGczlK&hR1{-($WPwGjfocwccIFj8G02!E|aD@_zB5(E~ z#jJnX=pf024uP8sAIO@t34HK60K|D)pN6kP|ISp@sXI-y3G@2y!LZFJLo%2*`t0F9 z-fMk4&Zz+7PS@h~-w2#t>T)3uuwBA(T-$dx(fmAj7!Q54R*M=Zd_#>8dE8LNrJfOe zOrASND$UcWKg9XwLsvkHrrJphRVv$iH*L+?Hu}>eHCz)P2r%AJ8XWN|csza$IWq}b9Z0_HE@z3Z1fJC8>Y z@$SOU6(JVr+Jz$DY@M3Em~vsW<6Wwg7;O%?fK%?sn0T|QSsp7jo)X9K>2s^ zWdDUcb-GoL7;6U#v4np68u_nkJ9VQ?ah>meSi-KmR9=6Z#jK~Z^NpwXF+b!14y^jf zQiTk380vXnP>7VL<26&+Vlq{;WT7%x-fX#fGoLX`*M-x)y&esti-jFVjd1(cyDtrx z70q;EtFvb&+qGAGbYV}Qky^t!_(#go!0qNz`WI_2-lmH+srU&6poZ0GLp>OlV)CK>p?pGt+(I7 zn@9IBvZ{^fNKl$hTgV>Pt8MAn1{yQJ3d;}{hxeU}biw7Lu8JFPTMXwdXK+~9zRX{h zamM5iueVD!e{1k_cI$qL3@$V71fbbUqkRxoQx7;iaU=kK@8mU{qbNVnD|vA z#!T}M1omJY)K)A0rfTCm{xVdNE>K4g=h~5T{e`6VHQ{DhUjK_ZMGA5nKgn`3=X}7;4 z&pe!O!mVSZ_}P*_cy^yRcVxL)OqDP+Rwei%H~Cj$^}fB0AMvV|gwHt(V96*v7p`sU z5j;;QAkb?yl0z-TW=+-Ou~e7`x5l2caW#TJ=1E z_;a~@IqGrAOqU&UGi@aIUtJYCHQ4PcyiZ1kATssoMi#|kbm8|=JM6(n(#50sh+TJ| z{~!?rhXa;0Pospqs1YWOVs4RBh_`1aJCVF)+{kXc?DNTd6(=#*2QF($-Qi*hF@Q|O zr6kxqHdr7Jz^)3}3S+>2P6vtjB_?MieI-II)rtz1FZWO&v0*1f^(fD&fcc`pQ#j+f z2pk6M$BR|M{-ePZ7WH^BU%tGUY6wOYfr!1!7yO)HRTwXeQM!WHzV3tepRjbdN7+$_ zuN>-V%`zp$V-NsncjI?#LI>Ek-VE z=1bI*KaE`%t2e$o5D3DIoKzyZqP!O2O8r5^XOhXm+^r(Vi=S7u{_%Q34`(p9dhvtx z2#z&iTZL$$8sH-;6ph;5UYtj6ul-im;M*q`&H^|sdf^XUHUYBDB&6wLiLXEoghwM~ zaufQEFd~Bp6gt^#ZB9D+x>{dwSRj|R_}NM;lYVu7l_0#@U_Kd`*cP{%%TA}#yo#6; zA*6>@=B@zg%k0l~s!Dg%R=2TYdF~GAO;&1z|{6nzn|o5>bwQ za5u}oms?{=m(0yJs_HB_tl&8+*vT@HC1&)jq;hgN>H>tXJkiWcK*=8^fZ!M9ws`&W z?Qo~Zx~^GNXtE8LO1lOs(SqU)e^v23Gyu`zCAj!GRmxJO89^t(;!wG~nWD}+UcFl5 zOKQ6E))+877XzGDOM&Y?=+BMHV++frMtYN3au%0?`B&Y04Q-@O^+w3Q7$YYyE&T3*tDE1Sig$a+C>qG#bJLhLfdP zvJ>5+W9K6z{KA~H<`NWg>UHB2!zg#x@)W>=4wL`=u~tX1=L5`0xb<N8{J`XSOvTzh@tthz%c_!@|kUwwq z@M#G%x$TCJz8;m?*BG|fp9eg8UBKr~q8MfkQNZd65THDJ8n#%68+}n3^|L8iU3cn@ zSqvjF=69KdLL(QHp}?hSAQDLE?}84uS}peK=H=W0hDiLeHfgt8WpP!POd{kTj4qUZ zv?y6JSEj%QP}Guu>Kq6ljU;4eD*6Y3&^46zAk4ZdnAMuynf|LqFCo>kc-U|uv2_fO zCPRwKo;mZRs|`E5)zDup{ti@wE0jqSN4g`hEoj>rCik5~BG)^$62mfRazEl8ik&Dp z%qR?P-TA#c1A2)=M5jU$9cDoH_YDDn#P<{=7#98ukTK_l*=(f^czNkQQ`Xuv^LxIX zO$kth-Ba<65+x)F`cadLVn~5Db#e0zurBmz;|;EW@7GVIn_#q5RF37 z>cvz>vJQO~CL(V7Wz>{3jPMEb60234Z-mr+@OKPP9>(8DfEOkY0C&jxgkZ^$fprtH zx+@xQ=IwD~M`gKM#tX>Slnngg{Xdbz&J2&GWbaIY8GwqFOTgDkfzFwj*p<3IOsu|s zm-^bN_PwLTP)M7>jN)y{&}#DGai z#FEvI?mMCs{s^AcXQGQIwZSisrQ=(PoN+XY*}HFfEAWFA-RjiLMQxchWyf;aIMe)L z+Gk$B>)blRLQBca6?SOJP{{|o>7awh%Nz6)IyZ@dIGTtUd*ToidVi`bi(!My2nFB| zZc#YD%uUVfFS;?okTwk&3HS((hnG${{@ZrN0k_n5C&hntA!jsfA}Bx& zI^(pdF4Z$QT~7oX5r|PQ$i8YZE?N4j|ceAUOAWD;j#Wl?V>pg#Qs;IHWLB zV`i|-Fve6WUiLOB+rpznz3#C%@mZ6^=b`UNzhtbaeK+N#4@pl}Ard`f2M~Q9i+q%D zLW$_WRtwU8%^*n%xCu4@K1M`UrY*~0{B-tfv!1r*IGpl4h)~Wb)^%1;gnNmuph)Faf5? z_j?+ELE0Z~3CU^+ZW}9&UNI-%aRBNW7O}W&L_r|$PkF9X=z#>nJC?-N8&A~0Yjrzy z?AIhs{SpqmkEV>bC33`gB!WEoN($Dyn5q=@iDs1tSwIc?#)GbrpH$2MgAm zwwqoAjm|38_nd*5Lvb9;I<2va!nwO`H%(51vlp``$xeKFknW?(eenM&sF|b+2bPBJ zJRod(g_c2ulKQ))ldzwR^cydhZP=`_L<;ebj#4Z@TLFh^LFx-O->wE0BwhyOFza z%m?MiM2{>2u{5hO{}n@v{((|ULe1t1Z5 z&3g9PI4HKamkzsIApmsqaYK-=9eZc@n@Sj*(ZdI+VR+pd-7 zFh&+wbBT5$Z7bvmh~=IK(Xo>~9Y7Q8zGgqxG1j{rC`@MYis|Z-oUJq>U&7EP5GzjB zIuCiTR&swm3JQ24O{9#d)M-x0&X#Yq23xMTxpm8Fnj$KuOueZ;>jH$HoxL9F9Gme% zrv$B%jj9(Ec?J)&2*%QBECeDcshOGKN6u`)0K?B}bKe6p&I-`x3!}S7GHM^W zu=~%;Q@IP=@H}|xItF;$ng7S+t1ZM~p%q>M!XKJi7{7OR3$pe2;P^H}jGoF+vjkJ^ zPuKF@rq@udXRrJJ*8fRivqTCG)HqQ5V>XsDlP3zj(qPrM)e~~G(#QZV*R%lMz9&#h zb3_JwyxaX7PZx_J;yhey{Tc#~UhMaB|EH+vYEDrYwzJB0UQXp3RC@aR5y_{PpEk3l zWRitFH%A99y4H5V`{J!uYtkW~5Uz7|JWdpI5%*#z$V$h+hP*)^xs-y z^{csCH`UjfH?TJGpUygR1wP@Rkj`ro$G&s8-1~Bi00b#mVciTQRJ}OXdOHr$i|dfY zJgDQ)De>|C#_K(g|A`I(Dx2S<*r+dT&1s1Adm3y0AG<3v>(9beLvnX--O%W)_G>)s z?CcPm1;v`x`uzg~Y`%|iDWhJ+pf?+r z{f2^nko>}lYetu*8>$!I#Kf2dungMWELj~Kt_LpBNrSDojN+n{@4gLE<{D@e?`#q4 zFK8+i<#wB0*+XPL;}`q#xkA~f#9TE_yOLe6*K+3LX+5XQLVzaIiL-qAhwE&7r-j{mb&u|oD}2B(cL>SWyZ z!bsfZIgLT`Z)RS$co)JTRckLyJ;C-yZs_s%a~gGzaIsKFjq>!W?3K1oeTM~Z7yL42 zdGNPXfW3!Im^*}3@3;*kWCDj28C0K>ep3j)U4I=&U>s}jsN<&pF(}Jf@uL&6S`*JM z^I713y#R%U73Tm_Ra-5Me+P`4KgE=(7;J2xW7l4Nizm|L%+0B_w6p~7mrPpZB)sRV z^fDo#;eN-*i+X$Wg}|YH;p9w~BKJ#jIhb@hKPB{kcjJA64Sjtaa}c<`9is|3BGh8k zZwrpTbul$HWi=bc5&q(FvJmrcA`_@I5*iwKiPZAna3shAUY(qsorfnUO+R`le4DpY zZ*ZM_vG{p;<8NI#dtohff!zojxan_DUMf>m7l9nb24t4R8Oz_Xq{5Y(?H7gsbIR1M z%%G#B8sxv?y%m#e`H8)5?_tq6o@fm)UxSp2B&w0j)xUl=iafll{x5bh@HL$FYTSp{ zN^U)l$BcAfmQy_pn4l9WRHL3LP<(1)=+Lgw?qH1n;*u?wF7xKOr&fwntM1a7yFxbK zz+csp%QljV)rL^oa@-L8xUqfur}C7n;h?^u;nD9jEZnJrKKGAz*}1uKL%VcOV`F1| zCwFx~#E_uDGq=Yaxde3QEq*oc-@?LHw7xS{_-k5|G*%nfe5=l_2I!!x?<(9grTaL9 z95vizaV53-C955#M;Q>3KAm-COw}G!3Pio%4GEaHGe+u+Q5t{F0J*Z3KPz&;R)J^< zWHB6t0_?{+^%G2(ZO=K$JN0UYDidwI@VJ~jcftss_viQ)K{7n%YYD%kn5~J)$$#k< zIXy?NJB~b_&qpOAV>juPk&%(V1>G0yRz^$np*U7^Tb?7M8ehGVeoQQdM;OX5#9h5f9<_-jl)7(i zjezL`3~Ty=VX>S`ZT0tDgOeWsLf6nr>K(7D%hkjJz+BD`=QA?oq-Ka?QiQ-Y&i_q5 zp8uJaXFhNobLhGv4-_-?YO?$buGKm_eET@c^a*O6w`Z#|llddpGP009DFvnSFwhYN z6>mh4pEujV3oI=e4Up}j@+4!3*4!5L`UeNE_frjKTbxWXIIMaX4FXUfZ>y86BXIW2 zYkf9FtBx_#)P(>g67{&K)_zW7XPj@`Jom2HYOeTu77u1fPd744hX@e?*If*8q#3hj}r#FBK+M3&RCB^FyQLclZKHBG;L+Q(F=UR3VXPMF(`cmL=x4J}g~ z+jlO%XLFqf^TkO1zCRT4=z`qER9_Jh%bG{jMWVR!IzX zNV75m_&l=K7!1cz(4wuJ)Zp!A^S22xrC@t{d-o6cHm(i`;#G&XAUpnt8kbdS$=F-) zsrO#I)PF16dd6i?l(}P7P2l%@+rubpyVQXS1d94P*rn+9XgL`*v$-H%4~6g)dEWP^ zC^e1Gjb1GLd1qzKzBw1wuKyP%Z9=A&3#ibnez3uAKgyK(j8mpuJp`NPtpq&Fdp1Aa zAR`#qCl#B+QDjkC=?wM=x;x!aw&6-APWxV*SiWeB##5bQM_1`i&z#CngrKE~l??EG{|Fy2-Uh-q-vH$A`|4!a`l(VwPTwBwqs0u1rRxb2SV34dHb%*dbG;>x zON3n-`R)EIlS5ZNYWWQD%`QK!7FJe^Uo)1bQ#tF=o!f9JRX@f^u?Lg0uwjCB2>(c0 zyD8+{KIrS(?kYzW$9(6@rtYz2&spsEgT11$j`6RWJYF*sCJ?I~dcvZ*X>?HBcNRUjQx2vT` z{{8F1S^M8aje1Ivt+E`wG|$~o5`S<4mR75~pw9!`Al=}=z)Z;$xL+c@=Oo`LgfF^t zLAXt`d}b%PWB&W!$r1L!CBvLBV-D*dnyiHKOq3(-PDAH!6!5nvgwv9Cg|2VjjS&$| z-q+X2XD3{?iGhK1{rm1F`@wxfrtE;M`ejUZ_=20U7r3rQAR<%O&sp_t2c8ZV^bmqu z^b`J&ovp&!`yz$`T}6h?{0$=Jy0+0eSD=e&?qq%l?)$^`U0=Y}h$!t}!gNShIug<>|4HTl|c0;&x z9&A)iJuc5JHfvm;tY^Qn%gD0%`-5^^bYHYv2+#1P-`&rRpe;*ZY zbC5_Y#b$*x_DQGdybTWF@pDfThWo{W=#$qK(nPCEdhuzN?x+U3bGsiQS5)rXl$^$w z$O5U9&Y*~hkWY-gBXzHmlIHh-neZSv*KFWdw)SiNF7w@a>FmBQxchg?1aY`;9BKQ# z2^XQ_>M5F>vvctA9BMI$RLD7tkG`>W?-PqmVN068wl=jQf}g`cZot{z__8b99jI*0 zU86o?h`c3xh@93WI+dkssPXXuTo&I8B$l^HI;1k?jsauMo zT@S&6eEu;hu~b5L(VbSGwR4In0|+>LfW#XY1jwOZy!P!x?&godY9$VwFT{@;rPj}x ziwy8?a-mV=0q-#UZDyzd{E}` z(SX*CZ_1YWwAas-tL}gG53W3ck(mr`p4_$w3Pa}Vz2fVMNa*TFt&>en#0k8Y9URss zmZ2)-@J01)&qAI%j>VGmIj;c?I`>fpX)5LBZs2Aj#*U-Q8IeS3w)s;<+#VoZ94=E8 zAcjJ1A#+Ttd~M|8Y8NzAwDkfNX*+)@prs8QF`ULH6L!*W%>MJ`4iLb8t#6l}JGQ*9 zJTH=|`(#Nac-{|q>;sEt!KYg_jRa2lCPCqUFjB{Ea-C$MXDxgB9wNn-`Ta<;v&#u+ z=IYgEB@uMjg^JDiKF9}nUeJRjGrvSj5~1Paly#`W5n!qx$e0y!gyi1Zju{T5zg5rW zgRN_|D@xGdwz!o+$i@WlxHq^qH?KYxW2!0gxXtx zmn5UKnyRFK`R^YSKaZaVQ#Tf691bQOJ|J(mMXCzUvJ%|dE_WeUDvZq(zJHCN^cqvg z1o&pBNXO!dpyTzQHt(VozHb;XfH1uaw^wmWVzP{b1+??DpZko$_Lhqn#z$Z%r3<2l zqLti`S9e@RJY3FT`_tT?@*mRGEpxZ)ookz|f8X!1_fS&$@dqA_AreccBmED@JQf1f z-rm2&?8G4eAuz(rjbl|vOYx++j`5to9$-T%Q^bc2QCwZlv5a_mdF4=ns?n5MMiZTD zYuS)607Ltpl;26)O@4SYjE(C=Q#`aU)p54w^IY%&Qsnw>l_YFz0P z?4mfJ9D#i;O)Jb!hfPd$$Ju>%9aR;mL&j`Z1`~gxt2}t5mW5yjxVGUuh#UxdB8o{t3wXG z{^RERy#b*gqC_D}pis8OkiVN~zTA$L=nfS{vM;``$n}Qcd~j zRE9bAq$dPE4tG4OK5uU+V$Px=P^x@pID%eXH~2F*H+i;(4g8w^h>iMc0~X`zKK|SR zFA3U#Pj__dO@Mii@!e}P&ZKL$k6f9 zi|Tuhd>1*-qrk*u=>l$e?P|7ncXn3R779(KxPm?;Up5!H$hG&KwK^o&xhETFDd7yT z>bM7dS{bP7pWn^rN;^7 zo-xwbfeel>F||@&`1Q_vGtZw(I_xYASqt{!t7vEKR@$;57dIFIHTxTI1VmB21UV~F z%xc%mHa-ZQC++8)B`1^+i%Qqn{sNp!_}_`)2t+4o{c1EG7oe?ky`H7(L>Xrv-B8r< zl!mkT?>EOW|cHN6vsvIWjTv041El z+o+ndZ-az{bW;tw6Y`p=4k8w(j1zaa*YEGZz#Vu%#+!54i*eTX7wuvk3kl3NTV1lf zM>qFtpLTev)DXlF*&Umvg;GH1Do~FiV@_tIS=@F#!h7$~b>z$T2z`hOrYO)Fr;_nj zD48=`Ig8cv?yS-__SOB-rCq|Zs>rqtQ#*e9w-_`5bW#GXX2_6@6->mYTIZB9v-v*c zeqkTJk6ztj=GA*t{4|=4U*y#LcH=$Y8$DXDCh0r{W{tE`_%Q{xH-iBPLKI8NukQRQ zjg00wdY5QYF{nqQ!FqPxaN2~ONH9>N9K44Zd`wR)s=BDVK@S5eh?#&9pexa zpFv_#jkK#J?ES_6;LI*9Q2k1SE`vsz6oLnGqmc96XN9nKS3WacC+h zL3VcnzF=bz>ta;==;Wm2S37^&NcQBOV zQ)?4-8e5dve0qs(WsMBK8D=fI=gXl~v$x#SRoVZO2i8EgYD8o>Un)P3o_>&4Bt4~?lwL7qLwVwoShDN0*mzyIw z)z0(}4JG`>ko)fHs)VS?#4Ebgnkg>QNozx;WJVP!Kt|_&RL5DN0}2JYTj;*>4*>!M z&B4F@h*mFH?UK17=xIw1vQB-?Qn0lP?69kkKmbIfM!D1SDVngDu2$mkj7iC~wc8LF z+?>Y)i2~={x_%xIzx;**R%s1qYJluIUM|68{S zD}le)+1ZqenU=z>=O2R9F}s11s$pe~D1ITKh0G)Z#l}4t`I!0R=Rf^CY!nQuB2)=z zDdArVW_U#l#+OO0~)% zZGim!hIA@))63uSe&29Gf@Wc6-OvoJapbC9K17i}XLU4POjW7FRia)$$YHbgo3GKv zuu;1vO2s`Itc20gRM_8<;5U?-kZTvY_cmR>;;Gl|4ZvOE{{vjWY`;#`*A4d@f}LGoRX2Bq z96-tF8{H@jP7J&oJdv8Iw}HoF(Hn?tB>n+<3Qp}cWc`hSFXx&R+OkBjW9)a{7To*o z`r0ol1`xkN7^OcSdBV`orHp+Aa(y|Gw@m1zxB>-4F_K<`zvQPmhR2c|gp;Sz8u)$P z1WYhHvN)d%#<5(;Ae}?M-&cO8gw{0K3N)CJdnyCT=}7DtPTg*JQAeMf$5aCV&tt=f zI%ZKZ_pZ=lu;R%9@}uns{JtSV&0*(aj@ct8zBQvMU~^e~h`L|!17ecYFkk`HzF@d4J>t9v?PqENQzkkQG;fr?Y);s(aA7ye~+)swIpFo zn2CG)uAd>Cdfu8QZkg?jQs~l_dpkG81EdMWT^T`sp-B?>3%Dv>d)or|HP*wswCKS= z2xVm)tJSjB8vmOG`}j-Cmh~Kuz}z$Up+P``hPv96ZZTd8tY%n^fB>vuYv!;_OYM?H zBX%O->-e164Gwb&oG08RP+*9=-+R+nu2j{aic+M$iva9n+TZg{X7#^w0eQQBP%;q% zKWf<&X+&W$-%bX$9}Sn}-~;77lS)^sA38hGTg2`Ys=>SsTlAw!C;nFb@d5M*#D5bx zY_7>fu7;;kPG$Ky){vIj3Pu3tR#VbtE$D;tNnEZ`Tkyg1QNGKmtsaJpktFZi` zXC-h;tAulAM;;MKRn(7-2^QcNo>(7FPweWJjG?{WT5o21y3bTAk*!}=v#$)BU21US z-=*G|`g_wV8ci)IR(k0qDzZ#=^FNf?YdDPKmC#^;IApqkgkxPCt6e;*IZX zoomIxLDl4FzL1ULeJ0dhk4Wtqu3kuI zWOZtiB_hgZ;m>FCj(35(*;mz2s{Wzmc6m(StB>lAH%vg45sj#>8&j9}w0PD#L`j}A zL^~Clh^t?ywO=MvAf2MmZaYD_WWKa6vUfyuq{6QH$_np0@M9;jfWXBC-9EtaleXWU z`c&5VxAU%dLA3L$2{pd#*nsr_8f=F)GG!cKsuQrrrequa9`py-T+wM1UY=QXP&!_( zDsM?Jv?Es>uhunf(%yS6vV@eeymO)y!H-?C_5_f+5kIl7LcgNJma5alTWoelG-da# zD(NYYr#D451JDmkECmC9%rBAJ@up>F9-(gE2>`VM@2@Ae8}9i#u5tboUUn=86XvGrN(k_ZH&%8I28D7POBzKAn9}-1Y z%lvEgyvM^Ztf@tIoO~9ulzr&MOzN0@tN=1=WHBC1##%s%E+mMhOo4^#;#D&Hfy-f~g>91*f5&i-)Xiham+JM|sTHvq&T^?q z&M(KyMgU^LQdOj99i;5kaIA#iJ~0w29A|y0{^64uh5M`Wo36san~TWe;g7E@{#)g1 zB|sV?VA#)pCZ*g?H1n(DWn$gZO(?>=>E{8?2XX%H z4XkR7WF`b0gevnQ4>SCBBx-xFn5tBw_85NVN3(0WHbFFB@kBtLXWa3L$YJ^yieF=* zEz2UdC$V>j+_&G{4naja;6ml7j5UE)@<&&XI~Z^EXgNE)Be)7Mv1_kKd2YAqwG2m$ zhM4X*OJ#z9!fS&0d%D@FJt%>1xVEs2=&9PhOfWy`rlJ`?69*1FH@mhcc?L`h4oKdm z{w+YA>T#q)d?5Ng{l`rHIVD4uQybO0Z?kQ+Nc?S(E;@Z0+81+kQ*YGApo|WTwD+qZ zFY{@&kdoR!+8GP?|;PfFxQ4HFWW8XfA@wZ&n1ajBq$WxQXym`Nbv?7#cI7W!z7s?E| zVZR~|5Xm2NDu(o&rCiEtNBN`!P8F+%x6RxAAoX2Nl+^v>Riq65>4S~| z7l&AZ>wxt2by$C|KNa=eAt_X$B)_y~hwzNtv+bt^i@2%CeqI?Ow;B#CkV67DMLR{q zOA|h6qr!T*pOb%Jx@V3x(_A_{_8d9h^sgJl=bH2xRWHr2*oep#m`p%S1eHc{RO6H- zIBG&KJjbZMxpG4k^12>1nZ12RJL8SguP(CHxB+diUU*w>65gNkM05^ERH#hS;@pxg zVfmWZ1xPvjngQ+3K3InRF?w$frI>w@Pn#X)!c2VgS9-*hyw@Kd4d*}N)>mWmX^}!h zamT1V7b(OdKX`?Vp;blIv8w7NDcB8v=zv1DE54gmk55((S;`BkV=>4dv z*`DfeixSBUP{32&fRsVLbb0P~4R6tr0^mbjZIS5grS`|%-7nmH0cyD0aF8hNJ)0z0 zldjgyt&pMsgx(A zx+o8m5(D2L3C1qou`%%LJf!xgT=;l4e{s*K=;dxV;Y2Z5Gf{cuWH=?1Yw$xsftJ%8 zxW28oWt97(>KNzdBDOQk6Bl`(B0npZxM#As1&`4^$qYDxmOd0&)9!e(-{(rlju2nc z@5pDW9&;bW%}w7+k$Tr*SS?+pKp|lms_G+kDs`h+oWYyW?JCxp`VoL)z+o1dLK}(I zlXz=4VOx9_9@2MKcwS8c?&Ho9o@U_sQhE=23vy~=OBRa$m06a zv!kC1?i@M3(|JjmD@JD?hMXHt$~3SD!>V^3VY(qX;GcB1KCz1Fud$;mmkFLz`wC_1 z`?FhIo*9Mu+7zzO~#|s zezCpd(px!{{i-KI>R8``TbF=cWP#dnZZ%lpS747&IpC)VX>PvDud$S!M6cXc`6~p<>fvv>$YU+FH zr!l0aFEp*o5-oaUYxXR1`1WGgo?3US%wQWFEi^gOT1moMal)R1f049bti*63>6kyu zrnT>|fv~S4l<`re=;XjDf~lf0)i{=VW0Vx*WBi8)o=wIaO$>OfthZV{Aa(0*jm}*y7 zSEAKV2H*aC^{dipj!pk+frV>AkC^gXKwnlffjP>h#O3L1sFE=qlx`Bv?x@H55EJYX}6Kg^#U%8==WbThWiPm6}olMq8K%jgjOqUTogkdMJMX=5p-r>=~HwOix3ve z)ETrJk@Rt?XIoTe+a;l-Vo4w`UjhN~nckfxhc9EK!iwRgIyCQpy#QJ-FZQ#F2g#;( ziYyNp{l*NBg?yvd` zE!1#<@*i|faSIBFAW2|sZWoMx&J+8jvFW`wP%vQ ztR>3=+aFcTi+bQEf5!(&p-+dSzu85-gLNYja3_?+g?%=_<0!WI1 z2Ub4UG1^2JCLc*-+p~&*%G**&X!X2U=-7g0`l^j?FKr-($0v=>o+m2wLFwu?+%*;- z`3dkuNZZVdQ6LrJpT;=Y%rx~7*&1@~3rvUqN7FS%SJHIdSQFdHL=)SYSQFcv*c034 z#I|kQwrx#p>+R=T>&wrqd;4}*ol|vcpS^d_7J`Gn-4{57G?hWnMD%`C?JXOZ8~Ro~ zEAERy#8PZ5!rt82=f65hB^5#mUn8(cz}`gSa>pnLLxa^AKJjrs-dJnUc$*#WO8VT` zy77hIJL)0m*RK{>>Ab}2GvlI&`W+O^ppieEs2JQ`2t&YPN%U6>q6c2`ayK)SR?&uo zx$FEBOb)xjD*G}vGRmQp9QdTiB6-0RdI5M*85#H>(qiBg`O`ni$KiDU{2SU9LpW6g zy4LX87^BB15`9jI7=tr~^FcB!!HnNlPo*UK<;k=;TNW5T?Y+=bQ&Y(Ym~o=KGbMi` zBkr6M>WBy;w>jj-*cPHF^np6s_C=aSf$_V3wMM4f12wkK*T0>wR;7ODB9t#mhdf@CD3I1G`MH6I$W8)ibGWLuS5;$Qp@p+wys5s-SDgA5MY1G>rr78gqc1&+qE zd4m|Q)-NwFKW`THH*V(`$)X4Py|!|sCKyflx&m+H;CJ`s`1?*A#Vlrtu^&EF!9*4M zwn7(*M3vA*XF*U(^9h+?ijNhieVg8k{0Vg}5kvu>Lb`68W$H8oJm*Rh4cm zd=*rsMpNjE%?lb>t!8rgz+W8=zNU;RLk9{Qe!o3l{QjUXo*$jiV8MQo|AU^hoZ>vN z#9n(It(W?|2E(cu%aOz@6#q#@tNp^8UqvvQCmPX5=pZHrwewaJIn<*uLlaUw+-R{? zpzbO{7Du8OSgzD=ViERh-h!sQL-Bdl@L3#VWEyX>HH?c6gE>qb&(W) zAHF14(hF#ILBU@AS(caaSx>KjqH7P90JB-$Kzf7yL32y}6IqEbwa0^~18*rrL0vs` z)BCqpVcImSHmqr5N?3G6qF)1$eW0<75d$N2R+YLJLPZ5*3p{M;lQ75qV-cHLYGBe7 zXqfr_o2sFsaRaN7VA}LpuXo{YZq$Kf5}vOwKO)~~{;%JslB$TQ*R|a<%rEQVM2NdK#K<5^*zu zoT6%t`R{3;oiB#V%(xA>vWl8=OBEjhU0zjCV;q>%=^RL+XNAKE*6)S!*FG1SE*k_Y zjZfXCvVY{Rc-`iN-F+co^kYau(M2IZ70%(* z>Fp;K=_Bci1+mQ9p19C!@65rXXVPZ`Vdgwv#jV2bX>4Z4W4GR%z2S$ny#S$;8O66L z9N9dnF=V84vlLM&Sx2M>*t9=^YGdu;Vk|IBi5s>UHCJnH&t zi$3de>56Lu!wH~8LlA!%7A|*t{80`Q8Ir=`Ln!kwxzM)RqN(F~)XRW{?~A)^D_e|^ z%~4Xe8-9}Cv}8giQ6{FL^l)5OEp801?SQ)zE}k*J`)eI)WqDcL%gfoIU$GRMFx%tC z$%NqzDvs=TT?-B_0xCR|O6DEs=3Vv^qqzc_R8>Z0isEWdu-fDG+q#}_Oe}-fXSud* zFC^@IhGdXGDPl_*?hn2{@;dz(;4Mc*&%>(va=i!z@?1~|;S$!;X$y)V$DyeUu0@4m zxSZ(RuXs2(zs;m6Xw~Eo(D>XTKIuGB zs`>2X|GZnM*R~%ePnkT*0QdK#H*%;e7TXi@IhT{0WxaqSQ3&h)I+5+$MV?ejx;wwCF5w#n=$Vr`PaA`# zge?iDm>hMq^X;a-{2Mo5{2!y!EjL4WOvg-Ix-i){$@40?Ad^*jZV^nSl_$7WWYo|* z)CA7$8Ex`6{4(#lp}b$i5<4_nFxyDWhXR?215F?r6a1Kt@87#mw^rGGfBjdtw zf4I%H2bnr_T$!Jdm!uAUrhM2TEWA^cIUSv|{nPotUKLC(B9fX_Dd#YB;(*P4iK8Bl)Au}}jOWI~!Nu)6WZfe;8$K;DJkBz-+Pop6ZT$_k ztWytR8pNARAgXZCJIy;iVP4;QJ4Q4AOnuz3+tVYEw_RRVX5dp#r_c1nO71loccYjD zG9kT+iK~$&$x`GDKE{k7-?RwqSj%^d)jl{<^#lGG7_3t+LtuPS*a%`BA*A>oE-Bre zZ`erhuikmTV~3WY`gUP{kM{a?Vxc`oBO`>ky^e;|wp&S?LMdTwjTE>M6?m;i&~_Q1 ziuHyD1`4txBS^h}CaGN${=JRZ_F|A)Ib3iots3s~dNV7j&M#u-VJgt4+=!6etv3oD z2W_eF{w5wS$Ia5%9&ss=S)o&&5qfd=B8kYRS8syAvHdDqqbe(6yF_z39RCx+3 zAK&*Il(1lG!hy!@FLS)cKQ9we8*O`EWL-~BPxFh5D#}{<`T3q+gVxe-fti0J&SASH zlZ;8eqfGZQD#BPLnpm{qV_SiV2^Qr|N~*Mj=f^Dg`|A%-VkaIP93%=If^4j`EGIt((2fU^#^wbu{%O1He_#GUs{PveE&!@+YRashi_CwpEB+bU zsFU~LiFcFwX=o{o@5a|x`#^Xu%QFQBZP>prl21bCKyA*wDfkvgecHZRLRTYI2MVxB zBfsIFyM10YNOy*Zr#;zi2W^uwUha1uXAkE`Z^!^nZh8C^X?tB&fV?~WNv~(Zd{e^- z9!gjt_sfv!2@0FWPc-MJFW$||JCug393%z|L&Xi#s(VBJU~5!_2H?&nuNzFVwKQMlgYcz-N(mfh*0$=vYOty^!ntxuLY zojYsw;-!O#4Ws(vEtV%EC4$=yiz=|fBSy*VB~KhelGee5T$JTia3WOy<;%woZYmri31lZdCV+jLsKxpMi4i4rgJ zN2g%DXP2T_p4;TbNgf9b4TcWwXE!z`o_y{>a(%IS4+Pj3k;6n#eR*Qz7wvj<-nSf1 zvNEg@;Dh_>hzsUs4Z}c3YZYAsZ$aomY@h$lvi6MDO{5G<$*4U3qSCXlw3Y>_k`EUl zdEZm}lGztUg+cWy&lCBp-g*^A1T_B~dyx@CR%?rqjt}HWVW2X0o1cd`R7=usBN()Y z@);`g%09rXqOfkBv$J>Ovnk(4W)+Kn?Rhnr^=34DbROn_^M>pbMMQYW<;JPuPMF+r zfk1n>*N^q(G8*n@$Wi>Tjdyem>l9KW|Pw3Fe7O{pZ_wH`H87Ev z@MTGr*3)gW(lC;A;W*zB>3JveM8m1CMtR2zdlD9Zo{WmfITX|7cQ^YTWJ5yyb7(PYAHIf_3(~^ zvF%Je5R7z0-Htu$@2Xd@btF# z+r?vXmU}ofp{sBH@SEc{qeL7mgg*q>nB(77gCUDr{n`{2g0(540`1ob> zUt5X|y<^9;{OdGcugHp5)EimoBtOhK)^kTth^tAPRGBrq7uvZhY*1s}fe?It2aY*h z#~BWcsnYMa$=wI3Sh`2`MBVIH>?FG|u{CwnzZtO^Q)N>+ z_eRaij{B#?K|Wc3K$J1w;TCiin$B7p;qi6%H||OgeozOod{OZcw13~{15dU%fWiC5 zWVe|A;{@r$`-`9QcUZZ4yhW2{rC#U9QYA zP@!GM-Cu!b$|7+E4L*Q;F)BQe6vaOmna{`9E;@O4mI(?f#7A^jSlP4C9IIT9NV?0lHD>Pu$>jpT8QP&i}SL&4>;E^ZmO2=WDX5nYGP9 z28-NrnJgW`0}iv6->A90$ zNQx&#={9>?9#@s$Ai$PEyXg{^g+N_2S#dMJQl~w|mV+5|+7b$-lI@Q(T>x2FAD|^0 zJY6hz1Lnf*C@A@5oP~knbZK#ka_1~d#MFc4!HJEPeL+c0DQeWLpiMDM^CLM}>4yS3 z4vqo{ka2^#Tls-~-(R88^(uU9Q?mLy&jF_2rpMF%1NjqEyAHjqUR~gSKlk5Vy58(% z_(5Yv1o2#T%zLj}H+yj)!~7KTi|>V`+u_3hdt42OJ?^gBue$PzmEOJid0Z}y)|;%& zeUJ#|^w>^LPxr!pWKljgeQhi1=@BJs+lSczx^7d-ZCO^@Q6(2gV{Lo5w~3Tqw2PwzMR(TjK2|mJr9GLSy>o*T7_PD9`mrWvd+xS z2^kua5)crS@w`dpiW9m80vQ}3+^b)csSYB}m+*AGWKdht0#x**XrSyM&FhmBsVo{@ zo?LHFSH)FTu}Hcf5>tF?ZXO@rKAO~JKCwb}YW-GCwl9f9q{uLgE|=O>ZKu>zKkuZA zq*E20Ye2QV9u`?SICvlhpRK+xAAV~QlZdNozi3(?A z*JPq(pl~QHwcfcW9|v{48N_;^r}~ByC3e|%MxQ=v^vb?7_0-nk$rL|&e@HHs6nb|& z%|P#84dVW4PCmPx(OL{Duc4Ao+5Y)@CMD_o{6!ihH0B+!wgz>Imv(ZV)(1#+dU&5U zSazSf_QGje-Y(ne+FhuDYem5EmDySIaT8-4T6W?gGx&o8_D|^D~SXvH!cA^|H3ZBnP>~E2LXs&M|n4 zoQsnaP7;HjE_Vo*nX79`qvdj)?G1Ba!tDG5hj*B5Nezx;`ef|VkVq_c6f(6hN51}m z1%uwdxK*Pei~9ME3XuU?J{iyr{?8jEzNgmd#0x%6M_^o7fZ+KJgqvQNSpN2y3P0A) zgme4HnahS<$cG4i|MLXP^naz!5vy#y309K8IwiK9S^~MJRJKv-&CRU8l2(gOBKG(J zuClUnaVrjsDP7M+)j<%JF6Vb&@c(-bHJ~V7_R9uq`|DgK*w2=jn24dK>t?)~GM}HU z>keDLI&qJ4CuuRPd|(nQM1>89o+wU^O&e{g&zxH4shK}J((xjt-s$et*PU+0ebsTt z`LLj595HzNpzY+O$I`y(J#5{)X~K*0w~b7qWgUeEdMnAB>Dedvz<^$Qc?z|g>wFS2UjlWn{T)az< z3GsUlfn&o~=r8M5&2MnvZz9!$8!ABlo74j9m2bAi!Tq~Rz%D75daJ0 zZq4`tg7Z{CbHF?xi{1oQ^KS!Ux20cX_TV|(uq99}qr0}xalnH5FaZDHo2=)pk%~yQ zn2|<@=FJHsP$2Ca5jZ~o7k4tf8$u9SwnZb8%@Q+N$2rxgdHqU}W*RJG(L%>=lssAF zhkFutqORA23WT7(v)`R{su6oO_7@#>Qqo^q**MxntF*=u9fd(4iR2^$C!4!Jtfe^d zKN%TW>S@tz5%MUvo)fD*jrRz0(KQ<_FfIt1(9uZJ6!GV8|Echtb=A|W{`oLHd(u$lNFEr zftX>|_Qd~Z-!FSxr|FTxeuvSp$i8Wr|LbKMwmk&X>r88#Ii!F035}gIIK3{$$)Gn4MCcr5bCp-5&glSH}g@73^N*bifI8w8* zHpr`O4*+^!ih>Vq$6*4Ol)l|-dwq%bX3m{R#E|_*3>wvp7|~Z{_q$0BbDNVKYVylU zY;KfKJj&Mn0p|2x06qhmNMVWw7L_BfM+>gf*49)xuT!O9WjZYAjYU8j`6^8GIs=!y%kCIAo2nkAgDNa8bQusaA3Q@|7=QQ@F! z*2Y)@l%=WPgi6XSsmr&2_8@s*2ZTkVNxw5>K3=R38%Fc0ChgNi)Z;m_)K zUUT2|R)-&$<&_5?--ggwzn~cTvKB11ZtPPR!mtE&ywK>?Ac4I_*jhn&={*mr>^^!G ztLW2yj%@>YT66PDU(}Ckw9?(7;b63E-mTjgTQnJ{0Kdn-Hcd90Wo5HtkkpeVitj+5 zwm&RyB#YO>-27}phYLEh1Ov)exZnc?w-RQQpwf+0|Pv2G^+SX0X2i*a`<&n<#oexPN%4w^Odr<9?cS8$V z-(JSW?60KCX^#Y$_Rn?W4z#)v{hmu(RM-$U9^^d-+rxR=I@V@jO2D(?%Z3ghF1ais!sZMnlW=r)o?BcTT=~biTl2c5m_8{;hCkIx zQ?C6E9UvK*HC~umM7Oov+1V*NnR^J9NH3uD%t@8M;ELKAPW>S~KVkAMj%@eN&6ax; zjh!Ts3?*9)G=095zM*S>>?R{KQwNYP8)=x>?!eNwxBovEfRnB=Q^N@|9|iD<+`bZx z6FFxOIeiot{!NvnLWK?owc}<)xMX2~x`2>( zEq}FY)~N5$o>UZrvE1BTL$8603L737{@Jc=gKPO09VU%xdDLud>_+}52^928r0p&l zsEwH!Oft_`d|o9x4`w++_jY-t?zli6<=#ys3e2+|rw-sBuYe|Xfl}r z*%R9j?RfPYIg(SyCej3rFI)WPOc9E_P8ok^?}#5=kyWefjVptm13({N0^pqFbD^mT ze1%{s*wDk7Irj!BVK@mewPh{0_(oS&uo1<7-&m@aBjoFoAzz~)h}W!MjEF|w<|&ZR zk>W<_*J{?TQs3^q$xrKYb8`bqeMd*&k+}JHuxWl-qZ|Jjf+TVt)nNQihUAs8^_N z{Ot>LDlug+&F-YVDFcjCBKo;lwW(pnND+(im~hF_;dX8U0bwvOdVqsF{lA3y4-$v~ zP%?!96BqPE3Tb2r5eZ4q489Up@3Haa@eCYljeZ0~kNK{*_wRrr4++Bt&xx&-g&yOg z8yn&r5i5=o6%G=c2v3j^E=Ae|fSQG^$OV$fL?lGe^MB<90@H3$Pp;^OA=KXw?)uYE zI-7~K_n~4H==kp484V*kPBaJ9ZVbTS4AB(oL|vtF-4U|>L#|pTb0(w>$GoSWEgR4 zd9;Bb0G!kSY@ISpgcKz&7AB7NccgjDSQ+ldr6et;mO|fJyB3$O*HlI;?6%RTe^6-N zNbg5`#L>1z{k~&MX-P$&MI(T+AP-W;hBALqjx(FoLngP!jg6(HP|y-u+2vp`Zz^euxnSJImA=doaz65xFG`YWX1Uh2gr$yjkhD8nL+Rh_ z&7&6mzR6PpH8#8l96G{nw194AFSdvI5B7dBGSnf9M&neT5q3XwV+|tKF_lv?bZsv2 z(#pwEh$+&1zl_j7W>!X46+a`k(OQdHWZRvs@@i|tmYN<1>L)X}@PJ;mpVumrNl*-B zCSdkO+7Hgk)>twgjGsnrIFMSsM6<4GaT-OV%`QzbSP@LBFvKgknR*`l3gxLQr%}Y2 zht^>%7bo&~xjK|LSX|^(rq9ak^C|RrzN)GC_G2#y)L@<&6c-oQp$e{KRMrU+j4nQd zi3sZro(rUSSiQ^(3S0$a6oH(S)O!=!?CKw&Ft~hP8+JV`ES3bzqLkGxgYfn8aM`(Y zQ+FRq)$p=C(lBND$=Y*c`1<4fX)Nvduekv~dUUwk^Ld|xqiG#P0VRisqNcfYVWsqF zDM$?aSI>;J$3X0zThK(pXEF`3xHz=5{`@ZZqxsbKmx$);ZT@Z|m!SA+WCZp|MD#Flj8 zP3;#OIUEk5`3E|bD1OMtdd68%9WrCry&A7p7CNh8?l9xUEb^Ib5AJ`f{e8CEuq`I$ zbu!O@JAnfarsjQ3UG=`Ba=y})DNgta9?kd4*zNI8{QWYQXm)LBume&mmEVU-sW>yQ z)s-kL@Q2Z{*VXQa)q30Caz0N4sDPM7ag&IkyXo%GZr(T64=-^uGb_4NK7&{sw)|O2 zo~G*K&Bk)|rzyPq`*Y9)FVyU>zGRJ#gU)B6u@M*7h3q5u1O#OqelvDF8b*-YA{ng| z)>f@IRE_rBWZmtl+OGR&Z=ij`rj}7AqVmr^?L$d#+s;Jx%H%a+6h11FnX@3$q*9d! zxWB%G%Tz~~>6}V0W~Ya!FkTd(%}5@U?(x&`o^We=i+6ps>q4i=T4Q)*gczjztAAhH zb(XyBo$u~G-t#q{14Ld&XZSw#E|vWQy^TVqxUdzxo#lzFWz}fPO*x)c3ZLIQP#F5D z#^VXo`D(3i>H-f2Dv@}xYz!_bqwPP9C@IcF$L$ceXBRRE03;`goA@1kEdlv8Q8kE z<_;9e#SB@~Ixe0;!kyx7FHy~|pM}eS=P{2~bA`-8qk08vv!kNgvVfry&BFtovg)FN z3n!rCRj*Wq=uHXL0<4s7Zcv8ALvRT2OdO43y7tBYakYfp+#F;n)vpFln)Dkn-(2U* z*#ik@I{9PpB=1RQY+vr@xZc<84?>T7qzaE?wxy0nW(8dZ4N=Q8BLHnFDXZvJQ3zVs z32G;4Z7``{t63!o6#fn-$Pd~b4Oj+=1w&(FbBg(FE4U+F$w^J0)~u&;IWZ^eIub7H zcyVQ3ggfgvUKX~LoyHp5a#o~Bi{&X85B8mX+qExZ2s@z)6cS8SrK_)0j;Ag*tMODZ zCq_2;Fc}=>z=cerJvG@J0>{5n1L0)8EG}9rbhkFiFE#epFDNSu&;1PvLSRkj`M!p9 zb7PsPY#{!Z!1S&u(`u_N9zmS{V6N&+wqX+A(zX-+ESeS*@Pv(QZFOsRE?%{-t*1Q^kKpea3A-7rr7W`4%6O% z6n-S>f`@CyAx4LXqX?nmmOOn?QZy*St$0GeH$#I~%bnUZt1Glg^;byp@G7|cDlWFHM3z^)hiQ`=PrQzxi&8H36mzYb&vgg|q7D&JIElx0X~uSLQu ztC6v-{YzocsyrIi*3seh8b&NvnaDF!2JM`*Ft%5Nj&J9_ZKKC(Y^20J_&1wh=4r|& zb`{&Ntg56OsRb;OP5^dwIUpr3i;Isdh%Rbzb{gxK8*BIX=|?j1Ft4`^2u5C| zqSWof!yFs_iT(_@6D9WH<|H&IBODlzRQ|pC0!g`b%&V*UhN(!&{0@yyPFCwldP8D% z{uDA=S_uez;?ab&_`St{x1x!7B&C%41Fz#49+0R@DJ?8cL&H(ioAH0qJpBWuKNGe{ z{ajCquEzSZ4ELG0zhySRC;|`k9}_t{$FWv!BcS^9M=+#4qDKF_hZJ0+H7_H?H__P8 zz{Jizlrk(_sO4b@Y zmCf&k7R`@LLK2>$EPnS2_g#-jyHQdo|D=L-;7{D^daNNS{;5h4Tp2Y1lTuoS7dtk5 z%O;hScK&DJDZ)w{aR82DRviptrO!! zHQp3E^n>W_G5^+%R>azi4TrqX%3jo6R0er}St>cz;st%Hl^-d)a3g4gm33pM)3xsp z{2FKGkNeOF+O^Ir5Q_NJSvx8w;#9W$cu5yU3f*Wu4ER=o#ORHN(?ZP|bRzoxs@PAb z{UqUM@UfpW6Qz~PDx4~lHpZgiL>yWDrZbf388fyoNca_KHFd^~>M+2IdrX82qlI$i z1xd%kasd0_nYlH;>FNm{_64>175RQ3V*S1wKk9GbP4M0>P$k#%l_ZpY#6WRs98yqi zzd;YLZqGwg`lc~el7ps9j8kt))HAamEkh*Cm0rjBKcJDjp-4E&kz@wueffuAK z8igVLu&&64GT8ZIQ*PLZ9DBivoykEAs}7!CP4UjXgD=Nb-uR;;H>a*j0y1MQ=eukk zWu47Fp8!)~#Ic;LmfD07y;H&I4J0x-6;-HbO2h}J+wDTUjUr@k44!RF%FHCzA2bwM zHb%~Yk$>0`hOqu7(r0}J>@>a(Who1+!OE@+FM{zN7?9{+t!ZPjEB9l<_8Q;M@|F+K-)L}Zu8sy_<5WM*lK{yENx9roaku07b2Q<=yr44{`#B; zSD|0^Q{^vBSx~O!x?rbwjkc$a9kfzKm{q{Dq|M;rss+<;N)ebMWhBmz<#I@zhvfLP zK6+2pdIut&Ph#TWrc~(a#Ru=7Djf1wt4YU6pqe3@Ue;Q>O7I-eIv*i}-q~(W1H|Ov zc(P%w`rH8-xJ@zIbjx11TZgyBS(%D-PPpI#N=Dj7gLk;kvW^s+mNcCH9;7+E1&b+r z`dnbPMcl2Eh+Jwt2vEKxU9`8H)dXeEt?gr)ol0eB6G8$k*zp@>~S@*Y}fN;c*uB6wE5>gbfZZ;pZ5EZg#{}T zT1t`R;+Ug!Tw3>zz1qwoTWaOHi}{lnoalJR9x{9ovM5RfKwfe%TIgRQfUb9KTF-<4 z`yt-{0uJhYwK?q{RBWCmP)xH}0CmgkXK|^>SEn=_XP=#dqy|xPZ=)@eX_YNFYrpP% zIcm8hf4Lv(=?(dFzF5{^73a>(S|Yd|3H~oCMj@-Nwey?Aye%I4A8~Z@qY0e?Djw6Z zUyDTW-^IwiS`Waf56YK{AFBFwM%mh5CSS+)C)9zYyh@8lRp zATS}ygp7+p?SC!Wwk1Xgd(schl%n~3!vd}=pli~YLAvqY|*xSxgy{4mbJ{{wBfKQ4_OivgZp8w!BmXkmyXmTs0uynS*hZ!D+uqA zr3K!<^HY{tGQ16a}?ex*uTsflf9M<5i_|DW~cI$ruEmWHpA5E=}_|z#|s>< zXHYaXd0>M2&m$oH;rs;*g_%QYr?Gxr>#A*su3F_j{j06p;3sl_%X|P}2Y*qiZ_p}# zlX<1I4uOyc8WQGG(M;h+^vl9WV6P`q)8x1qZXVW`w4#SFDB&V`_*~8zOHK5E>3l-U z{@FjX+%Ec7g`s0PdK4T8QkluhdqTSD+&?cfQq@%H-M|n!%L*Qob7Z<(INeXwY2p}F zu^#BtFKfjXRsx4;3$2r*kI4EhKwDqo&Xam17?CGR5iC&%xc&jsokW!f_(66fo=ikd ztu))VI0rjF0Y*&R!t2esHHoWOH0YMZ>PnNb1&sxcCXgU-F*U72Q0fN{a%1tL?O&b% z&C`L~=@vW@&Y`aDNler_{>$mu@5gAuGsEG_&J6D%dnaoZkP`b*dOXLqoH@&HtVg@yCbADvGho2= z&9<*1S&|)cju}JTqfg=T@Ez=+4SdVP4~s4TSoWSsPfhUPkHYF_PiM~zOjZ&Aqh4$H zL4Fq#@p)5bAS+P<^5{EO=liQgg(}S*W|l&n zULr%uT*D7)V>b{2X0{C3L+OEd1_#YQ$zlX_Inz3}xQHR~Dz&LjbkxJ2!9d}%W7|?v zQt{jVi3}db{O77tQVW=0frRx{2G~KfUpsmnS(ut?=VV8asX^_EBbziu$qZ-`aI2&A zL^%Pc?7aifd&Zh%mfo|^5=&H4RPr<|skF4*542m_cNDQ^==DPZJ$@TpeWy}AVQlz6y{4H&UkIC_1xy4~;sg!?_oOk^>}VzeIRPK-~J zOTQALH)e(MtrwBhsEv1s!%5x{3G)gYBa)6N zRI)1AE%!nT%~&PL@CC(ImuF^Hn0Ik)jIFp27Raxt#>NgT!4^-=#^^9;xDQ7aEUMBi z7N{nc$G^TSl-3_a9)bYjnM|bCp}{*d4|E?*md@Suk(FPD^m~5G1!ui^jjpszNzC7F z$!XC7gRix)#Kzd_VbTjY96}sLfXd{=dlQsIDc=32&kmtY;9BH9L9{kqRt9QqH?Iet zHZUq<6rW>4$3PEfftbC3WO6#3o zn$`hBIUnMaVJErNAlmWsl}@=k!orFEkB>wS7;zF>t~39Q9K~H8DH~j;5i1&`pcnus zVeX5!e^uck6o}wrMb?vE=;OL-#g>SB9=0rJ4N?r3*laVza&8YON|IH{+IwG9ED{6z zG_u+_C>=AOZFdAwgX4Yc8wczrWp&&EUw7)xPciZDl+1Jx;BX9(>uLhtMokj~RnC zZuJR={H^)3Er>|?bE0Q9CO$gkB*2K zZM&ZN)!W-Ij;f8t(r!~r83Mxpu)**8JS~Sp)!&gbO|)~g-Ys*yQGe~)+0Y<6cah&= zoE(xkJD=5hnLLVSMsJM)QkDTSIg?X+Tvsb-vhk)?@n6a9Mja zK_XM*>FM>oP|p2yik@ecMWapc?I|S?E@)O${(CDsOg{e za&`rnEKhr%mjmqqUDrbi@6U98`!C_>nXRRX+xqDp;pFaUhor2kv~R|DOPTMxH$U8d zAD*4_xWY)p0i?wYpQ*RjJ+I~KgS+UO+o5hdE) zU|}ezmyt-B_h^8Q(dfRE+49&8X(0R_Rjj;*#>3lI?gfX?7)7g9mtoDrakJF6ywc)C zfW-T5O6dMPjSUY4tmT;Xu!6`i!F?hmv4J%DjM)D!(}wEEuMW@2kCjw%GpOu__8c@` zh8*4xZ{9R+uAYtumBj44?Ui0~XObBd$ff#ALFu6#%tbYqk9xI5jmc$;(f(L%jS)%3 z7B0^1^w^omWrvU--uE;e$yR;#L}`Ec0eS`;eZ6TLcf_lG^)Hm4TXsLUm@LYv?b@l$ zEvyR5E7LXRkLH%*DnZk66G#J{FIGm}?0PVAaQ+hUL=wX+EK$bxLP5x;*7_=$&nGJcUs3+q5)R8ezlwx=vG1u2=(OZ(C$%BC_A)vxv)J(RP( z%*9JBO&fzHa?c!A$Z@ggV~%Ig2jEN;g3%*ilPcxA-y9)W#*oEI5GXZj7S&rbD^X0& z&`yiWhSarhvG64={dlnm*6)d4H0U=Qk(K$wik-^zfPOTgEzjy>QXf>+WP|v4NmC6w zA#PK$ONOWHj&QKyR(*r((MCe^yL{zwI6O@&n{LmD^Y^(Brx z37Z@F+Vm+vjK7fkEcO?6#9*0Mfc7af(&_0x3RGAYK6cHN4SA~hl6>m_5R9@}3S8{a z0LJ5~x5((#ghEM`u0z*v)>NQTe)`0{Qb)#}A&CtijcljTPq+}E5=p>%(DU2HpqX5z zo=n2E!w?(74^FM{Xn#Pwxk%l6!_ixXkSsqNLXfzE<%V{241H>5gqXb?A!IX^Q&BO= zxPuRA)-3R}OmmX`)R1QY@qBYE1Jcy|Ryk#3t~zww+OY+x+65h4pEcEvo5oI6+3=T) zSw;xdBzEDE>&+MlHJz-Hz7?4pp)xBf5YGkxC94bxf$|NKdh^sY0(hf<0;8gqPBi}x zFqI6Cn3C!fs@hd>6BHl$$->{sWs>`MWx*;y91I)RaPSs&og)}57gd*0;32UI2_i+Q z##5v*H)J_%)Pm@v7Zc&IJrK{25?-i%rGB&jD4Fvaqy7fb5g0H!-}SM@{V=f!>s&5fLOpsPY>7O(^oICSU|wdHeKZd1hJPGAhGgwCkxC( z6JwWFl=Pk0ejg1Fmg03#jv5-mxL#gj!(i9LI8!KYj{_o*u^=mhkxzmE=JnA>7nGs)$H@hB$D0uz)HQ+nLCXN4nxwNd` zikXGA!rS_mEAcp5q>ubXCC#4H&00o>$4MsM>?Fl+FwIfHA5=~d`xi$kPG)LO0|PGq zD7SIxqB0!x>)Xv+;^gEQlcZ#!FMo_G{|(dNvI1#9ukWf+QDZ51_V)OWEk8v*mGfSX zADj!h3dINb%HIbXJhP%{UFC%G87?=da$T>qE?jJv`%9J15TKt}%c3is3&LI=H5P7BR&K$RJ^>-zsz*^R=k{54`uctt+@#$A z62DqS1!-%*TA=<{gxAZ5e=6`*G9bf9v2nhR&n%&QAvP8Vhj8FnC9a<{)27U@{#Gm> zMFBmQJWQT3l+D;15l3@9@vt~IR#3$uQ4+yP#?@;dm=jV7MWGz`2E%F9yjjY6SuT}z z>w*27HTmjT$zP}$RbecjYM_2R=IFOY(=p%B~#joJwZ9a zBnplmVq$EWU$y4hwv@h$rwbSPoN_nZr?9X}HhCegUQtDVpUudGzU+**iv*F)EG~XE z#KYA9o_K@v3pis2PG33heb`j{j%L8tj3Wc+9x9mG)>>p%SAAxUiNA8O?l&BQs#8J}Z3@10%ZP51$3d*i9Pi5J#jOi|(hS+D16tE@YtW zzP~tAv^-4AG%ny%xCBHIj~dpvoDbv;{Mx#5A?sJuN(^OiAcOpSE7&n@P5F z*^qQ#DYYm+yPLDgc}`_-r}?w3nW%>mltbODu9b&vY{lsn;6j$63}?2%@Q@W_XfB1M z-0F|I$lAX$!blj6ie=KwQ(9FjM9)dNhxKTgo6^tX`WlSU(Ivk5x=L}Fj6 zHgmdfbewsOI;ooB`0>Mtq5_7x`WDRb6a^`DELMBsEssVbM5EEHI?I+{jx5V;+NPfJ zAy~u#YyNvh90-_Ge63K%e4H%HeBeW$LJ$Pr+3=>SK%&t|&gH92df}D75)1}-|COK8 z2bEqQklqJXHH~z2pTf4Co9OA?uX^8LFi?<2_6K~d-||nYsvD??O^N97L| z@sXnmMd5JQVYa$4ron}E!V zuRg6R7+IE;NZLFE`S6p!CJ^*<&Bs?{9j%@ZsMPBz&(b}8KGS9{V%x5bY|t37#zT^F zE!an*5uW|WpD~%tTy_1|^m)ohZhEpqS%UM9BvV#6tJ)H&FN~(7uN{p9(Hw|m@C3Q0 zMKTyuwa+BLn^orbc^Ho^mRqe3#SEx0m)c?Glt_?J$e(>r;IuO@qQ0?>)&G2j!$%LQ z3sREsddKkxe40lhZQ8buzrXqnb@eS=cGVZOpI=jmxu+Js`mKjBnaup|Pxs@|SRs*R z8Nc6$$2*Rw5li>nOHa^0c${U+uR<`I3o?_r{3^9F@90s)sN$N~Ty>Z&uEg^iO{psm zLDhazlN;nbFE!t{B12orGO{EwK01KaX2)u?Q{UVHXq(dbKeN@5xF(@-RO)@@j$a@n z^P5M1s;kLDp&(=9BY1rt!r>6{h3|o(K7Rbb-58BVzWsy076ht7rWR^yo4NKA-(=h~ z$^(D64^h@O_0edAanBfD-*`$Sh_bQgv8R5E)oSNcU;3fC_Q(akeB>5;zDU?d(5EO0 z!DzuAGo|r=MsrfbiIEVpB<1EQ4RDzqmM^IEI^JNQzIhTY?K5a;pMlwI&tgE0IYBVd z)HZ{rwiz_GPSxzNS#!_e%yTZIr}qGVc;W%%#3QLwBofBw_cA&*LhrFd-1VccG3FWN z#?O9-mbUJ6P-$o@El}kW^ravA0<-6x$+n#v`Td`MiHv>_K_n8v>+>+?8KrOF7~lKZ zorEJ{K7ZSPBM9aKdD8Z))cBT=@Qxl%wCJpC%Syp>cuJ9+b;FZ0{Se}O3F-sRpmbey~I`#Oh@_Hh2i*K^4g zpDvmwy*gyn`PTQJqN!yPtJbXK7r*@x;XFptqoYII^Wzot9_!)U3$NkyWgpN0m5S_C zB2<_tkbqA~h;0ze*xb!&eUtjxXtEIrDFGN_B!Fy7wxf3Y z__YQ={?()FXl|X9AE;6a^yE}5sIq=mSe|(m=UVjQnl4z)FV&=3tR9Ds_ zNfO8VkFsgoIzr(fR-1#HKL4M*=b~%UHi8rjD$U2x>7$2!zJis{{T5M-Qe9Qc+*u2$ zsj4FyiPArGjLqBE6A1b-n=M@PiGSy!%RXHYPsPYhmSw!ddl4f_c$BkxGA4^#^PbV* zDE_e?08ADaj;d~5p8Ad*;P%g+rU34NM`D1>ler|5I)(9T^8r=H4{D&who_$X+xguu z|D7FM-zFSNPBRz`2HX`jEIsEkZurdq!D3e56H%~6j};WI(Pakvd-?q@zsctHuPXD6 zu}TJm0jH~q#Y^AEM?ZBJ4rfK`Wo6+>O)eo{KS8e&g=4YTV5?}?efgwl2;cC|7|$kL zb+c2dXEWM=>;Sia?sWZ~_jxm~au3y?viK+*RG`gk$D~=@`Q1Mw1L1IxLkD(XHd*PM zoHVhi{zT@D%CFHgUY(RWwe@X$`J2B5AQTRA_~3Rd7CY@-Gt+r0QZBmJX7D6MLxj9a zjGZ8uu{c}OuT=rM!Dt490J4NEN0Fp327_6Xr!*g~5ax8tHJH6Z>cZ@MoFLaF8u4SYD|2Px0SOsbBIof%$+c<)?}6Fm0X+=SQ8V(m_=UPkM4Gg{Oqpu~lIrxz|^9aFW@K z3?_SH)^AzY$?0WjCN(#6CmB?#JgHw)oc7cL3OKg9%=U8AQ0P9|HR zc{5)CbGqd!JfvRn|GWuQ#n1m-6PQ_0td&XPP5ER380hQGJ#|nn&5%n+7xGg+9hOJ= zL=9>*FFesEFrV@~c+wMkE}n)5WBXb3c=#9ni^@&^7w>vCbuaWgZ~y=R07*qoM6N<$ Ef=7*${{R30 literal 0 HcmV?d00001 diff --git a/doc/dialects/lis.svg b/doc/dialects/lis.svg new file mode 100644 index 00000000..58013a35 --- /dev/null +++ b/doc/dialects/lis.svg @@ -0,0 +1,903 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + λ + λ + λ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + λ + λ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md new file mode 100644 index 00000000..1970bee3 --- /dev/null +++ b/doc/dialects/lispython.md @@ -0,0 +1,184 @@ +## Lispython: the love child of Python and Scheme + +![mascot](lis.png) + +Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. + +```python +from unpythonic.dialects import dialects, Lispython # noqa: F401 + +def factorial(n): + def f(k, acc): + if k == 1: + return acc + f(k - 1, k*acc) + f(n, acc=1) +assert factorial(4) == 24 +factorial(5000) # no crash + +t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), + (oddp, lambda x: (x != 0) and evenp(x - 1))) in + evenp(10000)] +assert t is True + +square = lambda x: x**2 +assert square(3) == 9 +assert square.__name__ == "square" + +g = lambda x: [local[y << 2*x], + y + 1] +assert g(10) == 21 + +c = cons(1, 2) +assert tuple(c) == (1, 2) +assert car(c) == 1 +assert cdr(c) == 2 +assert ll(1, 2, 3) == llist((1, 2, 3)) +``` + +### Features + +In terms of ``unpythonic.syntax``, we implicitly enable ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, and ``quicklambda`` for the whole module: + + - TCO in both ``def`` and ``lambda``, fully automatic + - Omit ``return`` in any tail position, like in Lisps + - Multiple-expression lambdas, ``lambda x: [expr0, ...]`` + - Named lambdas (whenever the machinery can figure out a name) + - The underscore: ``f[_*3] --> lambda x: x*3`` (name ``f`` is **reserved**) + +We also import some macros and functions to serve as dialect builtins: + + - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax`` + - ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil``, ``prod`` + - ``dyn``, for dynamic assignment + +For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. + +The multi-expression lambda syntax uses ``do[]``, so it also allows lambdas to manage local variables using ``local[name << value]`` and ``delete[name]``. See the documentation of ``do[]`` for details. + +The builtin ``let[]`` constructs are ``let``, ``letseq``, ``letrec``, the decorator versions ``dlet``, ``dletseq``, ``dletrec``, the block versions (decorator, call immediately, replace def'd name with result) ``blet``, ``bletseq``, ``bletrec``, and the code-splicing variants ``let_syntax`` and ``abbrev``. Bindings may be made using any syntax variant supported by ``unpythonic.syntax``. + +The builtin ``do[]`` constructs are ``do`` and ``do0``. + +If you need more stuff, `unpythonic` is effectively the standard library of Lispython, on top of what Python itself already provides. + + +### What Lispython is + +Lispython is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.lispython`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_lispython.py). + +The goal of the Lispython dialect is to fix some glaring issues that hamper Python when viewed from a Lisp/Scheme perspective, as well as make the popular almost-Lisp, Python, feel slightly more lispy. + +We take the approach of a relatively thin layer of macros (and underlying functions that implement the actual functionality), minimizing magic as far as reasonably possible. + +Performance is only a secondary concern; performance-critical parts fare better at the other end of [the wide spectrum](https://en.wikipedia.org/wiki/Wide-spectrum_language), with [Cython](http://cython.org/). Lispython is for [the remaining 80%](https://en.wikipedia.org/wiki/Pareto_principle), where the bottleneck is human developer time. + + +### Comboability + +The aforementioned block macros are enabled implicitly for the whole module; this is the essence of the Lispython dialect. Other block macros can still be invoked manually in the user code. + +Of the other block macros in ``unpythonic.syntax``, code written in Lispython supports only ``continuations``. ``autoref`` should also be harmless enough (will expand too early, but shouldn't matter). + +``prefix``, ``curry``, ``lazify`` and ``envify`` are **not compatible** with the ordering of block macros implicit in the Lispython dialect. + +``prefix`` is an outside-in macro that should expand first, so it should be placed in a lexically outer position with respect to the ones Lispython invokes implicitly; but nothing can be more outer than the dialect template. + +The other three are inside-out macros that should expand later, so similarly, also they should be placed in a lexically outer position. + +Basically, any block macro that can be invoked *lexically inside* a ``with tco`` block will work, the rest will not. + +If you need e.g. a lazy Lispython, the way to do that is to make a copy of the dialect module, change the dialect template to import the ``lazify`` macro, and then include a ``with lazify`` in the appropriate position, outside the ``with namedlambda`` block. Other customizations can be made similarly. + + +### Lispython and continuations (call/cc) + +Just use ``with continuations`` from ``unpythonic.syntax`` where needed. See its documentation for usage. + +Lispython works with ``with continuations``, because: + + - Nesting ``with continuations`` within a ``with tco`` block is allowed, for the specific reason of supporting continuations in Lispython. + + The dialect's implicit ``with tco`` will just skip the ``with continuations`` block (``continuations`` implies TCO). + + - ``autoreturn``, ``quicklambda`` and ``multilambda`` are outside-in macros, so although they will be in a lexically outer position with respect to the manually invoked ``with continuations`` in the user code, this is correct (because being on the outside, they run before ``continuations``, as they should). + + - The same applies to the outside-in pass of ``namedlambda``. Its inside-out pass, on the other hand, must come after ``continuations``, which it does, since the dialect's implicit ``with namedlambda`` is in a lexically outer position with respect to the ``with continuations``. + +Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in factorial tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython happily auto-injects a ``return`` to whatever is the last statement in any particular function. + + +### Why extend Python? + +[Racket](https://racket-lang.org/) is an excellent Lisp, especially with [sweet](https://docs.racket-lang.org/sweet/), sweet expressions [[1]](https://sourceforge.net/projects/readable/) [[2]](https://srfi.schemers.org/srfi-110/srfi-110.html) [[3]](https://srfi.schemers.org/srfi-105/srfi-105.html), not to mention extremely pythonic. The word is *rackety*; the syntax of the language comes with an air of Zen minimalism (as perhaps expected of a descendant of Scheme), but the focus on *batteries included* and understandability are remarkably similar to the pythonic ideal. Racket even has an IDE (DrRacket) and an equivalent of PyPI, and the documentation is simply stellar. + +Python, on the other hand, has a slight edge in usability to the end-user programmer, and importantly, a huge ecosystem of libraries, second to ``None``. Python is where science happens (unless you're in CS). Python is an almost-Lisp that has delivered on [the productivity promise](http://paulgraham.com/icad.html) of Lisp. Python also gets many things right, such as well developed support for lazy sequences, and decorators. + +In certain other respects, Python the base language leaves something to be desired, if you have been exposed to Racket (or Haskell, but that's a different story). Writing macros is harder due to the irregular syntax, but thankfully MacroPy already exists, and any set of macros only needs to be created once. + +Practicality beats purity ([ZoP §9](https://www.python.org/dev/peps/pep-0020/)): hence, fix the minor annoyances that would otherwise quickly add up, and reap the benefits of both worlds. If Python is software glue, Lispython is an additive that makes it flow better. + + +### PG's accumulator-generator puzzle + +The puzzle was posted by Paul Graham in 2002, in the essay [Revenge of the Nerds](http://paulgraham.com/icad.html). It asks to implement, in the shortest code possible, an accumulator-generator. The desired behavior is: + +```python +f = foo(10) +assert f(1) == 11 +assert f(1) == 12 +assert f(5) == 17 +``` + +(The original name of the function is literally `foo`; we have chosen to keep the name, although [nowadays one should do better than that](https://docs.racket-lang.org/style/reference-style.html#%28part._examples-style%29).) + +Even Lispython can do no better than this let-over-lambda (here using the haskelly let-in syntax to establish let-bindings): + +```python +foo = lambda n0: let[(n, n0) in + (lambda i: n << n + i)] +``` + +This still sets up a separate place for the accumulator (that is, separate from the argument of the outer function). The modern pure Python solution avoids that, but needs many lines: + +```python +def foo(n): + def accumulate(i): + nonlocal n + n += i + return n + return accumulate +``` + +The problem is that assignment to a lexical variable (including formal parameters) is a statement in Python. Python 3.8's walrus operator does not solve this, because `n := n + i` by itself is a syntax error. + +If we abbreviate ``accumulate`` as a lambda, it needs a ``let`` environment to write in, to use `unpythonic`'s expression-assignment (`name << value`). + +But see ``envify`` in ``unpythonic.syntax``, which shallow-copies function arguments into an `env` implicitly: + +```python +from unpythonic.syntax import macros, envify + +with envify: + def foo(n): + return lambda i: n << n + i +``` + +or as a one-liner: + +```python +with envify: + foo = lambda n: lambda i: n << n + i +``` + +``envify`` is not part of the Lispython dialect definition, because this particular, perhaps rarely used, feature is not really worth a global performance hit whenever a function is entered. + + +### CAUTION + +No instrumentation exists (or is even planned) for the Lispython layer; you'll have to use regular Python tooling to profile, debug, and such. The Lispython layer should be thin enough for this not to be a major problem in practice. + + +### Etymology? + +*Lispython* is obviously made of two parts: Python, and... diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md new file mode 100644 index 00000000..196f9f59 --- /dev/null +++ b/doc/dialects/listhell.md @@ -0,0 +1,58 @@ +## Listhell: it's not Lisp, it's not Python, it's not Haskell + +Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. + +```python +from unpythonic.dialects import dialects, Listhell # noqa: F401 + +from unpythonic import foldr, cons, nil, ll + +(print, "hello from Listhell") + +double = lambda x: 2 * x +my_map = lambda f: (foldr, (compose, cons, f), nil) +assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) +``` + +### Features + +In terms of ``unpythonic.syntax``, we implicitly enable ``prefix`` and ``curry`` for the whole module. + +The following are dialect builtins: + + - ``apply``, aliased to ``unpythonic.fun.apply`` + - ``compose``, aliased to unpythonic's currying right-compose ``composerc`` + - ``q``, ``u``, ``kw`` for the prefix syntax (note these are not `mcpyrate`'s + ``q`` and ``u``, but those from `unpythonic.syntax`, specifically for ``prefix``) + +For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md). + +If you need more stuff, `unpythonic` is effectively the standard library of Listhell, on top of what Python itself already provides. + + +### What Listhell is + +Listhell is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.listhell`](../../unpythonic/dialects/listhell.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_listhell.py). + +Listhell is essentially a demonstration of how Python could look, if it had Lisp's prefix syntax for function calls and Haskell's automatic currying. + +It's also a minimal example of how to make an AST-transforming dialect. + + +### Comboability + +Only outside-in macros that should expand after ``curry`` (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``curry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Listhell dialect. + + +### Notes + +If you like the idea and want autocurry for a Lisp, try +[spicy](https://github.com/Technologicat/spicy) for [Racket](https://racket-lang.org/). + +### CAUTION + +Not intended for serious use. + +### Etymology? + +Prefix syntax of **Lis**p, speed of Py**th**on, and readability of Hask**ell**, all in one. diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md new file mode 100644 index 00000000..8bfcdbe2 --- /dev/null +++ b/doc/dialects/pytkell.md @@ -0,0 +1,95 @@ +## Pytkell: Because it's good to have a kell + +Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. + +```python +from unpythonic.dialects import dialects, Pytkell # noqa: F401 + +from operator import add, mul + +def addfirst2(a, b, c): + return a + b +assert addfirst2(1)(2)(1/0) == 3 + +assert tuple(scanl(add, 0, (1, 2, 3))) == (0, 1, 3, 6) +assert tuple(scanr(add, 0, (1, 2, 3))) == (0, 3, 5, 6) + +my_sum = foldl(add, 0) +my_prod = foldl(mul, 1) +my_map = lambda f: foldr(compose(cons, f), nil) +assert my_sum(range(1, 5)) == 10 +assert my_prod(range(1, 5)) == 24 +assert tuple(my_map((lambda x: 2*x), (1, 2, 3))) == (2, 4, 6) + +pt = forall[z << range(1, 21), # hypotenuse + x << range(1, z+1), # shorter leg + y << range(x, z+1), # longer leg + insist(x*x + y*y == z*z), + (x, y, z)] +assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), + (8, 15, 17), (9, 12, 15), (12, 16, 20)) + +factorials = scanl(mul, 1, s(1, 2, ...)) # 0!, 1!, 2!, ... +assert last(take(6, factorials)) == 120 + +x = let[(a, 21) in 2*a] +assert x == 42 +x = let[2*a, where(a, 21)] +assert x == 42 +``` + +### Features + +In terms of ``unpythonic.syntax``, we implicitly enable ``curry`` and ``lazify`` for the whole module. + +We also import some macros and functions to serve as dialect builtins: + + - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax`` + - ``lazy[]`` and ``lazyrec[]`` for manual lazification of atoms and data structure literals, respectively + - If-elseif-else expression ``cond[]`` + - Nondeterministic evaluation ``forall[]`` (do-notation in the List monad) + - Function composition, ``compose`` (like Haskell's ``.`` operator), aliased to `unpythonic`'s currying right-compose ``composerc`` + - Linked list utilities ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil`` + - Folds and scans ``foldl``, ``foldr``, ``scanl``, ``scanr`` + - Memoization ``memoize``, ``gmemoize``, ``imemoize``, ``fimemoize`` + - Functional updates ``fup`` and ``fupdate`` + - Immutable dict ``frozendict`` + - Mathematical sequences ``s``, ``m``, ``mg`` + - Iterable utilities ``islice`` (`unpythonic`'s version), ``take``, ``drop``, ``split_at``, ``first``, ``second``, ``nth``, ``last`` + - Function arglist reordering utilities ``flip``, ``rotate`` + +For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md). + +The builtin ``let[]`` constructs are ``let``, ``letseq``, ``letrec``, the decorator versions ``dlet``, ``dletseq``, ``dletrec``, the block versions (decorator, call immediately, replace `def`'d name with result) ``blet``, ``bletseq``, ``bletrec``. Bindings may be made using any syntax variant supported by ``unpythonic.syntax``. + +The builtin ``do[]`` constructs are ``do`` and ``do0``. + +If you need more stuff, `unpythonic` is effectively the standard library of Pytkell, on top of what Python itself already provides. + + +### What Pytkell is + +Pytkell is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.pytkell`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_pytkell.py). + +Pytkell essentially makes Python feel slightly more haskelly. + +It's also a minimal example of how to make an AST-transforming dialect. + + +### Comboability + +**Not** comboable with most of the block macros in ``unpythonic.syntax``, because ``curry`` and ``lazify`` appear in the dialect template, hence at the lexically outermost position. + +Only outside-in macros that should expand after ``lazify`` has recorded its userlambdas (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``curry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Pytkell dialect. + + +### CAUTION + +No instrumentation exists (or is even planned) for the Pytkell layer; you'll have to use regular Python tooling to profile, debug, and such. + +This layer is not quite as thin as Lispython's, but the dialect is not intended for serious use, either. + + +### Etymology? + +The other obvious contraction, *Pyskell*, sounds like a serious programming language (or possibly the name of a fantasy airship), whereas *Pytkell* is obviously something quickly thrown together for system testing. diff --git a/unpythonic/dialects/listhell.py b/unpythonic/dialects/listhell.py index 552bdfb1..ae39a96a 100644 --- a/unpythonic/dialects/listhell.py +++ b/unpythonic/dialects/listhell.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""LisThEll: it's not Lisp, it's not Python, it's not Haskell. +"""Listhell: it's not Lisp, it's not Python, it's not Haskell. Powered by `mcpyrate` and `unpythonic`. """ diff --git a/unpythonic/dialects/tests/test_listhell.py b/unpythonic/dialects/tests/test_listhell.py index d469f1ae..ded37944 100644 --- a/unpythonic/dialects/tests/test_listhell.py +++ b/unpythonic/dialects/tests/test_listhell.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""Test the LisThEll dialect.""" +"""Test the Listhell dialect.""" # from mcpyrate.debug import dialects, StepExpansion from ...dialects import dialects, Listhell # noqa: F401 @@ -38,7 +38,7 @@ def runtests(): # Be careful: # - # In LisThEll, `(x,)` means "call the 0-arg function `x`". + # In Listhell, `(x,)` means "call the 0-arg function `x`". # But if `x` is not callable, `currycall` will return # the value as-is (needed for interaction with `call_ec` # and some other replace-def-with-value decorators). diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index b2b4c4d6..308eb9dd 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -479,6 +479,7 @@ def withec2(ec): # Introducing the HasThon programming language. # For a continuation-enabled HasThon, use "with lazify, autocurry, continuations". + # If you want to play around with this idea, see `unpythonic.dialects.pytkell`. with testset("HasThon, with 100% more Thon than popular brands"): with lazify, autocurry: def add3(a, b, c): diff --git a/unpythonic/syntax/tests/test_prefix.py b/unpythonic/syntax/tests/test_prefix.py index 59225a69..5bf306c4 100644 --- a/unpythonic/syntax/tests/test_prefix.py +++ b/unpythonic/syntax/tests/test_prefix.py @@ -102,9 +102,10 @@ def double(x): (double, x)]] test[a == 6] - # Introducing the LisThEll programming language: an all-in-one solution with + # Introducing the Listhell programming language: an all-in-one solution with # the prefix syntax of Lisp, the speed of Python, and the readability of Haskell! - with testset("LisThEll"): + # If you want to play around with this idea, see `unpythonic.dialects.listhell`. + with testset("Listhell"): # `prefix` is a first-pass macro, so placed on the outside, it expands first. with prefix: with autocurry: From 3c8ea8971995761e0e5ec8f24516e272a7eebdf7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:25:05 +0300 Subject: [PATCH 062/832] update dialect docs --- README.md | 4 +++- doc/dialects.md | 8 ++++---- doc/dialects/lispython.md | 6 ++++-- doc/dialects/listhell.md | 2 +- unpythonic/dialects/lispython.py | 4 ++-- unpythonic/dialects/listhell.py | 4 ++-- unpythonic/dialects/pytkell.py | 8 ++++---- 7 files changed, 20 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 5b2d14de..9584ba3b 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ I'm also considering renaming 0.15 to 1.0, since the codebase is mostly stable a None required. - - [mcpyrate](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, and an interactive macro REPL. + - [mcpyrate](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. The officially supported language versions are **CPython 3.8** and **PyPy3 3.7**. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). @@ -37,6 +37,7 @@ The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (langua [Pure-Python feature set](doc/features.md) [Syntactic macro feature set](doc/macros.md) +[Examples of creating dialects using `mcpyrate`](doc/dialects.md): Python the way you want it. [REPL server](doc/repl.md): interactively hot-patch your running Python program. [Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. @@ -45,6 +46,7 @@ The features of `unpythonic` are built out of, in increasing order of [magic](ht - Pure Python (e.g. batteries for `itertools`), - Macros driving a pure-Python core (`do`, `let`), - Pure macros (e.g. `continuations`, `lazify`, `dbg`). + - Whole-module AST transformations (dialects). This depends on the purpose of each feature, as well as ease-of-use considerations. See the design notes for more information. diff --git a/doc/dialects.md b/doc/dialects.md index ae810a81..720a517e 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -1,4 +1,4 @@ -# Python dialect examples in ``unpythonic.dialects`` +# Examples of creating dialects using `mcpyrate` What if Python had automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas? Look no further: @@ -37,9 +37,9 @@ Hence *dialects*. As examples of what can be done with a dialects system together with a kitchen-sink language extension macro package such as `unpythonic`, we currently provide the following dialects: - - [**Lispython**: Python with tail-call optimization (TCO), implicit return, multi-expression lambdas](dialects/lispython.md) - - [**Pytkell**: Python with automatic currying and lazy functions](dialects/pytkell.md) - - [**Listhell**: Python with prefix syntax and automatic currying](dialects/listhell.md) + - [**Lispython**: The love child of Python and Scheme](dialects/lispython.md) + - [**Pytkell**: Because it's good to have a kell](dialects/pytkell.md) + - [**Listhell**: It's not Lisp, it's not Python, it's not Haskell](dialects/listhell.md) All three dialects support `unpythonic`'s ``continuations`` block macro, to add ``call/cc`` to the language; but it is not enabled automatically. diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 1970bee3..d140ec86 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -1,4 +1,4 @@ -## Lispython: the love child of Python and Scheme +## Lispython: The love child of Python and Scheme ![mascot](lis.png) @@ -67,7 +67,9 @@ If you need more stuff, `unpythonic` is effectively the standard library of Lisp Lispython is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.lispython`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_lispython.py). -The goal of the Lispython dialect is to fix some glaring issues that hamper Python when viewed from a Lisp/Scheme perspective, as well as make the popular almost-Lisp, Python, feel slightly more lispy. +Lispython essentially makes Python feel slightly more lispy, in parts where that makes sense. + +It's also a minimal example of how to make an AST-transforming dialect. We take the approach of a relatively thin layer of macros (and underlying functions that implement the actual functionality), minimizing magic as far as reasonably possible. diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index 196f9f59..68fd80b0 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -1,4 +1,4 @@ -## Listhell: it's not Lisp, it's not Python, it's not Haskell +## Listhell: It's not Lisp, it's not Python, it's not Haskell Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index ab1d8269..2743107b 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""Lispython: the love child of Python and Scheme. +"""Lispython: The love child of Python and Scheme. Powered by `mcpyrate` and `unpythonic`. """ @@ -35,7 +35,7 @@ def transform_ast(self, tree): # tree is an ast.Module local, delete, do, do0, let_syntax, abbrev, cond) - # auxiliary syntax elements for the macros + # Auxiliary syntax elements for the macros. from unpythonic.syntax import where, block, expr # noqa: F401, F811 from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn # noqa: F401, F811 with namedlambda, autoreturn, quicklambda, multilambda, tco: diff --git a/unpythonic/dialects/listhell.py b/unpythonic/dialects/listhell.py index ae39a96a..f26b1dae 100644 --- a/unpythonic/dialects/listhell.py +++ b/unpythonic/dialects/listhell.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""Listhell: it's not Lisp, it's not Python, it's not Haskell. +"""Listhell: It's not Lisp, it's not Python, it's not Haskell. Powered by `mcpyrate` and `unpythonic`. """ @@ -18,7 +18,7 @@ def transform_ast(self, tree): # tree is an ast.Module with q as template: __lang__ = "Listhell" # noqa: F841, just provide it to user code. from unpythonic.syntax import macros, prefix, autocurry # noqa: F401, F811 - # auxiliary syntax elements for the macros + # Auxiliary syntax elements for the macros from unpythonic.syntax import q, u, kw # noqa: F401 from unpythonic import apply # noqa: F401 from unpythonic import composerc as compose # compose from Right, Currying # noqa: F401 diff --git a/unpythonic/dialects/pytkell.py b/unpythonic/dialects/pytkell.py index 1f27a038..cfcf3cc8 100644 --- a/unpythonic/dialects/pytkell.py +++ b/unpythonic/dialects/pytkell.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""Pytkell: Python with automatic currying and lazy functions. +"""Pytkell: Because it's good to have a kell. Powered by `mcpyrate` and `unpythonic`. """ @@ -23,9 +23,9 @@ def transform_ast(self, tree): # tree is an ast.Module blet, bletseq, bletrec, local, delete, do, do0, cond, forall) - # auxiliary syntax elements for the macros + # Auxiliary syntax elements for the macros. from unpythonic.syntax import where, insist, deny # noqa: F401 - # functions that have a haskelly feel to them + # Functions that have a haskelly feel to them. from unpythonic import (foldl, foldr, scanl, scanr, # noqa: F401 s, imathify, gmathify, frozendict, memoize, fupdate, fup, @@ -33,7 +33,7 @@ def transform_ast(self, tree): # tree is an ast.Module islice, take, drop, split_at, first, second, nth, last, flip, rotate) from unpythonic import composerc as compose # compose from Right, Currying (Haskell's . operator) # noqa: F401 - # this is a bit lispy, but we're not going out of our way to provide + # This is a bit lispy, but we're not going out of our way to provide # a haskelly surface syntax for these. from unpythonic import cons, car, cdr, ll, llist, nil # noqa: F401 with lazify, autocurry: From aede2d12d84771366ee5a4fb89878e812c37e142 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:31:20 +0300 Subject: [PATCH 063/832] update dialect docs --- doc/dialects/lispython.md | 8 +++++--- doc/dialects/listhell.md | 2 ++ doc/dialects/pytkell.md | 18 ++++++++++-------- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index d140ec86..e2765660 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -1,9 +1,11 @@ ## Lispython: The love child of Python and Scheme -![mascot](lis.png) +Python with automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas. Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. +![mascot](lis.png) + ```python from unpythonic.dialects import dialects, Lispython # noqa: F401 @@ -11,7 +13,7 @@ def factorial(n): def f(k, acc): if k == 1: return acc - f(k - 1, k*acc) + f(k - 1, k * acc) f(n, acc=1) assert factorial(4) == 24 factorial(5000) # no crash @@ -25,7 +27,7 @@ square = lambda x: x**2 assert square(3) == 9 assert square.__name__ == "square" -g = lambda x: [local[y << 2*x], +g = lambda x: [local[y << 2 * x], y + 1] assert g(10) == 21 diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index 68fd80b0..030da744 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -1,5 +1,7 @@ ## Listhell: It's not Lisp, it's not Python, it's not Haskell +Python with prefix syntax for function calls, and automatic currying. + Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. ```python diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index 8bfcdbe2..50d44bd8 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -1,5 +1,7 @@ ## Pytkell: Because it's good to have a kell +Python with automatic currying and implicitly lazy functions. + Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. ```python @@ -9,7 +11,7 @@ from operator import add, mul def addfirst2(a, b, c): return a + b -assert addfirst2(1)(2)(1/0) == 3 +assert addfirst2(1)(2)(1 / 0) == 3 assert tuple(scanl(add, 0, (1, 2, 3))) == (0, 1, 3, 6) assert tuple(scanr(add, 0, (1, 2, 3))) == (0, 3, 5, 6) @@ -19,12 +21,12 @@ my_prod = foldl(mul, 1) my_map = lambda f: foldr(compose(cons, f), nil) assert my_sum(range(1, 5)) == 10 assert my_prod(range(1, 5)) == 24 -assert tuple(my_map((lambda x: 2*x), (1, 2, 3))) == (2, 4, 6) +assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) pt = forall[z << range(1, 21), # hypotenuse - x << range(1, z+1), # shorter leg - y << range(x, z+1), # longer leg - insist(x*x + y*y == z*z), + x << range(1, z + 1), # shorter leg + y << range(x, z + 1), # longer leg + insist(x * x + y * y == z * z), (x, y, z)] assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), (8, 15, 17), (9, 12, 15), (12, 16, 20)) @@ -32,9 +34,9 @@ assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), factorials = scanl(mul, 1, s(1, 2, ...)) # 0!, 1!, 2!, ... assert last(take(6, factorials)) == 120 -x = let[(a, 21) in 2*a] +x = let[(a, 21) in 2 * a] assert x == 42 -x = let[2*a, where(a, 21)] +x = let[2 * a, where(a, 21)] assert x == 42 ``` @@ -92,4 +94,4 @@ This layer is not quite as thin as Lispython's, but the dialect is not intended ### Etymology? -The other obvious contraction, *Pyskell*, sounds like a serious programming language (or possibly the name of a fantasy airship), whereas *Pytkell* is obviously something quickly thrown together for system testing. +The other obvious contraction, *Pyskell*, sounds like a serious programming language - or possibly the name of a fantasy airship - whereas *Pytkell* is obviously something quickly thrown together for system testing. From 47045a7cf05cead35b08d0d31aec161e61c221bb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:33:35 +0300 Subject: [PATCH 064/832] update dialect docs --- doc/dialects/lispython.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index e2765660..0528472c 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -4,8 +4,6 @@ Python with automatic tail-call optimization, an implicit return statement, and Powered by [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) and `unpythonic`. -![mascot](lis.png) - ```python from unpythonic.dialects import dialects, Lispython # noqa: F401 @@ -186,3 +184,5 @@ No instrumentation exists (or is even planned) for the Lispython layer; you'll h ### Etymology? *Lispython* is obviously made of two parts: Python, and... + +![mascot](lis.png) From 36ed8817bccba680e00ddc781a281f55b5b1d3a5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:35:39 +0300 Subject: [PATCH 065/832] update dialect docs --- doc/dialects/lispython.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 0528472c..4785b964 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -10,7 +10,7 @@ from unpythonic.dialects import dialects, Lispython # noqa: F401 def factorial(n): def f(k, acc): if k == 1: - return acc + return acc # `return` is available to cause an early return f(k - 1, k * acc) f(n, acc=1) assert factorial(4) == 24 From dd8742c322184f78175790cba550a4ceee108e18 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:39:31 +0300 Subject: [PATCH 066/832] update dialect docs --- doc/dialects.md | 16 +--------------- doc/dialects/lispython.md | 4 ++++ doc/macros.md | 2 +- 3 files changed, 6 insertions(+), 16 deletions(-) diff --git a/doc/dialects.md b/doc/dialects.md index 720a517e..6615ba91 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -1,6 +1,6 @@ # Examples of creating dialects using `mcpyrate` -What if Python had automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas? Look no further: +What if Python had automatic tail-call optimization and an implicit return statement? Look no further: ```python from unpythonic.dialects import dialects, Lispython # noqa: F401 @@ -13,20 +13,6 @@ def factorial(n): f(n, acc=1) assert factorial(4) == 24 factorial(5000) # no crash - -# - brackets denote a multiple-expression lambda body -# (if you want to have one expression that is a literal list, -# double the brackets: `lambda x: [[5 * x]]`) -# - local[name << value] makes an expression-local variable -lam = lambda x: [local[y << 2 * x], - y + 1] -assert lam(10) == 21 - -t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x:(x != 0) and evenp(x - 1))) in - [local[x << evenp(100)], - (x, evenp.__name__, oddp.__name__)]] -assert t == (True, "evenp", "oddp") ``` The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 4785b964..d9d27622 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -25,6 +25,10 @@ square = lambda x: x**2 assert square(3) == 9 assert square.__name__ == "square" +# - brackets denote a multiple-expression lambda body +# (if you want to have one expression that is a literal list, +# double the brackets: `lambda x: [[5 * x]]`) +# - local[name << value] makes an expression-local variable g = lambda x: [local[y << 2 * x], y + 1] assert g(10) == 21 diff --git a/doc/macros.md b/doc/macros.md index 31ca8442..e551522f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1411,7 +1411,7 @@ with prefix: assert (apply, g, "hi", "ho", lst) == (q, "hi" ,"ho", 1, 2, 3) ``` -This comboes with ``autocurry`` for an authentic *LisThEll* programming experience: +This comboes with ``autocurry`` for an authentic *Listhell* programming experience: ```python from unpythonic.syntax import macros, autocurry, prefix, q, u, kw From 56e7b6b5c3a6d2e999a3c3fa7695ac3c320c127f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:46:44 +0300 Subject: [PATCH 067/832] README: mention dialects --- README.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/README.md b/README.md index 9584ba3b..84b63b9b 100644 --- a/README.md +++ b/README.md @@ -568,6 +568,31 @@ with continuations: # enables also TCO automatically +#### Unpythonic in 30 seconds: Language extensions with dialects + +
Lispython: The love child of Python and Scheme. + +[[docs](doc/dialects.md)] + +The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). We provide some example dialects based on `unpythonic`'s macro layer. + +For example, what if Python had automatic tail-call optimization and an implicit return statement? Look no further: + +```python +from unpythonic.dialects import dialects, Lispython # noqa: F401 + +def factorial(n): + def f(k, acc): + if k == 1: + return acc + f(k - 1, k * acc) + f(n, acc=1) +assert factorial(4) == 24 +factorial(5000) # no crash +``` +
+ + ## Installation **PyPI** From 0fb13d14e6bada8a084647c4e0b80bd6e8661d48 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:51:18 +0300 Subject: [PATCH 068/832] update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 203b094b..82b2628d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand **New**: +- **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) together with a kitchen-sink language extension macro package such as `unpythonic`. - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), and `call_cc[]` (for `with continuations`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. - `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. From 3bc4082f9af5f105726d6c7659b82ca369f1838d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 01:52:45 +0300 Subject: [PATCH 069/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index d7444de4..7ec30285 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -134,9 +134,7 @@ # TODO: with mcpyrate, do we really need to set `ctx` in our macros? (does our macro code need it?) -# TODO: Move dialect examples from `pydialect` into a new package, `unpythonic.dialects`. -# TODO: `mcpyrate` now provides the necessary infrastructure, while `unpythonic` has the macros -# TODO: needed to make interesting things happen. Update docs accordingly for both projects. +# TODO: Now that `unpythonic` provides dialects, update `mcpyrate` docs. # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... From 4dd3359d73a06a096fe17e6482400c3d1c05b206 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 02:39:35 +0300 Subject: [PATCH 070/832] make `callsite_filename` ignore our call helpers This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`). --- CHANGELOG.md | 4 ++++ unpythonic/misc.py | 22 ++++++++++++++-------- unpythonic/syntax/__init__.py | 4 ---- unpythonic/syntax/tests/test_conts_topo.py | 6 +++--- 4 files changed, 21 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82b2628d..eb517636 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,10 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. +**Fixed**: + +- Make `callsite_filename` ignore our call helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`). + --- diff --git a/unpythonic/misc.py b/unpythonic/misc.py index 32c329d2..a10d1689 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -8,15 +8,16 @@ "ulp", "slurp", "async_raise", "callsite_filename", "safeissubclass"] -from types import LambdaType, FunctionType, CodeType, TracebackType -from time import monotonic from copy import copy from functools import partial -from sys import version_info, float_info +from itertools import count +import inspect from math import floor, log2 from queue import Empty +from sys import float_info, version_info import threading -import inspect +from time import monotonic +from types import CodeType, FunctionType, LambdaType, TracebackType # For async_raise only. Note `ctypes.pythonapi` is not an actual module; # you'll get a `ModuleNotFoundError` if you try to import it. @@ -814,10 +815,15 @@ def callsite_filename(): This works also in the REPL (where `__file__` is undefined). """ stack = inspect.stack() - frame = stack[1].frame - filename = frame.f_code.co_filename - del frame, stack - return filename + for k in count(start=1): # ignore callsite_filename() itself + framerecord = stack[k] + # ignore our call-helpers + if framerecord.function not in ("maybe_force_args", # lazify + "curried", "curry", "_currycall", # autocurry + "call", "callwith"): # manual use of misc utils + frame = framerecord.frame + filename = frame.f_code.co_filename + return filename def safeissubclass(cls, cls_or_tuple): """Like issubclass, but if `cls` is not a class, swallow the `TypeError` and return `False`.""" diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7ec30285..b5dd8fed 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -82,10 +82,6 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: debugging: -# TODO: The HasThon test (grep for it), when putting the macros in the wrong order on purpose, -# TODO: confuses the call site filename detector of the test framework. Investigate. - # TODO: Consistent naming for syntax transformers? `_macroname_transform`? `_macroname_stx`? # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? diff --git a/unpythonic/syntax/tests/test_conts_topo.py b/unpythonic/syntax/tests/test_conts_topo.py index ebdc3d8c..e61e8572 100644 --- a/unpythonic/syntax/tests/test_conts_topo.py +++ b/unpythonic/syntax/tests/test_conts_topo.py @@ -7,14 +7,14 @@ from ...syntax import macros, test, the # noqa: F401 from ...test.fixtures import session, testset -from inspect import stack +import inspect from ...syntax import macros, continuations, call_cc # noqa: F401, F811 def me(): """Return the caller's function name.""" - callstack = stack() - framerecord = callstack[1] # ignore me() itself, get caller's record + stack = inspect.stack() + framerecord = stack[1] # ignore me() itself, get caller's record return framerecord.function # Continuation names are gensymmed, so `mcpyrate` adds a uuid to them. From 715da15dde15ab21de7aa91923572d3de49ca4a1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 03:08:18 +0300 Subject: [PATCH 071/832] internal cleanups to macro layer code In a module that implements macros, the syntax transformers are considered internal functions and are named starting with an underscore. Usually the name is the same as for the macro itself, unless it has separate expr and block variants, in which case there are usually two syntax transformer functions with names that indicate this. Some names have been made more descriptive (e.g. `_let_expr_impl` instead of another `_let`, since `unpythonic.lispylet` already has a `_let` in the regular-code layer.) The `__all__` export lists have been checked and corrected. Ordering of the code in the source file should now match those lists, too. --- unpythonic/syntax/__init__.py | 2 - unpythonic/syntax/dbg.py | 65 +++++++++++++++---------------- unpythonic/syntax/ifexprs.py | 10 ++--- unpythonic/syntax/lazify.py | 22 +++++------ unpythonic/syntax/letdo.py | 55 +++++++++++++------------- unpythonic/syntax/letdoutil.py | 12 ++---- unpythonic/syntax/letsyntax.py | 54 ++++++++++++------------- unpythonic/syntax/simplelet.py | 8 ++-- unpythonic/syntax/testingtools.py | 52 ++++++++++++------------- 9 files changed, 136 insertions(+), 144 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index b5dd8fed..ac9628f4 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -82,8 +82,6 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: Consistent naming for syntax transformers? `_macroname_transform`? `_macroname_stx`? - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index ed720d07..5e629b39 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -108,8 +108,6 @@ def dbg(tree, *, args, syntax, expander, **kw): else: # syntax == "block": return _dbg_block(body=tree, args=args) -# -------------------------------------------------------------------------------- - def dbgprint_block(ks, vs, *, filename=None, lineno=None, sep=", ", **kwargs): """Default debug printer for the ``dbg`` macro, block variant. @@ -161,37 +159,6 @@ def dbgprint_block(ks, vs, *, filename=None, lineno=None, sep=", ", **kwargs): else: print(header + sep.join(f"{k}: {v}" for k, v in zip(ks, vs)), **kwargs) -def _dbg_block(body, args): - if args: # custom print function hook - # TODO: add support for Attribute to support using a method as a custom print function - # (the problem is we must syntactically find matches in the AST, and AST nodes don't support comparison) - if type(args[0]) is not Name: # pragma: no cover, let's not test the macro expansion errors. - raise SyntaxError("Custom debug print function must be specified by a bare name") - pfunc = args[0] - pname = pfunc.id # name of the print function as it appears in the user code - else: - pfunc = q[h[dbgprint_block]] - pname = "print" # override standard print function within this block - - class DbgBlockTransformer(ASTTransformer): - def transform(self, tree): - if is_captured_value(tree): - return tree # don't recurse! - if type(tree) is Call and type(tree.func) is Name and tree.func.id == pname: - names = [q[u[unparse(node)]] for node in tree.args] # x --> "x"; (1 + 2) --> "(1 + 2)"; ... - names = q[t[names]] - values = q[t[tree.args]] - tree.args = [names, values] - # can't use inspect.stack in the printer itself because we want the line number *before macro expansion*. - lineno = tree.lineno if hasattr(tree, "lineno") else None - tree.keywords += [keyword(arg="filename", value=q[h[callsite_filename]()]), - keyword(arg="lineno", value=q[u[lineno]])] - tree.func = pfunc - return self.generic_visit(tree) - return DbgBlockTransformer().visit(body) - -# -------------------------------------------------------------------------------- - def dbgprint_expr(k, v, *, filename, lineno): """Default debug printer for the ``dbg`` macro, expression variant. @@ -229,6 +196,38 @@ def dbgprint_expr(k, v, *, filename, lineno): print(f"[{filename}:{lineno}] {k}: {v}") return v # IMPORTANT! (passthrough; debug printing is a side effect) +# -------------------------------------------------------------------------------- +# Syntax transformers + +def _dbg_block(body, args): + if args: # custom print function hook + # TODO: add support for Attribute to support using a method as a custom print function + # (the problem is we must syntactically find matches in the AST, and AST nodes don't support comparison) + if type(args[0]) is not Name: # pragma: no cover, let's not test the macro expansion errors. + raise SyntaxError("Custom debug print function must be specified by a bare name") + pfunc = args[0] + pname = pfunc.id # name of the print function as it appears in the user code + else: + pfunc = q[h[dbgprint_block]] + pname = "print" # override standard print function within this block + + class DbgBlockTransformer(ASTTransformer): + def transform(self, tree): + if is_captured_value(tree): + return tree # don't recurse! + if type(tree) is Call and type(tree.func) is Name and tree.func.id == pname: + names = [q[u[unparse(node)]] for node in tree.args] # x --> "x"; (1 + 2) --> "(1 + 2)"; ... + names = q[t[names]] + values = q[t[tree.args]] + tree.args = [names, values] + # can't use inspect.stack in the printer itself because we want the line number *before macro expansion*. + lineno = tree.lineno if hasattr(tree, "lineno") else None + tree.keywords += [keyword(arg="filename", value=q[h[callsite_filename]()]), + keyword(arg="lineno", value=q[u[lineno]])] + tree.func = pfunc + return self.generic_visit(tree) + return DbgBlockTransformer().visit(body) + def _dbg_expr(tree): ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index 2bc1e0fa..b05dcd26 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -8,7 +8,7 @@ from mcpyrate.quotes import macros, q, a # noqa: F811, F401 -from .letdo import implicit_do, _let +from .letdo import _implicit_do, _let from ..dynassign import dyn @@ -44,7 +44,7 @@ def aif(tree, *, syntax, expander, **kw): return _aif(tree) def _aif(tree): - test, then, otherwise = [implicit_do(x) for x in tree.elts] + test, then, otherwise = [_implicit_do(x) for x in tree.elts] bindings = [q[(it, a[test])]] body = q[a[then] if it else a[otherwise]] # TODO: we should use a hygienically captured macro here. @@ -101,11 +101,11 @@ def _cond(tree): raise SyntaxError("Expected cond[test1, then1, test2, then2, ..., otherwise]") # pragma: no cover def build(elts): if len(elts) == 1: # final "otherwise" branch - return implicit_do(elts[0]) + return _implicit_do(elts[0]) if not elts: raise SyntaxError("Expected cond[test1, then1, test2, then2, ..., otherwise]") # pragma: no cover test, then, *more = elts - test = implicit_do(test) - then = implicit_do(then) + test = _implicit_do(test) + then = _implicit_do(then) return q[a[then] if a[test] else a[build(more)]] return build(tree.elts) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 9b1e623c..5ad380db 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -485,8 +485,8 @@ def _lazy(tree): # variant `frozendict(mapping1, mapping2, ...)`. _ctorcalls_that_take_exactly_one_positional_arg = {"tuple", "list", "set", "dict", "frozenset", "llist"} -unexpanded_lazy_name = "lazy" -expanded_lazy_name = "Lazy" +_unexpanded_lazy_name = "lazy" +_expanded_lazy_name = "Lazy" def _lazyrec(tree): # This helper doesn't need to recurse, so we don't need `ASTTransformer` here. def transform(tree): @@ -497,9 +497,9 @@ def transform(tree): elif type(tree) is Call and any(isx(tree.func, ctor) for ctor in _ctorcalls_all): p, k = _ctor_handling_modes[getname(tree.func)] lazify_ctorcall(tree, p, k) - elif type(tree) is Subscript and isx(tree.value, unexpanded_lazy_name): + elif type(tree) is Subscript and isx(tree.value, _unexpanded_lazy_name): pass - elif type(tree) is Call and isx(tree.func, expanded_lazy_name): + elif type(tree) is Call and isx(tree.func, _expanded_lazy_name): pass else: # mcpyrate supports hygienic macro capture, so we can just splice unexpanded @@ -517,25 +517,25 @@ def lazify_ctorcall(tree, positionals="all", keywords="all"): newargs = [] for arg in tree.args: if type(arg) is Starred: # *args in Python 3.5+ - if is_literal_container(arg.value, maps_only=False): + if _is_literal_container(arg.value, maps_only=False): arg.value = rec(arg.value) # else do nothing - elif positionals == "all" or is_literal_container(arg, maps_only=False): # single positional arg + elif positionals == "all" or _is_literal_container(arg, maps_only=False): # single positional arg arg = rec(arg) newargs.append(arg) tree.args = newargs for kw in tree.keywords: if kw.arg is None: # **kwargs in Python 3.5+ - if is_literal_container(kw.value, maps_only=True): + if _is_literal_container(kw.value, maps_only=True): kw.value = rec(kw.value) # else do nothing - elif keywords == "all" or is_literal_container(kw.value, maps_only=True): # single named arg + elif keywords == "all" or _is_literal_container(kw.value, maps_only=True): # single named arg kw.value = rec(kw.value) rec = transform return rec(tree) -def is_literal_container(tree, maps_only=False): +def _is_literal_container(tree, maps_only=False): """Test whether tree is a container literal understood by lazyrec[].""" if not maps_only: if type(tree) in (List, Tuple, Set): @@ -667,7 +667,7 @@ def transform_starred(tree, dstarred=False): tree = self.visit(tree) # lazify items if we have a literal container # we must avoid lazifying any other exprs, since a Lazy cannot be unpacked. - if is_literal_container(tree, maps_only=dstarred): + if _is_literal_container(tree, maps_only=dstarred): tree = _lazyrec(tree) return tree @@ -691,7 +691,7 @@ def transform_starred(tree, dstarred=False): # Lazy() is a strict function, takes a lambda, constructs a Lazy object # _autoref_resolve doesn't need any special handling elif (isdo(tree) or is_decorator(tree.func, "namelambda") or - any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, expanded_lazy_name) or + any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, _expanded_lazy_name) or any(isx(tree.func, s) for s in ("_autoref_resolve", "ExpandedAutorefMarker"))): # here we know the operator (.func) to be one of specific names; # don't transform it to avoid confusing lazyrec[] (important if this diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index cc8ef97c..431193c1 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -4,8 +4,7 @@ __all__ = ["let", "letseq", "letrec", "dlet", "dletseq", "dletrec", "blet", "bletseq", "bletrec", - "local", "delete", "do", "do0", - "implicit_do"] # used by some other unpythonic.syntax constructs + "local", "delete", "do", "do0"] # Let constructs are implemented as sugar around unpythonic.lispylet. # @@ -328,7 +327,7 @@ def result(): # Syntax transformers def _let(bindings, body): - return _letimpl(bindings, body, "let") + return _let_expr_impl(bindings, body, "let") def _letseq(bindings, body): if not bindings: @@ -339,9 +338,9 @@ def _letseq(bindings, body): return _let([first], _letseq(rest, body)) def _letrec(bindings, body): - return _letimpl(bindings, body, "letrec") + return _let_expr_impl(bindings, body, "letrec") -def _letimpl(bindings, body, mode): +def _let_expr_impl(bindings, body, mode): """bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn)""" assert mode in ("let", "letrec") @@ -349,7 +348,7 @@ def _letimpl(bindings, body, mode): # invocations in both bindings and body. # # But apply the implicit `do` (extra bracket syntax) first. - body = implicit_do(body) + body = _implicit_do(body) body = dyn._macro_expander.visit(body) if not bindings: # Optimize out a `let` with no bindings. The macro layer cannot trigger @@ -365,7 +364,7 @@ def _letimpl(bindings, body, mode): e = gensym("e") envset = Attribute(value=q[n[e]], attr="set", ctx=Load()) - transform = partial(letlike_transform, envname=e, lhsnames=names, rhsnames=names, setter=envset) + transform = partial(_letlike_transform, envname=e, lhsnames=names, rhsnames=names, setter=envset) if mode == "letrec": values = [transform(rhs) for rhs in values] # RHSs of bindings values = [q[h[namelambda](u[f"letrec_binding{j}_{lhs}"])(a[rhs])] @@ -384,7 +383,7 @@ def _letimpl(bindings, body, mode): newtree = q[h[letter](t[bindings], a[body], mode=u[mode])] return newtree -def letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): +def _letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): """Common transformations for let-like operations. Namely:: @@ -400,13 +399,13 @@ def letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): setter: function, (k, v) --> v, side effect to set e.k to v """ - tree = transform_envassignment(tree, lhsnames, setter) - tree = transform_name(tree, rhsnames, envname) + tree = _transform_envassignment(tree, lhsnames, setter) + tree = _transform_name(tree, rhsnames, envname) if dowrap: - tree = envwrap(tree, envname) + tree = _envwrap(tree, envname) return tree -def transform_envassignment(tree, lhsnames, envset): +def _transform_envassignment(tree, lhsnames, envset): """x << val --> e.set('x', val) (for names bound in this environment)""" # names_in_scope: according to Python's standard binding rules, see scopeanalyzer.py. # Variables defined in let envs are thus not listed in `names_in_scope`. @@ -419,7 +418,7 @@ def transform(tree, names_in_scope): return tree return scoped_transform(tree, callback=transform) -def transform_name(tree, rhsnames, envname): +def _transform_name(tree, rhsnames, envname): """x --> e.x (in load context; for names bound in this environment)""" # names_in_scope: according to Python's standard binding rules, see scopeanalyzer.py. # Variables defined in let envs are thus not listed in `names_in_scope`. @@ -451,7 +450,7 @@ def transform(tree, names_in_scope): return tree return scoped_transform(tree, callback=transform) -def envwrap(tree, envname): +def _envwrap(tree, envname): """... -> lambda e: ...""" lam = q[lambda _: a[tree]] lam.args.args[0] = arg(arg=envname) # lambda e44: ... @@ -461,25 +460,25 @@ def envwrap(tree, envname): # Syntax transformers for decorator versions, for "let over def". def _dlet(bindings, body): - return _dletimpl(bindings, body, "let", "decorate") + return _let_decorator_impl(bindings, body, "let", "decorate") def _dletseq(bindings, body): - return _dletseqimpl(bindings, body, "decorate") + return _dletseq_impl(bindings, body, "decorate") def _dletrec(bindings, body): - return _dletimpl(bindings, body, "letrec", "decorate") + return _let_decorator_impl(bindings, body, "letrec", "decorate") def _blet(bindings, body): - return _dletimpl(bindings, body, "let", "call") + return _let_decorator_impl(bindings, body, "let", "call") def _bletseq(bindings, body): - return _dletseqimpl(bindings, body, "call") + return _dletseq_impl(bindings, body, "call") def _bletrec(bindings, body): - return _dletimpl(bindings, body, "letrec", "call") + return _let_decorator_impl(bindings, body, "letrec", "call") -# Very similar to _letimpl, but perhaps more readable to keep these separate. -def _dletimpl(bindings, body, mode, kind): +# Very similar to _let_expr_impl, but perhaps more readable to keep these separate. +def _let_decorator_impl(bindings, body, mode, kind): assert mode in ("let", "letrec") assert kind in ("decorate", "call") if type(body) not in (FunctionDef, AsyncFunctionDef): @@ -498,7 +497,7 @@ def _dletimpl(bindings, body, mode, kind): e = gensym("e") envset = Attribute(value=q[n[e]], attr="set", ctx=Load()) - transform1 = partial(letlike_transform, envname=e, lhsnames=names, rhsnames=names, setter=envset) + transform1 = partial(_letlike_transform, envname=e, lhsnames=names, rhsnames=names, setter=envset) transform2 = partial(transform1, dowrap=False) if mode == "letrec": values = [transform1(rhs) for rhs in values] @@ -522,7 +521,7 @@ def _dletimpl(bindings, body, mode, kind): body.args.kw_defaults = body.args.kw_defaults + [None] return body -def _dletseqimpl(bindings, body, kind): +def _dletseq_impl(bindings, body, kind): # What we want: # # @dletseq[(x, 1), @@ -587,7 +586,7 @@ def _dletseqimpl(bindings, body, kind): body=[innerdef, ret], decorator_list=[], returns=None) # no return type annotation - return _dletseqimpl(rest, outer, kind) + return _dletseq_impl(rest, outer, kind) # ----------------------------------------------------------------------------- # Imperative code in expression position. Uses the "let" machinery. @@ -879,7 +878,7 @@ def transform(self, tree): # the name transform (RHS) should use the previous bindings, so that any # changes to bindings take effect starting from the **next** do-item. updated_names = [x for x in names + newnames if x not in deletednames] - expr = letlike_transform(expr, e, lhsnames=updated_names, rhsnames=names, setter=envset) + expr = _letlike_transform(expr, e, lhsnames=updated_names, rhsnames=names, setter=envset) expr = q[h[namelambda](u[f"do_line{j}"])(a[expr])] names = updated_names lines.append(expr) @@ -907,7 +906,7 @@ def _do0(tree): # TODO: Would be cleaner to use `do[]` as a hygienically captured macro. return _do(newtree) # do0[] is also just a do[] -def implicit_do(tree): +def _implicit_do(tree): """Allow a sequence of expressions in expression position. Apply ``do[]`` if ``tree`` is a ``List``, otherwise return ``tree`` as-is. @@ -916,7 +915,7 @@ def implicit_do(tree): [expr0, ...] - To represent a single literal list where ``implicit_do`` is in use, use an + To represent a single literal list where ``_implicit_do`` is in use, use an extra set of brackets:: [[1, 2, 3]] diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 97752649..9579c630 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -1,9 +1,5 @@ # -*- coding: utf-8 -*- -"""Detect let and do forms, and destructure them writably. - -Separate from letdo.py for dependency reasons. -Separate from util.py due to the length. -""" +"""Detect let and do forms, and destructure them writably.""" __all__ = ["where", "canonize_bindings", # used by the macro interface layer @@ -22,7 +18,7 @@ def where(*bindings): """[syntax] Only meaningful in a let[body, where((k0, v0), ...)].""" raise RuntimeError("where() is only meaningful in a let[body, where((k0, v0), ...)]") # pragma: no cover -letf_name = "letter" # must match what ``unpythonic.syntax.letdo._letimpl`` uses in its output. +letf_name = "letter" # must match what ``unpythonic.syntax.letdo._let_expr_impl`` uses in its output. dof_name = "dof" # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. currycall_name = "currycall" # output of ``unpythonic.syntax.curry`` @@ -113,9 +109,9 @@ def islet(tree, expanded=True): mode = getconstant(mode[0]) kwnames = [kw.arg for kw in tree.keywords] if "_envname" in kwnames: - return (f"{kind}_decorator", mode) # this call was generated by _dletimpl + return (f"{kind}_decorator", mode) # this call was generated by _let_decorator_impl else: - return (f"{kind}_expr", mode) # this call was generated by _letimpl + return (f"{kind}_expr", mode) # this call was generated by _let_expr_impl # dlet[(k0, v0), ...] (usually in a decorator list) deconames = ("dlet", "dletseq", "dletrec", "blet", "bletseq", "bletrec") diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 0853b0f4..40173800 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -4,7 +4,7 @@ # at macro expansion time. If you're looking for regular run-time let et al. macros, # see letdo.py. -__all__ = ["let_syntax", "abbrev"] +__all__ = ["let_syntax", "abbrev", "expr", "block"] from mcpyrate.quotes import macros, q, a # noqa: F401 @@ -16,7 +16,7 @@ from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer -from .letdo import implicit_do, _destructure_and_apply_let +from .letdo import _implicit_do, _destructure_and_apply_let from .util import eliminate_ifones # -------------------------------------------------------------------------------- @@ -132,9 +132,9 @@ def let_syntax(tree, *, args, syntax, expander, **kw): tree = expander.visit(tree) if syntax == "expr": - return _destructure_and_apply_let(tree, args, expander, let_syntax_expr, allow_call_in_name_position=True) + return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, allow_call_in_name_position=True) else: # syntax == "block": - return let_syntax_block(block_body=tree) + return _let_syntax_block(block_body=tree) @parametricmacro def abbrev(tree, *, args, syntax, expander, **kw): @@ -163,15 +163,33 @@ def abbrev(tree, *, args, syntax, expander, **kw): # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. if syntax == "expr": - return _destructure_and_apply_let(tree, args, expander, let_syntax_expr, allow_call_in_name_position=True) + return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, allow_call_in_name_position=True) else: - return let_syntax_block(block_body=tree) + return _let_syntax_block(block_body=tree) + +# TODO: convert to mcpyrate magic variable +class expr: + """[syntax] Magic identifier for ``with expr:`` inside a ``with let_syntax:``.""" + def __repr__(self): # in case one of these ends up somewhere at runtime + return "" # pragma: no cover + def __call__(self, tree, **kw): # make `expr` look like a macro + pass +expr = expr() + +# TODO: convert to mcpyrate magic variable +class block: + """[syntax] Magic identifier for ``with block:`` inside a ``with let_syntax:``.""" + def __repr__(self): # in case one of these ends up somewhere at runtime + return "" # pragma: no cover + def __call__(self, tree, **kw): # make `block` look like a macro + pass +block = block() # -------------------------------------------------------------------------------- # Syntax transformers -def let_syntax_expr(bindings, body): # bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) - body = implicit_do(body) # support the extra bracket syntax +def _let_syntax_expr(bindings, body): # bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) + body = _implicit_do(body) # support the extra bracket syntax if not bindings: # Optimize out a `let_syntax` with no bindings. The macro layer cannot trigger # this case, because our syntaxes always require at least one binding. @@ -227,7 +245,7 @@ def register_bindings(): # body0 # ... # -def let_syntax_block(block_body): +def _let_syntax_block(block_body): names_seen = set() templates = [] barenames = [] @@ -291,24 +309,6 @@ def isbinding(tree): raise SyntaxError("let_syntax: expected at least one statement beside definitions") # pragma: no cover return new_block_body -# TODO: convert to mcpyrate magic variable -class block: - """[syntax] Magic identifier for ``with block:`` inside a ``with let_syntax:``.""" - def __repr__(self): # in case one of these ends up somewhere at runtime - return "" # pragma: no cover - def __call__(self, tree, **kw): # make `block` look like a macro - pass -block = block() - -# TODO: convert to mcpyrate magic variable -class expr: - """[syntax] Magic identifier for ``with expr:`` inside a ``with let_syntax:``.""" - def __repr__(self): # in case one of these ends up somewhere at runtime - return "" # pragma: no cover - def __call__(self, tree, **kw): # make `expr` look like a macro - pass -expr = expr() - # ----------------------------------------------------------------------------- def _analyze_lhs(tree): diff --git a/unpythonic/syntax/simplelet.py b/unpythonic/syntax/simplelet.py index 5b7a07f2..f97f258c 100644 --- a/unpythonic/syntax/simplelet.py +++ b/unpythonic/syntax/simplelet.py @@ -76,10 +76,10 @@ def letseq(tree, *, args, syntax, expander, **kw): if not args: return tree first, *rest = args - body = q[a[our_letseq][t[rest]][a[tree]]] - return q[a[our_let][a[first]][a[body]]] + body = q[a[_our_letseq][t[rest]][a[tree]]] + return q[a[_our_let][a[first]][a[body]]] # for hygienic macro recursion -our_let = capture_as_macro(let) -our_letseq = capture_as_macro(letseq) +_our_let = capture_as_macro(let) +_our_letseq = capture_as_macro(letseq) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index e61fd501..9ac26f54 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -4,10 +4,10 @@ See also `unpythonic.test.fixtures` for the high-level machinery. """ -__all__ = ["isunexpandedtestmacro", "isexpandedtestmacro", "istestmacro", - "the", "test", +__all__ = ["the", "test", "test_signals", "test_raises", - "fail", "error", "warn"] + "fail", "error", "warn", + "isunexpandedtestmacro", "isexpandedtestmacro", "istestmacro"] from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 @@ -243,9 +243,9 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax == "expr": if args: raise SyntaxError("test[] in expression mode does not take macro arguments") - return test_expr(tree) + return _test_expr(tree) else: # syntax == "block": - return test_block(block_body=tree, args=args) + return _test_block(block_body=tree, args=args) @parametricmacro def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -293,9 +293,9 @@ def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax == "expr": if args: raise SyntaxError("test_signals[] in expression mode does not take macro arguments") - return test_expr_signals(tree) + return _test_expr_signals(tree) else: # syntax == "block": - return test_block_signals(block_body=tree, args=args) + return _test_block_signals(block_body=tree, args=args) @parametricmacro def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 @@ -341,9 +341,9 @@ def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax == "expr": if args: raise SyntaxError("test_raises[] in expression mode does not take macro arguments") - return test_expr_raises(tree) + return _test_expr_raises(tree) else: # syntax == "block": - return test_block_raises(block_body=tree, args=args) + return _test_block_raises(block_body=tree, args=args) def fail(tree, *, syntax, expander, **kw): # noqa: F811 """[syntax, expr] Produce a test failure, unconditionally. @@ -371,7 +371,7 @@ def fail(tree, *, syntax, expander, **kw): # noqa: F811 # Expand outside in. The ordering shouldn't matter here. # The underlying `test` machinery needs to access the expander. with dyn.let(_macro_expander=expander): - return fail_expr(tree) + return _fail_expr(tree) def error(tree, *, syntax, expander, **kw): # noqa: F811 """[syntax, expr] Produce a test error, unconditionally. @@ -391,7 +391,7 @@ def error(tree, *, syntax, expander, **kw): # noqa: F811 # Expand outside in. The ordering shouldn't matter here. # The underlying `test` machinery needs to access the expander. with dyn.let(_macro_expander=expander): - return error_expr(tree) + return _error_expr(tree) def warn(tree, *, syntax, expander, **kw): # noqa: F811 """[syntax, expr] Produce a test warning, unconditionally. @@ -415,7 +415,7 @@ def warn(tree, *, syntax, expander, **kw): # noqa: F811 # Expand outside in. The ordering shouldn't matter here. # The underlying `test` machinery needs to access the expander. with dyn.let(_macro_expander=expander): - return warn_expr(tree) + return _warn_expr(tree) # ----------------------------------------------------------------------------- # Helpers for other macros to detect uses of the ones we defined here. @@ -537,7 +537,7 @@ def unpythonic_assert(sourcecode, func, *, filename, lineno, message=None): # we send to `func` as its argument. A `the[]` is also implicitly injected # by the comparison destructuring mechanism. e = env(captured_values=[]) - testexpr = func # descriptive name for stack trace; if you change this, change also in `test_expr`. + testexpr = func # descriptive name for stack trace; if you change this, change also in `_test_expr`. mode, test_result = _observe(thunk=(lambda: testexpr(e))) # <-- run the actual expr being asserted if e.captured_values: # Convenience for testing/debugging macro code: @@ -737,20 +737,20 @@ def unpythonic_assert_raises(exctype, sourcecode, thunk, *, filename, lineno, me def _unconditional_error_expr(tree, syntaxname, marker): thetuple = q[(a[marker], a[tree])] # consider `test[tree, message]` thetuple = copy_location(thetuple, tree) - return test_expr(thetuple) + return _test_expr(thetuple) # Here `tree` is the AST for the failure message. -def fail_expr(tree): +def _fail_expr(tree): return _unconditional_error_expr(tree, "fail", q[h[_fail]]) # TODO: stash a copy of the hygienic value? -def error_expr(tree): +def _error_expr(tree): return _unconditional_error_expr(tree, "error", q[h[_error]]) -def warn_expr(tree): +def _warn_expr(tree): return _unconditional_error_expr(tree, "warn", q[h[_warn]]) # -------------------------------------------------------------------------------- # Expr variants. -def test_expr(tree): +def _test_expr(tree): # Note we want the line number *before macro expansion*, so we capture it now. ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] @@ -803,7 +803,7 @@ def test_expr(tree): # to be displayed upon test failure, using `the[...]`: # test[myconstant in the[computeset(...)]] # test[the[computeitem(...)] in expected_results_plus_uninteresting_items] -# These are used by `test_expr` and `test_block`. +# These are used by `_test_expr` and `_test_block`. def _is_important_subexpr_mark(tree): return type(tree) is Subscript and type(tree.value) is Name and tree.value.id == "the" def _record_value(envname, sourcecode, value): @@ -841,9 +841,9 @@ def transform(self, tree): return tree, transformer.collected -def test_expr_signals(tree): +def _test_expr_signals(tree): return _test_expr_signals_or_raises(tree, "test_signals", q[h[unpythonic_assert_signals]]) -def test_expr_raises(tree): +def _test_expr_raises(tree): return _test_expr_signals_or_raises(tree, "test_raises", q[h[unpythonic_assert_raises]]) def _test_expr_signals_or_raises(tree, syntaxname, asserter): @@ -882,7 +882,7 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): # The strategy is we capture the block body into a new function definition, # and then `unpythonic_assert` on that function. -def test_block(block_body, args): +def _test_block(block_body, args): if not block_body: return [] # pragma: no cover, cannot happen through the public API. first_stmt = block_body[0] @@ -917,7 +917,7 @@ def test_block(block_body, args): # End of first pass. block_body = dyn._macro_expander.visit(block_body) - testblock_function_name = gensym("test_block") + testblock_function_name = gensym("_test_block") thetest = q[(a[asserter])(u[sourcecode], n[testblock_function_name], filename=a[filename], @@ -954,9 +954,9 @@ def _insert_funcname_here_(_insert_envname_here_): return newbody -def test_block_signals(block_body, args): +def _test_block_signals(block_body, args): return _test_block_signals_or_raises(block_body, args, "test_signals", q[h[unpythonic_assert_signals]]) -def test_block_raises(block_body, args): +def _test_block_raises(block_body, args): return _test_block_signals_or_raises(block_body, args, "test_raises", q[h[unpythonic_assert_raises]]) def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): @@ -988,7 +988,7 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): # End of first pass. block_body = dyn._macro_expander.visit(block_body) - testblock_function_name = gensym("test_block") + testblock_function_name = gensym("_test_block") thetest = q[(a[asserter])(a[exctype], u[sourcecode], n[testblock_function_name], From 4ab82d398b5792548be2a3f126108dfc5d0ff542 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 03:19:32 +0300 Subject: [PATCH 072/832] be careful about the phrase "import time" --- doc/dialects.md | 2 +- doc/features.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/dialects.md b/doc/dialects.md index 6615ba91..61c77fe1 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -17,7 +17,7 @@ factorial(5000) # no crash The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). It provides the plumbing that allows to create, in Python, dialects that compile into Python -at import time. It is geared toward creating languages that extend Python +at macro expansion time. It is geared toward creating languages that extend Python and look almost like Python, but extend or modify its syntax and/or semantics. Hence *dialects*. diff --git a/doc/features.md b/doc/features.md index 6d08e379..d6159ba4 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2069,7 +2069,7 @@ Our `sym` is like a Lisp/Scheme/Racket symbol, which is essentially an [interned Our `gensym` is like the [Lisp `gensym`](http://clhs.lisp.se/Body/f_gensym.htm), and the [JavaScript `Symbol`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol). -If you're familiar with `mcpyrate`'s `gensym` or MacroPy's `gen_sym`, those mean something different. Their purpose is to create, at import time, a lexical identifier that is not already in use in the source code being compiled, whereas our `gensym` creates an uninterned symbol object for run-time use. Lisp macros use symbols to represent identifiers, hence the potential for confusion in Python, where that is not the case. (The symbols of `unpythonic` are a purely run-time abstraction.) +If you're familiar with `mcpyrate`'s `gensym` or MacroPy's `gen_sym`, those mean something different. Their purpose is to create, in a macro, a lexical identifier that is not already in use in the source code being compiled, whereas our `gensym` creates an uninterned symbol object for run-time use. Lisp macros use symbols to represent identifiers, hence the potential for confusion in Python, where that is not the case. (The symbols of `unpythonic` are a purely run-time abstraction.) If your background is in C++ or Java, you may notice the symbol abstraction is a kind of a parametric [singleton](https://en.wikipedia.org/wiki/Singleton_pattern); each symbol with the same name is a singleton (as is any gensym with the same UUID). From 1647d9fda58cf79a0a8e824b6b0b911cb31ee60d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 11:53:20 +0300 Subject: [PATCH 073/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index ac9628f4..e4f6a5d7 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -82,12 +82,20 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: Have a common base class for all `unpythonic` `ASTMarker`s? +# TODO: Upgrade anaphoric if's `it` into a `mcpyrate` magic variable that errors out at compile time when it appears in an invalid position (i.e. outside any `aif`). Basically, take the `aif` from `mcpyrate`. +# TODO: also let_syntax block, expr +# TODO: also kw() in unpythonic.syntax.prefix + +# TODO: let_syntax block, expr: syntactic consistency: change parentheses to brackets + +# TODO: Now that `unpythonic` provides dialects, update `mcpyrate` docs. # TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) # TODO: `make_dynvar` needs to be better advertised in the docs. A workflow example would also be nice. +# TODO: Have a common base class for all `unpythonic` `ASTMarker`s? + # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. # TODO: Test the q[t[...]] implementation in do0[] @@ -112,14 +120,6 @@ # TODO: Check expansion order of several macros in the same `with` statement -# TODO: grep for any remaining mentions of "macropy" - -# TODO: Upgrade anaphoric if's `it` into a `mcpyrate` magic variable that errors out at compile time when it appears in an invalid position (i.e. outside any `aif`). Basically, take the `aif` from `mcpyrate`. -# TODO: also let_syntax block, expr -# TODO: also kw() in unpythonic.syntax.prefix - -# TODO: let_syntax block, expr: syntactic consistency: change parentheses to brackets - # TODO: grep codebase for "0.15", may have some pending interface changes that don't have their own GitHub issue (e.g. parameter ordering of `unpythonic.it.window`) # TODO: ansicolor: `mcpyrate` already depends on Colorama anyway (and has a *nix-only fallback capability). @@ -127,8 +127,8 @@ # TODO: to provide our own colorizer; we can use the one from `mcpyrate`. (It would be different if regular code needed it.) # TODO: with mcpyrate, do we really need to set `ctx` in our macros? (does our macro code need it?) - -# TODO: Now that `unpythonic` provides dialects, update `mcpyrate` docs. +# - At least `lazify` and `autoref` need it. Consider calling `mcpyrate.astfixers.fix_ctx` in macros +# to generate that information when needed, and not filling `ctx` manually anywhere. # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... From 0f44168db1f5995c43f269f96c3f7a4f6375855b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 12:00:45 +0300 Subject: [PATCH 074/832] advertise example dialects in README --- README.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 84b63b9b..bb664562 100644 --- a/README.md +++ b/README.md @@ -570,13 +570,15 @@ with continuations: # enables also TCO automatically #### Unpythonic in 30 seconds: Language extensions with dialects -
Lispython: The love child of Python and Scheme. - [[docs](doc/dialects.md)] The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). We provide some example dialects based on `unpythonic`'s macro layer. -For example, what if Python had automatic tail-call optimization and an implicit return statement? Look no further: +
Lispython: The love child of Python and Scheme. + +[[docs](doc/dialects/lispython.md)] + +Python with automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas. ```python from unpythonic.dialects import dialects, Lispython # noqa: F401 @@ -589,9 +591,56 @@ def factorial(n): f(n, acc=1) assert factorial(4) == 24 factorial(5000) # no crash + +square = lambda x: x**2 +assert square(3) == 9 +assert square.__name__ == "square" ``` -
+
+
Pytkell: Because it's good to have a kell. + +[[docs](doc/dialects/pytkell.md)] + +Python with automatic currying and implicitly lazy functions. + +```python +from unpythonic.dialects import dialects, Pytkell # noqa: F401 + +from operator import add, mul +def addfirst2(a, b, c): + return a + b +assert addfirst2(1)(2)(1 / 0) == 3 + +assert tuple(scanl(add, 0, (1, 2, 3))) == (0, 1, 3, 6) +assert tuple(scanr(add, 0, (1, 2, 3))) == (0, 3, 5, 6) + +my_sum = foldl(add, 0) +my_prod = foldl(mul, 1) +my_map = lambda f: foldr(compose(cons, f), nil) +assert my_sum(range(1, 5)) == 10 +assert my_prod(range(1, 5)) == 24 +assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) +``` +
+
Listhell: It's not Lisp, it's not Python, it's not Haskell. + +[[docs](doc/dialects/listhell.md)] + +Python with prefix syntax for function calls, and automatic currying. + +```python +from unpythonic.dialects import dialects, Listhell # noqa: F401 + +from unpythonic import foldr, cons, nil, ll + +(print, "hello from Listhell") + +double = lambda x: 2 * x +my_map = lambda f: (foldr, (compose, cons, f), nil) +assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) +``` +
## Installation From ee341938a8b4d389774c5d62a456e9a19510ce87 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 12:05:27 +0300 Subject: [PATCH 075/832] update README --- README.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index bb664562..6ad3b235 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ The features of `unpythonic` are built out of, in increasing order of [magic](ht - Pure Python (e.g. batteries for `itertools`), - Macros driving a pure-Python core (`do`, `let`), - Pure macros (e.g. `continuations`, `lazify`, `dbg`). - - Whole-module AST transformations (dialects). + - Whole-module transformations a.k.a. dialects. This depends on the purpose of each feature, as well as ease-of-use considerations. See the design notes for more information. @@ -570,9 +570,7 @@ with continuations: # enables also TCO automatically #### Unpythonic in 30 seconds: Language extensions with dialects -[[docs](doc/dialects.md)] - -The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). We provide some example dialects based on `unpythonic`'s macro layer. +The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). We provide some example dialects based on `unpythonic`'s macro layer. See [documentation](doc/dialects.md).
Lispython: The love child of Python and Scheme. @@ -595,6 +593,14 @@ factorial(5000) # no crash square = lambda x: x**2 assert square(3) == 9 assert square.__name__ == "square" + +# - brackets denote a multiple-expression lambda body +# (if you want to have one expression that is a literal list, +# double the brackets: `lambda x: [[5 * x]]`) +# - local[name << value] makes an expression-local variable +g = lambda x: [local[y << 2 * x], + y + 1] +assert g(10) == 21 ```
Pytkell: Because it's good to have a kell. From 30b476d7ed1d93879ed9e2ecbc9261eca7668fa0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 12:48:12 +0300 Subject: [PATCH 076/832] aif's `it` now detects its context at compile time Also, now that we have `mcpyrate`'s `@namemacro`, we have to be more careful with parsing name nodes in the AST. It is valid e.g. for a let binding to be to a name that is actually an expanded name macro (so `mcpyrate` will wrap it in a `mcpyrate.core.Done` `ASTMarker`). --- CHANGELOG.md | 2 +- unpythonic/syntax/__init__.py | 6 +-- unpythonic/syntax/ifexprs.py | 68 ++++++++++++++++--------- unpythonic/syntax/lambdatools.py | 7 +-- unpythonic/syntax/letdo.py | 5 +- unpythonic/syntax/letdoutil.py | 24 ++++++--- unpythonic/syntax/nameutil.py | 8 +++ unpythonic/syntax/prefix.py | 7 +-- unpythonic/syntax/scopeanalyzer.py | 4 ++ unpythonic/syntax/tailtools.py | 5 +- unpythonic/syntax/tests/test_ifexprs.py | 3 +- 11 files changed, 93 insertions(+), 46 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb517636..74d0a910 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) together with a kitchen-sink language extension macro package such as `unpythonic`. - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) -- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), and `call_cc[]` (for `with continuations`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. +- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), and `it` (for `aif[]`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. - `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. - Python 3.8 and 3.9 support added. diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index e4f6a5d7..4a398078 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -82,8 +82,7 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: Upgrade anaphoric if's `it` into a `mcpyrate` magic variable that errors out at compile time when it appears in an invalid position (i.e. outside any `aif`). Basically, take the `aif` from `mcpyrate`. -# TODO: also let_syntax block, expr +# TODO: upgrade let_syntax block, expr into `mcpyrate` magic variables # TODO: also kw() in unpythonic.syntax.prefix # TODO: let_syntax block, expr: syntactic consistency: change parentheses to brackets @@ -137,7 +136,7 @@ from .autoref import autoref # noqa: F401 from .dbg import dbg # noqa: F401 from .forall import forall # noqa: F401 -from .ifexprs import aif, cond # noqa: F401 +from .ifexprs import aif, it, cond # noqa: F401 from .lambdatools import multilambda, namedlambda, f, quicklambda, envify # noqa: F401 from .lazify import lazy, lazyrec, lazify # noqa: F401 from .letdo import (let, letseq, letrec, # noqa: F401 @@ -157,7 +156,6 @@ # Re-exports - regular code from .dbg import dbgprint_block, dbgprint_expr # noqa: F401, re-export for re-use in a decorated variant. from .forall import insist, deny # noqa: F401 -from .ifexprs import it # noqa: F401 from .letdoutil import where # noqa: F401 from .lazify import force, force1 # noqa: F401 from .letsyntax import block, expr # noqa: F401 diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index b05dcd26..fe123339 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -6,9 +6,14 @@ from ast import Tuple -from mcpyrate.quotes import macros, q, a # noqa: F811, F401 +from mcpyrate.quotes import macros, q, n, a, h # noqa: F811, F401 -from .letdo import _implicit_do, _let +from mcpyrate import namemacro +from mcpyrate.expander import MacroExpander +from mcpyrate.utils import extract_bindings, NestingLevelTracker + +from .letdo import macros, let # noqa: F811, F401 +from .letdo import _implicit_do from ..dynassign import dyn @@ -39,30 +44,47 @@ def aif(tree, *, syntax, expander, **kw): if syntax != "expr": raise SyntaxError("aif is an expr macro only") + # Detect the name(s) of `it` at the use site (this accounts for as-imports) + macro_bindings = extract_bindings(dyn._macro_expander.bindings, it) + if not macro_bindings: + raise SyntaxError("The use site of `aif` must macro-import `it`, too.") + # Expand outside-in, but the implicit do[] needs the expander. with dyn.let(_macro_expander=expander): - return _aif(tree) - -def _aif(tree): - test, then, otherwise = [_implicit_do(x) for x in tree.elts] - bindings = [q[(it, a[test])]] - body = q[a[then] if it else a[otherwise]] - # TODO: we should use a hygienically captured macro here. - return _let(bindings, body) - -# TODO: `mcpyrate` has a rudimentary capability like Racket's "syntax-parameterize". -# TODO: Make `it` a name macro that errors out unless it appears inside an `aif`. -# -# We could just leave "it" undefined by default, but IDEs are happier if the -# name exists, and this also gives us a chance to provide a docstring. -class it: - """[syntax] The result of the test in an aif. - - Only meaningful inside the ``then`` and ``otherwise`` branches of an aif. + return _aif(tree, macro_bindings) + +_aif_level = NestingLevelTracker() + +def _aif(tree, bindings_of_it): + with _aif_level.changed_by(+1): + # expand any `it` inside the `aif` (thus confirming those uses are valid) + def expand_it(tree): + return MacroExpander(bindings_of_it, dyn._macro_expander.filename).visit(tree) + + name_of_it = list(bindings_of_it.keys())[0] + expanded_it = expand_it(q[n[name_of_it]]) + + tree = expand_it(tree) + + test, then, otherwise = [_implicit_do(x) for x in tree.elts] + let_bindings = q[(a[expanded_it], a[test])] + let_body = q[a[then] if a[expanded_it] else a[otherwise]] + # We use a hygienic macro reference to `let[]` in the output, + # so that the expander can expand it later. + return q[h[let][a[let_bindings]][a[let_body]]] + +@namemacro +def it(tree, *, syntax, **kw): + """[syntax, name] The `it` of an anaphoric if. + + Inside an `aif` body, evaluates to the value of the test result. + Anywhere else, is considered a syntax error. """ - def __repr__(self): # pragma: no cover, we have a repr just in case one of these ends up somewhere at runtime. - return "" -it = it() + if syntax != "name": + raise SyntaxError("`it` is a name macro only") + if _aif_level.value < 1: + raise SyntaxError("`it` may only appear within an `aif[...]`") + return tree # -------------------------------------------------------------------------------- diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 27468a76..aa04fbab 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -28,6 +28,7 @@ from .astcompat import getconstant, Str, NamedExpr from .letdo import _do from .letdoutil import islet, isenvassign, UnexpandedLetView, UnexpandedEnvAssignView, ExpandedDoView +from .nameutil import getname from .util import (is_decorated_lambda, isx, has_deco, destructure_decorated_lambda, detect_lambda) @@ -304,7 +305,7 @@ def transform(self, tree): if islet(tree, expanded=False): # let bindings view = UnexpandedLetView(tree) for b in view.bindings: - b.elts[1], thelambda, match = nameit(b.elts[0].id, b.elts[1]) + b.elts[1], thelambda, match = nameit(getname(b.elts[0]), b.elts[1]) if match: thelambda.body = self.visit(thelambda.body) else: @@ -321,14 +322,14 @@ def transform(self, tree): view.value = self.visit(view.value) return tree elif issingleassign(tree): # f = lambda ...: ... - tree.value, thelambda, match = nameit(tree.targets[0].id, tree.value) + tree.value, thelambda, match = nameit(getname(tree.targets[0]), tree.value) if match: thelambda.body = self.visit(thelambda.body) else: tree.value = self.visit(tree.value) return tree elif type(tree) is NamedExpr: # f := lambda ...: ... (Python 3.8+, added in unpythonic 0.15) - tree.value, thelambda, match = nameit(tree.target.id, tree.value) + tree.value, thelambda, match = nameit(getname(tree.target), tree.value) if match: thelambda.body = self.visit(thelambda.body) else: diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 431193c1..ee7e7314 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -43,6 +43,7 @@ from .letdoutil import (isenvassign, UnexpandedEnvAssignView, UnexpandedLetView, canonize_bindings) +from .nameutil import getname from .scopeanalyzer import scoped_transform # -------------------------------------------------------------------------------- @@ -359,7 +360,7 @@ def _let_expr_impl(bindings, body, mode): bindings = dyn._macro_expander.visit(bindings) names, values = zip(*[b.elts for b in bindings]) # --> (k1, ..., kn), (v1, ..., vn) - names = [k.id for k in names] # any duplicates will be caught by env at run-time + names = [getname(k, accept_attr=False) for k in names] # any duplicates will be caught by env at run-time e = gensym("e") envset = Attribute(value=q[n[e]], attr="set", ctx=Load()) @@ -492,7 +493,7 @@ def _let_decorator_impl(bindings, body, mode, kind): bindings = dyn._macro_expander.visit(bindings) names, values = zip(*[b.elts for b in bindings]) # --> (k1, ..., kn), (v1, ..., vn) - names = [k.id for k in names] # any duplicates will be caught by env at run-time + names = [getname(k, accept_attr=False) for k in names] # any duplicates will be caught by env at run-time e = gensym("e") envset = Attribute(value=q[n[e]], attr="set", ctx=Load()) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 9579c630..c968f937 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -11,8 +11,10 @@ Tuple, List, Constant, BinOp, LShift, Lambda) import sys +from mcpyrate.core import Done + from .astcompat import getconstant, Str -from .nameutil import isx +from .nameutil import isx, getname def where(*bindings): """[syntax] Only meaningful in a let[body, where((k0, v0), ...)].""" @@ -41,9 +43,12 @@ def canonize_bindings(elts, allow_call_in_name_position=False): # public as of in the call, the "function" is the template name, and the positional "parameters" are the template parameters (which may then appear in the template body). """ + def isname(x): + # The `Done` may be produced by expanded `@namemacro`s. + return type(x) is Name or (isinstance(x, Done) and isname(x.body)) def iskey(x): - return ((type(x) is Name) or - (allow_call_in_name_position and type(x) is Call and type(x.func) is Name)) + return (isname(x) or + (allow_call_in_name_position and type(x) is Call and isname(x.func))) if len(elts) == 2 and iskey(elts[0]): return [Tuple(elts=elts)] # TODO: `mcpyrate`: just `q[t[elts]]`? if all((type(b) is Tuple and len(b.elts) == 2 and iskey(b.elts[0])) for b in elts): @@ -56,7 +61,10 @@ def isenvassign(tree): The only way this differs from a general left-shift is that the LHS must be an ``ast.Name``. """ - return type(tree) is BinOp and type(tree.op) is LShift and type(tree.left) is Name + if not (type(tree) is BinOp and type(tree.op) is LShift): + return False + # The `Done` may be produced by expanded `@namemacro`s. + return type(tree.left) is Name or (isinstance(tree.left, Done) and type(tree.body) is Name) def islet(tree, expanded=True): """Test whether tree is a ``let[]``, ``letseq[]``, ``letrec[]``, @@ -260,11 +268,15 @@ def __init__(self, tree): self._tree = tree def _getname(self): - return self._tree.left.id + return getname(self._tree.left, accept_attr=False) def _setname(self, newname): if not isinstance(newname, str): raise TypeError(f"expected str for new name, got {type(newname)} with value {repr(newname)}") - self._tree.left.id = newname + # The `Done` may be produced by expanded `@namemacro`s. + if isinstance(self._tree.left, Done): + self._tree.left.body.id = newname + else: + self._tree.left.id = newname name = property(fget=_getname, fset=_setname, doc="The name of the assigned var, as an str. Writable.") def _getvalue(self): diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index b8e2eadc..c2307e4b 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -9,6 +9,7 @@ from ast import Name, Attribute +from mcpyrate.core import Done from mcpyrate.quotes import is_captured_value def isx(tree, x, accept_attr=True): @@ -25,11 +26,16 @@ def isx(tree, x, accept_attr=True): - bare name ``x`` + - the name ``x`` inside a `mcpyrate.core.Done`, which may be produced + by expanded `@namemacro`s + - the name ``x`` inside a `mcpyrate` hygienic capture, which may be inserted during macro expansion - ``x`` as an attribute (if ``accept_attr=True``) """ + if isinstance(tree, Done): + return isx(tree.body, x, accept_attr=accept_attr) # Here hygienic captures only come from `unpythonic.syntax` (unless there are # also user-defined macros), and we use from-imports and bare names for anything # `q[h[]]`'d; but any references that appear explicitly in the user code may use @@ -79,6 +85,8 @@ def getname(tree, accept_attr=True): If no match on ``tree``, return ``None``. """ + if isinstance(tree, Done): + return getname(tree.body, accept_attr=accept_attr) if type(tree) is Name: return tree.id key = is_captured_value(tree) # AST -> (name, frozen_value) or False diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 17dcb082..9f52a797 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -15,6 +15,7 @@ from mcpyrate.walkers import ASTTransformer from .letdoutil import islet, isdo, UnexpandedLetView, UnexpandedDoView +from .nameutil import getname from ..it import flatmap, rev, uniqify @@ -124,9 +125,9 @@ def kw(**kwargs): # -------------------------------------------------------------------------------- def _prefix(block_body): - isquote = lambda tree: type(tree) is Name and tree.id == "q" - isunquote = lambda tree: type(tree) is Name and tree.id == "u" - iskwargs = lambda tree: type(tree) is Call and type(tree.func) is Name and tree.func.id == "kw" + isquote = lambda tree: getname(tree, accept_attr=False) == "q" + isunquote = lambda tree: getname(tree, accept_attr=False) == "u" + iskwargs = lambda tree: type(tree) is Call and getname(tree.func, accept_attr=False) == "kw" class PrefixTransformer(ASTTransformer): def transform(self, tree): diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 8efe0a8c..eafea116 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -77,6 +77,7 @@ Import, ImportFrom, Try, ListComp, SetComp, GeneratorExp, DictComp, Store, Del, Global, Nonlocal) +from mcpyrate.core import Done from mcpyrate.walkers import ASTTransformer, ASTVisitor from ..it import uniqify @@ -281,6 +282,9 @@ def examine(self, tree): for g in tree.generators: if type(g.target) is Name: targetnames.append(g.target.id) + # The `Done` may be produced by expanded `@namemacro`s. + elif isinstance(g.target, Done) and type(g.target.body) is Name: + targetnames.append(g.target.body.id) elif type(g.target) is Tuple: class NamesCollector(ASTVisitor): def examine(self, tree): diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index a222d3aa..df4fbaab 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -32,7 +32,7 @@ has_tco, sort_lambda_decorators, suggest_decorator_index, ExpandedContinuationsMarker, wrapwith, isexpandedmacromarker) from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView -from .ifexprs import _aif +from .ifexprs import _aif, it as aif_it from ..dynassign import dyn from ..it import uniqify @@ -1257,7 +1257,8 @@ def transform(tree): # or(data1, ..., datan, tail) --> it if any(others) else tail tree = _aif(Tuple(elts=[op_of_others, transform_data(Name(id="it")), - transform(tree.values[-1])])) # tail-call item + transform(tree.values[-1])]), # tail-call item + {'it': aif_it}) elif type(tree.op) is And: # and(data1, ..., datan, tail) --> tail if all(others) else False fal = q[False] diff --git a/unpythonic/syntax/tests/test_ifexprs.py b/unpythonic/syntax/tests/test_ifexprs.py index 439af3d0..6da31068 100644 --- a/unpythonic/syntax/tests/test_ifexprs.py +++ b/unpythonic/syntax/tests/test_ifexprs.py @@ -4,8 +4,7 @@ from ...syntax import macros, test # noqa: F401 from ...test.fixtures import session, testset -from ...syntax import macros, aif, cond, local # noqa: F401, F811 -from ...syntax import it +from ...syntax import macros, aif, it, cond, local # noqa: F401, F811 def runtests(): with testset("aif (anaphoric if, you're `it`!)"): From 89c4e918681a7e70a74d50a70527b3626bb7dd42 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:01:35 +0300 Subject: [PATCH 077/832] make `aif` and `letseq` macros expand into hygienic macro invocations --- unpythonic/syntax/ifexprs.py | 2 +- unpythonic/syntax/letdo.py | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index fe123339..5551e40e 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -7,12 +7,12 @@ from ast import Tuple from mcpyrate.quotes import macros, q, n, a, h # noqa: F811, F401 +from .letdo import macros, let # noqa: F811, F401 from mcpyrate import namemacro from mcpyrate.expander import MacroExpander from mcpyrate.utils import extract_bindings, NestingLevelTracker -from .letdo import macros, let # noqa: F811, F401 from .letdo import _implicit_do from ..dynassign import dyn diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index ee7e7314..14882c32 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -32,7 +32,7 @@ from mcpyrate import gensym, parametricmacro from mcpyrate.markers import ASTMarker -from mcpyrate.quotes import is_captured_value +from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer @@ -330,13 +330,21 @@ def result(): def _let(bindings, body): return _let_expr_impl(bindings, body, "let") +_our_let = capture_as_macro(let) +_our_letseq = capture_as_macro(letseq) def _letseq(bindings, body): if not bindings: return body first, *rest = bindings - # TODO: Could just return hygienic macro invocations. - # TODO: See `unpythonic.syntax.simplelet` for how to do it. - return _let([first], _letseq(rest, body)) + # We use hygienic macro references in the output, + # so that the expander can expand them later. + if rest: + nested_letseq = q[a[_our_letseq][t[rest]][a[body]]] + return q[a[_our_let][a[first]][a[nested_letseq]]] + else: + # We must do this optimization (no letseq with empty bindings) + # because empty bindings confuse `_destructure_and_apply_let`. + return q[a[_our_let][a[first]][a[body]]] def _letrec(bindings, body): return _let_expr_impl(bindings, body, "letrec") From 4df0ebc0ff3bb2532ea6ec6ca62f16de131c07a0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:02:09 +0300 Subject: [PATCH 078/832] update doc on `aif` --- doc/macros.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index e551522f..ee7ffe4d 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1545,14 +1545,14 @@ To denote a single expression that is a literal list, use an extra set of bracke This is mainly of interest as a point of [comparison with Racket](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/aif.rkt); ``aif`` is about the simplest macro that relies on either the lack of hygiene or breaking thereof. ```python -from unpythonic.syntax import macros, aif +from unpythonic.syntax import macros, aif, it aif[2*21, print(f"it is {it}"), print("it is falsey")] ``` -Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` refers to the test result while (lexically) inside the ``aif``, and does not exist outside the ``aif``. +Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the ``aif``, and does not exist outside the ``aif``. Any part of ``aif`` may have multiple expressions by surrounding it with brackets (implicit ``do[]``): From 0645aea63d700f044f585e8b8962318900e389bb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:11:56 +0300 Subject: [PATCH 079/832] make `do0` macro expand into hygienic macro invocations --- unpythonic/syntax/letdo.py | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 14882c32..2d077c2c 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -897,23 +897,18 @@ def transform(self, tree): thecall.args = lines return thecall +_our_local = capture_as_macro(local) +_our_do = capture_as_macro(do) def _do0(tree): if type(tree) not in (Tuple, List): raise SyntaxError("do0 body: expected a sequence of comma-separated expressions") # pragma: no cover elts = tree.elts - newelts = [] - # TODO: Would be cleaner to use `local[]` as a hygienically captured macro. - # Now we call the syntax transformer directly, and splice in the returned AST. - with _do_level.changed_by(+1): # it's alright, `local[]`, we're inside a `do0[]`. - firstexpr = elts[0] - firstexpr = dyn._macro_expander.visit(firstexpr) - thelocalexpr = q[_do0_result << a[firstexpr]] # noqa: F821, the local[] defines it inside the do[]. - newelts.append(q[a[_local(thelocalexpr)]]) - newelts.extend(elts[1:]) - newelts.append(q[_do0_result]) # noqa: F821 - newtree = q[t[newelts]] - # TODO: Would be cleaner to use `do[]` as a hygienically captured macro. - return _do(newtree) # do0[] is also just a do[] + # Use `local[]` as a hygienically captured macro. + newelts = [q[a[_our_local][_do0_result << a[elts[0]]]], # noqa: F821, local[] defines it inside the do[]. + *elts[1:], + q[_do0_result]] # noqa: F821 + # Use `do[]` as a hygienically captured macro. + return q[a[_our_do][t[newelts]]] # do0[] is also just a do[] def _implicit_do(tree): """Allow a sequence of expressions in expression position. From e572457e50dccfa06d26d79a78d9495e0504fc32 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:31:07 +0300 Subject: [PATCH 080/832] `lazyrec` macro: expand to hygienic macro references to `lazy` --- unpythonic/syntax/lazify.py | 38 +++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 5ad380db..55037774 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -8,7 +8,7 @@ from mcpyrate.quotes import macros, q, a, h # noqa: F401 -from mcpyrate.quotes import is_captured_value +from mcpyrate.quotes import capture_as_macro, is_captured_macro, is_captured_value, lookup_macro from mcpyrate.walkers import ASTTransformer from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, @@ -487,7 +487,27 @@ def _lazy(tree): _unexpanded_lazy_name = "lazy" _expanded_lazy_name = "Lazy" +_our_lazy = capture_as_macro(lazy) def _lazyrec(tree): + def is_unexpanded_lazy(tree): + if not type(tree) is Subscript: + return False + if isx(tree.value, _unexpanded_lazy_name): + return True + + # hygienic captures and as-imports + key = is_captured_macro(tree) + if key: + name_node = lookup_macro(key) + elif type(tree) is Name: + name_node = tree + else: + return False + macrofunction = dyn._macro_expander.isbound(name_node.id) + if macrofunction is lazy: # does the macro binding (in the current expander) point to our macro definition? + return True + return False + # This helper doesn't need to recurse, so we don't need `ASTTransformer` here. def transform(tree): if type(tree) in (Tuple, List, Set): @@ -497,17 +517,14 @@ def transform(tree): elif type(tree) is Call and any(isx(tree.func, ctor) for ctor in _ctorcalls_all): p, k = _ctor_handling_modes[getname(tree.func)] lazify_ctorcall(tree, p, k) - elif type(tree) is Subscript and isx(tree.value, _unexpanded_lazy_name): + elif is_unexpanded_lazy(tree): pass elif type(tree) is Call and isx(tree.func, _expanded_lazy_name): pass else: - # mcpyrate supports hygienic macro capture, so we can just splice unexpanded - # (but hygienically unquoted) `lazy` invocations here. - # TODO: Doing so renames the macro, so detection needs to be adjusted. - # TODO: It must also be bound in the current expander for hygienic macro capture to work. - # tree = q[h[lazy][a[tree]]] - tree = _lazy(tree) + # `mcpyrate` supports hygienic macro capture, so we can just splice + # hygienic `lazy` invocations here. + tree = q[a[_our_lazy][a[tree]]] return tree def lazify_ctorcall(tree, positionals="all", keywords="all"): @@ -579,6 +596,9 @@ def _lazify(body): # first pass, outside-in userlambdas = detect_lambda(body) + # Expand any inner macro invocations. Particularly, this expands away any `lazyrec[]` and `lazy[]` + # so they become easier to work with. We also know that after this, any `Subscript` is really a + # subscripting operation and not a macro invocation. body = dyn._macro_expander.visit(body) # second pass, inside-out @@ -732,6 +752,8 @@ def transform_starred(tree, dstarred=False): tree = mycall return tree + # NOTE: We must expand all inner macro invocations before we hit this, or we'll produce nonsense. + # Hence it is easiest to have `lazify` expand inside-out. elif type(tree) is Subscript: # force only accessed part of obj[...] self.withstate(tree.slice, forcing_mode="full") tree.slice = self.visit(tree.slice) From bf52c34796c8835039145ab5cff3fd3cdd42bdc1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:32:47 +0300 Subject: [PATCH 081/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 7 ------- 1 file changed, 7 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 4a398078..3fe3c0a6 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -71,10 +71,6 @@ # # There are further cleanups of the macro layer possible with `mcpyrate`. For example: # -# - Quasiquotes no longer auto-expand macros in the quoted code. `letseq` could use hygienic *macro* -# capture and just return an unexpanded `let` and another `letseq` (with one fewer binding), -# similarly to how Racket implements `let*`. See `unpythonic.syntax.simplelet` for a demo. -# # - Many macros could perhaps run in the outside-in pass. Some need a redesign for their AST analysis, # but much of that has been sufficiently abstracted (e.g. `unpythonic.syntax.letdoutil`) so that this # is mainly a case of carefully changing the analysis mode at all appropriate use sites. @@ -102,9 +98,6 @@ # TODO: macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" -# TODO: Some macros look up others; convert lookups to mcpyrate style (accounting for as-imports) -# TODO: or hygienic macro references (`h[...]`), as appropriate. - # TODO: `isx` and `getname` from `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead # TODO: `mcpyrate` does not auto-expand macros in quasiquoted code. From 9f494274e78ac177500ceab403ccb79516c1f727 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:46:47 +0300 Subject: [PATCH 082/832] remove done TODO --- unpythonic/syntax/letdo.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 2d077c2c..7a45f7e6 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -808,8 +808,6 @@ class UnpythonicDoLocalMarker(UnpythonicLetDoMarker): class UnpythonicDoDeleteMarker(UnpythonicLetDoMarker): """AST marker for local variable deletion in a `do` context.""" -# TODO: fail-fast: promote `local[]`/`delete[]` usage errors to compile-time errors -# TODO: (doesn't currently work e.g. for `let` with an implicit do (extra bracket notation)) def _local(tree): # syntax transformer if _do_level.value < 1: raise SyntaxError("local[] is only valid within a do[] or do0[]") # pragma: no cover From 6954993ad00079be306341117d9d31187bd104b9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:46:54 +0300 Subject: [PATCH 083/832] update comment --- unpythonic/syntax/letdo.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 7a45f7e6..72853fa1 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -827,6 +827,9 @@ def _do(tree): # TODO: only the `local` and `delete` transformers to it - grabbing them from the current expander's # TODO: bindings to respect as-imports. (Expander instances are cheap in `mcpyrate`.) # TODO: Grep the `unpythonic` codebase (and `mcpyrate` demos) for `MacroExpander` to see how. + # + # TODO: We have to be careful with nested `do`, though - some local definitions may belong to + # TODO: nested invocations. So perhaps we need to expand `do`, `do0`, `local` and `delete` here. with _do_level.changed_by(+1): tree = dyn._macro_expander.visit(tree) From c5fa035bcbe7b0e0e45c00ca27c5fb1e441416d0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:47:25 +0300 Subject: [PATCH 084/832] add comment --- unpythonic/syntax/autoref.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index a62fcc54..127eb691 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -177,6 +177,8 @@ def add_to_resolver_list(tree, objnode): # x --> the autoref code above. def makeautoreference(tree): + # We don't need to care about `Done` markers from expanded `@namemacro`s + # because the transformer that calls this function recurses into them. assert type(tree) is Name and (type(tree.ctx) is Load or not tree.ctx) newtree = q[(lambda __ar_: __ar_[1] if __ar_[0] else a[tree])(h[_autoref_resolve]((n[o], u[tree.id])))] our_lambda_argname = gensym("_ar") From f8293d4ed2bfd0066a2384de151ae9d6feaed19a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:47:38 +0300 Subject: [PATCH 085/832] spelling --- unpythonic/syntax/lambdatools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index aa04fbab..479d0e65 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -492,7 +492,7 @@ def isourupdate(thecall): # transform references to currently active bindings elif type(tree) is Name and tree.id in bindings.keys(): # We must be careful to preserve the Load/Store/Del context of the name. - # The default lets mcpyrate fix it later. + # The default lets `mcpyrate` fix it later. ctx = tree.ctx if hasattr(tree, "ctx") else None out = deepcopy(bindings[tree.id]) out.ctx = ctx From c4f06ead8b06820480a22ee664c4288056fbe0d0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 13:48:00 +0300 Subject: [PATCH 086/832] AST `ctx` handling improvements - Don't bother manually populating `ctx`, since `mcpyrate` will fill it in. - In macros that require correctly populated `ctx`, call `mcpyrate.astfixers.fix_ctx` before proceeding with analysis. --- unpythonic/syntax/__init__.py | 4 ---- unpythonic/syntax/autoref.py | 12 +++++++++--- unpythonic/syntax/lazify.py | 4 ++++ unpythonic/syntax/letdo.py | 10 ++++------ 4 files changed, 17 insertions(+), 13 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 3fe3c0a6..3efc1b85 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -118,10 +118,6 @@ # TODO: `unpythonic` only needs the colorizer in the *macro-enabled* test framework; so we don't really need # TODO: to provide our own colorizer; we can use the one from `mcpyrate`. (It would be different if regular code needed it.) -# TODO: with mcpyrate, do we really need to set `ctx` in our macros? (does our macro code need it?) -# - At least `lazify` and `autoref` need it. Consider calling `mcpyrate.astfixers.fix_ctx` in macros -# to generate that information when needed, and not filling `ctx` manually anywhere. - # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... # Re-exports - macro interfaces diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 127eb691..cfad6d1f 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -9,6 +9,7 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym, parametricmacro +from mcpyrate.astfixers import fix_ctx from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer @@ -18,6 +19,7 @@ from .letdoutil import isdo, islet, ExpandedDoView, ExpandedLetView from .testingtools import _test_function_names +from ..dynassign import dyn from ..lazyutil import force1, passthrough_lazy_args # with autoref[o]: @@ -126,9 +128,8 @@ def autoref(tree, *, args, syntax, expander, **kw): target = kw.get("optional_vars", None) - tree = expander.visit(tree) - - return _autoref(block_body=tree, args=args, asname=target) + with dyn.let(_macro_expander=expander): + return _autoref(block_body=tree, args=args, asname=target) # -------------------------------------------------------------------------------- @@ -146,6 +147,11 @@ def _autoref(block_body, args, asname): if not block_body: raise SyntaxError("expected at least one statement inside the 'with autoref' block") # pragma: no cover + block_body = dyn._macro_expander.visit(block_body) + + # `autoref`'s analyzer needs the `ctx` attributes in `tree` to be filled in correctly. + block_body = fix_ctx(block_body, copy_seen_nodes=False) # TODO: or maybe copy seen nodes? + o = asname.id if asname else gensym("_o") # Python itself guarantees asname to be a bare Name. # TODO: We can't use `unpythonic.syntax.util.isexpandedmacromarker` here, because it diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 55037774..e1ab3a61 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -8,6 +8,7 @@ from mcpyrate.quotes import macros, q, a, h # noqa: F401 +from mcpyrate.astfixers import fix_ctx from mcpyrate.quotes import capture_as_macro, is_captured_macro, is_captured_value, lookup_macro from mcpyrate.walkers import ASTTransformer @@ -601,6 +602,9 @@ def _lazify(body): # subscripting operation and not a macro invocation. body = dyn._macro_expander.visit(body) + # `lazify`'s analyzer needs the `ctx` attributes in `tree` to be filled in correctly. + body = fix_ctx(body, copy_seen_nodes=False) # TODO: or maybe copy seen nodes? + # second pass, inside-out class LazifyTransformer(ASTTransformer): def transform(self, tree): diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 72853fa1..d03faee8 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -20,7 +20,7 @@ from functools import partial -from ast import (Name, Attribute, +from ast import (Name, Tuple, List, FunctionDef, Return, AsyncFunctionDef, @@ -371,7 +371,7 @@ def _let_expr_impl(bindings, body, mode): names = [getname(k, accept_attr=False) for k in names] # any duplicates will be caught by env at run-time e = gensym("e") - envset = Attribute(value=q[n[e]], attr="set", ctx=Load()) + envset = q[n[f"{e}.set"]] transform = partial(_letlike_transform, envname=e, lhsnames=names, rhsnames=names, setter=envset) if mode == "letrec": @@ -454,7 +454,7 @@ def transform(tree, names_in_scope): return tree attr_node = q[n[f"{envname}.{tree.id}"]] if hasctx: - attr_node.ctx = tree.ctx # let mcpyrate fix it if needed + attr_node.ctx = tree.ctx return attr_node return tree return scoped_transform(tree, callback=transform) @@ -504,7 +504,7 @@ def _let_decorator_impl(bindings, body, mode, kind): names = [getname(k, accept_attr=False) for k in names] # any duplicates will be caught by env at run-time e = gensym("e") - envset = Attribute(value=q[n[e]], attr="set", ctx=Load()) + envset = q[n[f"{e}.set"]] transform1 = partial(_letlike_transform, envname=e, lhsnames=names, rhsnames=names, setter=envset) transform2 = partial(transform1, dowrap=False) @@ -835,9 +835,7 @@ def _do(tree): e = gensym("e") envset = q[n[f"{e}._set"]] # use internal _set to allow new definitions - envset.ctx = Load() envdel = q[n[f"{e}.pop"]] - envdel.ctx = Load() def find_localdefs(tree): class LocaldefCollector(ASTTransformer): From 6e1f92300c0aca86f83b843cedc1afe3af10434b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 14:39:46 +0300 Subject: [PATCH 087/832] update comments --- unpythonic/syntax/lazify.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index e1ab3a61..b9432bc5 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -675,6 +675,10 @@ def f(tree): return tree elif type(tree) is Call: + # We don't need to expand in the output of `_lazyrec`, + # because we don't recurse further into the args of the call, + # so the `lazify` transformer never sees the confusing `Subscript` + # instances that are actually macro invocations for `lazy[]`. def transform_arg(tree): # add any needed force() invocations inside the tree, # but leave the top level of simple references untouched. @@ -722,7 +726,7 @@ def transform_starred(tree, dstarred=False): # is an inner call in the arglist of an outer, lazy call, since it # must see any container constructor calls that appear in the args) # - # TODO: correct forcing mode for `rec`? We shouldn't need to forcibly use "full", + # TODO: correct forcing mode for recursion? We shouldn't need to forcibly use "full", # since maybe_force_args() already fully forces any remaining promises # in the args when calling a strict function. tree.args = self.visit(tree.args) From 835aaccb5bfcebc0216c4749ff8d473775462da0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 14:56:35 +0300 Subject: [PATCH 088/832] bump mcpyrate to 3.4.0, should fix CI on pypy-3.6 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 8f75191e..b1112c4f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -mcpyrate>=3.3.0 +mcpyrate>=3.4.0 sympy>=1.4 From dc33bf23e0b542f89c38992cfa4e3bd88ae4b9c7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 15:08:41 +0300 Subject: [PATCH 089/832] change wording to "PRs welcome"; link CONTRIBUTING.md separately. --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6ad3b235..a5f77847 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f ![100% Python](https://img.shields.io/github/languages/top/Technologicat/unpythonic) ![supported language versions](https://img.shields.io/pypi/pyversions/unpythonic) ![supported implementations](https://img.shields.io/pypi/implementation/unpythonic) ![CI status](https://img.shields.io/github/workflow/status/Technologicat/unpythonic/Python%20package) [![codecov](https://codecov.io/gh/Technologicat/unpythonic/branch/master/graph/badge.svg)](https://codecov.io/gh/Technologicat/unpythonic) ![version on PyPI](https://img.shields.io/pypi/v/unpythonic) ![PyPI package format](https://img.shields.io/pypi/format/unpythonic) ![dependency status](https://img.shields.io/librariesio/github/Technologicat/unpythonic) -![license: BSD](https://img.shields.io/pypi/l/unpythonic) ![open issues](https://img.shields.io/github/issues/Technologicat/unpythonic) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](CONTRIBUTING.md) +![license: BSD](https://img.shields.io/pypi/l/unpythonic) ![open issues](https://img.shields.io/github/issues/Technologicat/unpythonic) [![PRs welcome](https://img.shields.io/badge/PRs-welcome-brightgreen)](http://makeapullrequest.com/) *Some hypertext features of this README, such as local links to detailed documentation, and expandable example highlights, are not supported when viewed on PyPI; [view on GitHub](https://github.com/Technologicat/unpythonic) to have those work properly.* @@ -39,7 +39,8 @@ The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (langua [Syntactic macro feature set](doc/macros.md) [Examples of creating dialects using `mcpyrate`](doc/dialects.md): Python the way you want it. [REPL server](doc/repl.md): interactively hot-patch your running Python program. -[Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. +[Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. +[Contribution guidelines](CONTRIBUTING.md): for understanding the codebase, or if you're interested in making a code or documentation PR. The features of `unpythonic` are built out of, in increasing order of [magic](https://macropy3.readthedocs.io/en/latest/discussion.html#levels-of-magic): From 9b4259e19f6583ad95c4d4e60302f1c33c923e34 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 16:03:42 +0300 Subject: [PATCH 090/832] update shading of box in Lispython mascot --- doc/dialects/lis.png | Bin 36164 -> 34742 bytes doc/dialects/lis.svg | 40 ++++++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/doc/dialects/lis.png b/doc/dialects/lis.png index 1e722eadd279e55759bfef2b96367ff1bb905369..600b5d9998e29cc6664dfbae317c3ab511e502ef 100644 GIT binary patch delta 16627 zcmV*G+xt1N=?Gm%dXa)Mo?aj8U)svozWTM4e_puAfO04mFea=2wO?>4IiLD2Hn$g)g#cQ>V_$>oR3H8(UGbx8hB zWJRTR^4UE9;`cdY`Lcxf^z`)bi~srs*ZufxEp-Jdf~pb5NfzRRxNo6rk;?ddIpx5wb>b6~hQF zeTuVLt;50q+uPeIDH)yEFjJ({#}{9Do9RpLMNt&=MhD$}e*AuaOyxX`@Bi@oG|Vq& z(dE-MQK|}TdUGFt`P$16(D5Iizl(FuIVU2Tqme3cgb6Ocf6#`Tq=qvsgunzx0mjIm7tnOzFk39REla&XZ4!3 zaqoHd*=Je3=54Nj;2c!6?^gcteSUG>6I^}CCt2~&=eh903o#fBV=f4iL3o9lU#LZD zS+k_WRaZp_e=g%V)oc)4rB0_)i;D`zzA_2eZJH-c{$P;Ji8hUdC;iF4{gd*h%aB3} zum0_u1cL!8CZ56bFTEM_Io^HuUGD$>{oL}<#S~OJP*s)IeVzREzL)vcFMrLAH{OUL z3FCMw{S3dH_B@TyEHOB5IwG}dIETv!*&sNl)nZYje>dtgNvjnW1L1heXh+Nw1w~O< zy>=tTRr3f)5|yPk+7E9*r?=5NAmj7-hE4pIEnE1HJMZAC?=EB7vib;jd+grVxcK6W zIr-$1bMj(DJbeexFtfxEsp&|cJVzu+PF|wxl0h~I&Tg|>hEYQ%sTjhm)IX2vP#SL=UBXOCf!HZAuB4*%B8%uYBj2=Z5uuF%ro5lnNM@h9rHQ$ z+UY2&f~qQPcySj8Hy-2mFWr&$*o=j+NPTDJP9U~OO$TAAgO(jDBV=RIV6%*H^i$OI zIh<|q;g&C*&+Bi#pSCS-EW6mgefv0aL`yYae_2_{q)C(FdMH2k)C-hMzn4%*rTg$I z&N$-%9)0RnWJRU4VHq#JazCmP;!l5lgk$YTx#8DmGV|;S5zgl7@$&dREBW#rcaoo< zpA&8^%W}lE{DN~A^V|!oW3E!u!AFhY3n<`>WYF}R; ze~&))D33kyD4|e@{Aw!#0)5B4$U&JIv!}DuZqwoJ@4#d&X7|1}w(t4@i*A_0b=NGymS>4@M_HD6__mjr+BBWZFTZ?P zzp`mJ?Ck7}jC;OQJH04Bf3yTlr%nBkf3BG^y=27!bL8Bn>RB3%DU&7u;Bt8YvbNi> zapOkry8Fu*D-`D5lSkPc+n~dz4xOOGeMF+|!yaDv@JSx}!|w}+znXw_qX^ASo*iYRoBP7EmuscTVD{mi(-*&w)4;}(?CPdYPcLOl~2YI%Qy zriXYuo{U}1*RNmCm+tr?ldiWi;bVE|L>-93E&h%=6Z!R4W?ZR5paD;t#6W8RB>;jT zVl?Zq)Ky88P1xci>3(DZ2wwdYwPdX=iGNC6qwp&&{C%uWM^fAv-bvxrVq z$*VSzUt>il4sk(I)S(k~TJmNAti^tO?q2jJCjplezbi=Z5fAVFWe=O4-_K>2UB+E^ zekJATwRlK0@nH^UH0tqs1Kf1&RWYwK7*e0|fIpHbGQESmqL2-O>mP7?hJB%`DnUs~ z+X?gL>#pLze)mV34{1t$e@90Lb#+6|3aRKJk3Iek=l&Sq zfQPfrK9eUN{~i-2Oo)s}t`XLC-EJS3U33mpntxBf3C=%8FUfI4#D+$ zJYGRnho^$_`TS`UoWY>y;a~lPQ<8si~=n)OqO8A=a*4%iHg~#q$qu1HC|gtrMd~z-R#-dV+45fICFMEz#fQ zCcn%}*@P=7tX_)$f7lBC@XrTg+P`v*@EBPvX1@8AFU7pZVlk&2*aXnmA0fEDMAs#Q zY!KXWS(b&a?p{iYhw{1k{Q>MYYto(`BeD48d6bnD)85&Y_QK()}J79?$% z^bxW{a6W(|$BqpnI8{~YceyFdAH7ma7S&Z15rRt!gWlFE^%RX%s~t=Md~e}sFc0587$I+Fc#R24j(n^|_+ zyto9hEy4E5qD%O$(tX#c~e?IpDfBws3DLreZyCLCKYHf98T4S9E2M)F%2!a9} z&a|#f39d>}lx3nQuw}>Ym>2kbe!6;k@dpAa6JC8yHISO|cVJ+EyY9Y|S69D4!|fr0 zeG0pOe^NoTph@=O5W6a zd2*GVf4;abXIpI};dZ+tuI15=u-U?UA8rPXyQO|6<_Kg&aC*J|%cq|mt4S_)oZX$kvQ^@k_837WxNV?Ah0hPABGPqrOaZxI9td^Y?xAM$5A+-r~3a^DM`Xwtor%T3{^zmt1)LnDgIbt{dFe&8HNI(Gup-Mc zp^!{alJExtc)UIaTyA>$`swWIroE$!BgfkK!dJhA$z$%Rj^vjU%&S&H=3j4PcQ%2GOqi?c|7{$KT#BgFWqtjRpsRgs!5_De?h=(HsSU9 z{!MTR5FVSN!u*JSoGuhaVdviceEUZavVGTHe*dd`sHxUen=jmQrSQOmk5>D9{;Pl| z)2Tllg1cC!6RU6d^pFX-TRu^xOF=pZrVmyPcNV)p=Kgo%ySqwfFQO_%cHpZ&Gi=L=j1 z{3+GGrb=)pheGn4+dg}FLK(hz(JU4(n#H%j{#kl@2Uz>=7T#I6g|+Lqvf|~HJhS4J z$hE~qg&OfqXrQj9e~OagLdr@@C@w5OuhUWB)Vx;z&EaB0mSuW-HG(ND$mekDF#^E= z9i5sk*3s2N%aK-g?`vl7{$|{sp*UuT-O4EoXR>(FES8=+hm+<{BUZ+xXc&zK{`0=i zSiXGsFMj0n2OcLyV{l3vcvh?Fr5mri=8GDgx7q)*Xm4-l!V51IEthf8Gj&0kvannsV0Z>+6%|L%A-~Z3sC@U?bs?`3kq*{-bWcKUl+{14lAQn!Jrh<2{#NvP{Yav4O_kr%i3(%;hK1+*F>tV@jUBt1nK@fKcRfFU$mQwZdfaG) z$E2{Rl!c4V0N|xp*X6ujolfKj-@eUk(CfYpIFh|DDP2sj7r(rC(JXoH?8(`GRYu_1 z6{}EHf0dcDP6oi4UljT6`s=UHDB0ovMFav`yg^Y>DJ7*9)YVVq#aGwyqx-**^R_>J z>2kjFgNKcK_O*ObQIsDhb5Am}=MV+)OZR>CQ`w2C9Rn*~T#eoCprLUxpot2r&5qaO z{J164yz+qkV6uI6#AwUb(k#K zb6qm%4VcUpDk?OOm<1@kr!iw{qquG7{yRe<CXzN5Bj=Ze6ZzYDRGBD7Wa%u~Q-H9$s zszF{}!7w#r>a@8y98O+(W!<`xN`g#lo1CyuB&Zr)Jn!zHe!XNOZf0akY@6V2Epx5h1>9jhXj@fe-v1Z*C0)g}c z(o?}%%TMOy1yfa{QGfSX*Cu?;Y!>r<_uO^8DU&48$wEtswV9^LvnVOeY}0?_Tb~Q4 zszP6%ipi9oYMYmrpLRss5MoQ6*>g|foi(rW?t9x1lHmlgW_K3pFmmS*@0D{>K+Tp|{({C#d(;mG1#C zbN0z&T$vb#G!K}^sAlC`<414+Kf3=5rcg+^g#@Wt z;}P6RQcybS*3Tr~q|YW^d*eL{3rnc3uFt3*qcP{UX+oh;$|v)UgrX{hLb=ni=FDBp z#?8Cw=^e;f8zV7q&J@l)dy#B58^0aznwSJ-v6#Mb(+w9Z&e$F}ue_S5k->`!j zvrf(^=e{5a=rTAXB3Xo_jHBfMQZlEcirI4(A(yI5u6mvPCcabT|Tzz^@@q2GKjYxU)F;V@;Y zcDC30_v2?p$Q8cxjn5f^!O%qv9^M>AG{IR72HhQBzxz|;n%wN2bz4wXmB~})jD2we z==I~Ha_-;1lf#EIe@-G4ft=)tf$3n%w7CE$gy8CG%DL(Ki?G?uKaT!Bn&3WLTV1X{ z_pC+Z+Oc=mX$Rd-nw;4Q*W)zXTLfS7W&gb^eU#+uR%|8VY z^m^TfP)MF%ROnRndfm95pyu}YM32WO{OrLeDK4!@yB>?hip7%tIS$Ff@Aq-#rSmbF z%sli+Mp>(d!QpV`%!U{WNnCZ=N$7N_&OCeWSZM+Tf2EMn+dClY^|}xBp^!ZP+N;kY z&tV_8ebX>reB~WP(U1|b0pqfr1w$M5$(}`@EOMz1==BDwYZ`I&9~UmWWZC%W9WgNA za`X68ug%v3eCyuN0?CmT^7fwP0|&v||cnIYhwk!(`5m1u30W_Nk!0 zeiG~6e`)5ypM06|@NJBR{mrdB{?uz(IleM29Byr=tGkc-hRGS#BM$n8jR*Ssdn12` zi7!*=bh@1H|JK$|A`lF+@4%6uYKnGpc7ieMmDP1)UpxlJYli`# zf3L46vZCzo>lx$f|KzG%32gf;c#nv%-_cgCxqZyTC_vy zN=o7!BBH7U{Q>+*L`G%XB$8-fv*UOxe+_k&)YX>L)s=wYlJx#a=Q!~AO{#c3uDG5o z#l@Pg-1?CqI8`0j(sEcMxbPs1f*@eA+K__5gswnhg4oP-psFfv*FenRmS8YIZ*Ld% zbrsatmeZYp;JjXUQqlE77P+(6>jpu<94Bu{$>46zk)tOzSMuO1rjF|;oL;r0e%+(LGEL<+K0O$Wr0BigMNHoFGiCI__=ay001BWNklV-*b;-ldA`v!^4F=j3mqtS%PWTLNc zAS33YraS0^O(v{u+e zgd(ox&^|;_WJ*&VGo~~Ee{l3jV#~cC>af`z=nQ&19+#%qMJ-BHCB2Cci~m4C<7~;s zbJ*;8^!B;NzA$m?8I(Z|H%g+88u!ryBL%(qT*nYZ9Y&KAqop8eousc{<5sD_WHO@@ zb@+T9LO}_$#e$l~DlZh$JUv1oX^^YsCDrf4E&HjfDaU3tQD0Y%f6Zp$XloK%MS_6Y zY{h6W;rDxSxdtNKMpiW9Nkx|ng(SRQH-aGIa5z)4L)vT(`ujcOsy;<9*ra~wf9j|o z6QTF`7Fu?#i2UDZc2d)HK859Dd+$mi8NDubmmt0EM4b++&4C;W5eWEjxdP~QdMp+j zX0s(B(P^jqN%;Lf6j{b>v8I(KOcZq}ijuJ?PX}r=hlA7?4c>v_DZqmMeww#Eg~?Kg zEiXZ8eIeX@nh!0ASgdvwStbzhG0@+Os1woa4d_H2!r)GTf2t_Rii|9WP-F$2UXRIa zj->KR1wB2TxLo~An_N%$ZJaT!kqGfFk4is5Nl7e`>E&_fZI=vp9 zPK&J#6Q82U$g-@35(;24Suoj6ncE&u2#3?!jF?4IEfw{rF>T>(G)%u7y}>eAr%HF* zro{KEDyowCe+o_1>9APs*z68%F(K-Z6`7DE5t1YnRY5n%L7e%8I2=xlMsr$|wSTV` z*ExH}#K>dJpEDU*mO0eCH|_qL%oc1`2YJqXOeQmeARx;jBuTSBs;Y{?V8rfl;&c`c za>5LVO&ywp?3+AN(&5yUVy~91QPg2H>VVv4*a%7i%q9n;bf$4(co}lI z=#p1FF;cD1eH>5{USC#PNPmBJHotQj(;3E|U0WzGE25^VB*GPDOl`nsvvBahu8iuI zmE1+%e~(0OPZyOH#SsQGE`{MMgh7Re(QGz_Vlo#ZirOMT4h0DK`V-PEcbG0Sxd|h=Zs;Y9|S3kv@f3N>DBf^vA+@(qYIIw>g9*>(Pr_3Dn zqMd%)9J;%^INo+R?fO+!MNx7`bb)}M=bw9&+i$xRfU1h(GzD4u3~3a>rP{zqwy7uz zYSf^4UB?r$**k9gI1l~TUF11*{P~ez@Rd8R=I>Abp0;BxX$@*FGZ1avqU9c5dg|;^ zf6uXeiI#w8=l0BUu;qfw9d6mrAOG+GH{Wmu5B>HB1LQ>8a4Vyk(OL38tnbR5=fB8%c=1#`pu(DyxkZ*QZwU=SP0dpC9=-Rn_&(o_h*&=A6Q$rWtWJfA1wJ zn3Kr+)+5dA*uH^luRH@$V8k;C7!7(ZIBOx#zOa^#_M?=RrFXQaoMuMNP$E*FTK*87PRCteyo&Rd6k=7i6}HdSN=CX^#?aap0CoGj4>O^mGObGXZ+)DB zp5YdEK@ga|k)L0fk-NFe)lch@X7=yf$<7@gvSZr@dV9M7c=|8j=Av_Br!jl_xz$|vIW5s+ zK|wK-r_P~i@(k)5CR15ai^Vqf7M?+G81twmRaH2C{3wSH?PKrWZS38%e~sNcH{thd zPOACNJQkfamxG6nAP6Bgzw;0QUoYF&{|W$6G%#oBJ>!xCITj8aJjV1zpU#ZKb+v7Z z5SuNpnt-pHkmN&ERs5a~td7))xTMI5PA9T(!5r?t?@kW49^;MG>v-w)RqWiolkMAv z&K@utO;l9YQCwU~S$P#DeWgHvFGLEaOC628`@eko6MLkR)T>5KED^Af7e5AZx_8iopg4# z(b0a4)+5cs(z9AE7N$;WbYdhCKGvic4p6*f5P0E(^zuKNu0WHKGjvR z$F{w*i%AovMfz*Dmf#uCeEIzDBg($t!JtxWoHiMhER8_@di?YSJ+tXeRl@vQUc04x0jYK%JquFGlvA&kZ`mu>48B|wl9OUS+&ZM~i zk)xgDI~~|<@y|jnDkz|+pa7@S5qSx+iLUNmWLcr4*a@h(`w!FCy+87MYD@sKyT}+O zQlsmRf78ey9fl+yt<9Q_A_zL_Ctoz^p`l{6mqsr4x)Ym55}~-r$+n%z8B^sP8BMDy zwE&-^ZCxbe*>dc7Hx=b^qR_?(9bH-~isHgNWI063o|lN8gG`t}rU=P z>8Q_m*n8k02E86JurVS!YZ}kNY9EUKe@&MOWQ)TkZ5BCt@GYc(CgRGf7h$oL#k|LC zjaoQ_-=-AN)Tr6-nhzzHKra-MG38)q%VabXltPTf-2nUnDWw%9G1z2el^SpLq9_uF_rD6L2!cq>gmV!P2Rl8wfoPE$@O7gqa%3S9vj=f7 zsj-p+2a_YXT!~as6j77}+<)-#f0e;-zhvnx7c0fiu5Nnz`l+qZ)RF25XAJ6YDpqT8 ziBb$u8!y_z5H7 zqQPWxGI{n*+J$9 z`cjX_i%}n>yL~HqgAKDKA5)yYNQ=dU*FSc8v0UN9E!#2ZMNU8cLJS7msClwjOYpk2 zBtm{qC(eSHCxqt?(Tj$WR2YWNU_uuabD7lYCf0pvaG$v|8W045e{DPWaOKA?81s5r zkui{6g7f-(C{jmcX9d9KsHDE>;wame2D{aa*LUJL+=rXDF{wVEvdWn;nSv!+kMky4 z%;gHhbhsSK?KF$SZlSTRob5YzC-pelJ}u852q4P}X5(-**W)_Ik>-`7M26LB!sk0N z9M0?Wv1flXC(WN3e^;cLY|&~?Cy8u|iAr{KxcD062`zJHHn3y&zN8io**=cQ26f9Fx%upFI!Fv9=}{)mlH zD|Xw7<8Yg|?f_uUjBwn==pyP27|mhpfr{VP9rxPQ$_R{cnK*P!YHN=4N@}0EGqto> z+maBg+1|wO?$Jy^1^Jo+E+AlYL~RjyyN10?`FVDF`m&QsKUI9VWjiLLf$5WL629=} zVb+7_{*ysTe}UnUFibE@jbu>f&S*fV6WO?B$C!g24p+<;IyyC1pOPYHL|jCB%lf@t z!$e70sgr>LH~u(jScmsDYuyB3{f14N0yc3e-V}mrl@!T|A>`dX|a}Iv8$^m z;=DN?Xz%Faz`?`Z{@IJhoI|Uv467~5XD{27p9q6ZWwvM-2QGKY{3eo)*!%KX6vLuT?Z$0e-8CA0b% z&l2@ne|%CCs;aVK^Vm5&vS}J=@6-scq}YiH1Oo%KAKMW5U6BI>1FpzWhB;hkS9X&0 zB#X80e~2ImESxuy@nQmL7e1^IUjtub85nNqm#1Yd_bkQ zb0_Y>P`hXZ{9QmD5Z3jeV;ni$fXmIcox8d2ilu}_R!)&g6VV(lbxKWLc4yF27B6UG z$DX~o+@6G2%AxFKQaW_FmD;Kzq@bI&qw4|fJp9} zen+#)1H}~!a25`2di#Ankv@u|KvhK{t;e!+=vy*)eU%pcy=2jpG#Ei9NT1@wm~~{e zk<7~(r%gu`1=hSfl;$BGLTM(3iN=wmf5(tzg|Z?6SN|aZ%$6dGOJ@$V#RUWXsEQQn zXI*tMtw+a?!>wHP4%3@zs4Oc?gAt7LLNHpSj+*3gWzS=mqI?@Or`EFO-S@|wglurQ zLx(keEzhE@R0Ra8YL+4(px0T@>xYtP`+dEUe%4i&aJ-|F;28PpvLW`5wY9TrfA0a7 zpEjKn$qbS&8phu&lEP;gmpgSj2U}V>+&Vm6|43xR7I$!v;K~X$%XV@3T+G(ONMEA4 zT);oHDy^+5LY5WU+H!wl-)nDa;`EGDr=M_!kW?8iQpb_fQ|M$gp`LN7=B=@M-Fpcq zA{(ZWa6oZIsTs4SkmAx=!@3qte{eo;Z={d))!JTV%aNlQ_hBr&@%CEGCIhF;n>enF zAX6@PLe5J~r<^>e5u4S_n)MqkhiQ zds)8ttWkaYNFpi;Vh(k(2$Xq`7 zVj9hP2!qK4W!X5E;K=VS-t1BnJhj%)TYLSe|<5UhLRcj{X?da z$qf}8@91K{?M}N7i7?=Dv;Kq4oO$ZBH1hS17dMR0C9}I|7>7=uvvdZVx9^TPNX9}Y z1h;iCzH@q0Wx{@#EkgvSrNz>|pVm}~s;cbTdmwHbnb_f8eRDNIN#dNPGfp@INUdo7 zkC;WW^2q|*9jlzbe|#2-qVU?PHE|~;6Op=o*B-1E6Sb8^3HxC(M{&47Uu0aTP0~^= z>`6dync(@C-@s(lbH?InA1OwVT=^I97Kva`dYpFhB=YPQo_lE}A3OhS9QM&U*^~?x z4I4LaXJTC`qL^4*S*@i^nSNc=H`@43YpO&PMYiqSlXf5Cf1#tZo0ndHi_=bS!frRA zlH{Cy6sC5NzD}4wTxZ8-j<>x_d;11@dv+0$yov7_d?^e%k&+?@A8g#h$!C6yhadgh zs8|M7O+86rS5Gfr`Q{HebmXWuKWT~%vr&)Hpy!`2zQ$vJ|0hxw?1CPzk1u`g+tkfE z4VT-a8JJJFe~3(nd!}g?$-JWqy*+#B>usjDdk^i$H`2WSIoz(JWA?eH*Uh2UZuELR zUZ0=)ANUpXmtV|p{`lyy0G4ET9Cq(Jz%@7DPTkDY_|-#y2B7(HXNm(kR1P2OCJ>a6 zLLu(E|EJ7Zav?u{@OSj|rykMO+1o7vMB z@bL>jf63YpR`KdvD>&ZKA91Qo1%G+^9aL52)=%Aq-|yq)H=pGYl&#x$^Vg@I<>i%cbKuZn6h%R=*K_vi7jfD7*YNC1kF#?1i)`9) zm^ssGhTWU}n5HbI?iv)S`jKDcEbcZaYYB5&kF@|QYd_e_<>${&c?7oY(l-97@dtuPk`z$|1VLazeG^L-oyF2cXOo|& zEm9UQSjNiLFY?f1Z}FS&Uq7tv*rT8O`An{+Xo4HY;NmorqmV$GtsINF7@w~b&{llW zE5MOB-r3Kz$#XD9ogP?SQODJn+{o3Jf80oSZzl&@_R@0X0Bs#dak&R@x%(-~cjB-+ zDKD#{zIp=Fnr2a5Ihq5Omujob%{vcs`T6rwex4(3J&2-6QK6PRMnGi7l)229G8b8u zX*+(D=9ay*v>u?dyB(K%07(i`TAW9oJ)hdDdKzjcGGpqz2$4l&#*{f23`W*&f7qWS zJ!Fq)GD8%>h0PjCT%Qw|&4m$y3k3QI+o?v<`1(!F$cjQk?WDMM)m1k#v0+NueX&`! z!`S);yir^~d3byQOhz-h7+F7cf{rPZW-xQ=+_d{=Fc_(?ot#k*sz#~CPPsaT<(4ZQ6$gqjCnr{t!|BaU5p_adyI6VknsFdMb%6=f8D)Zm`!G^ z7F%SCN>x?d9v7jIL_vQ1Q~yeewCMSrdyles{=_l+lokdltEXoVu7SgdqKHm!!Dg!< zFRyOUWh;4%(BV!jhjY7zym;kM06FACryG41ReRR}08U5j6JJD8L?`MfDk#D0^U&Gd zK4{8OFd9tQ?RgmViO+Bie+0A)NOoI#nL90J16{oXaeoi13q=LR_yazA`nm?alvVV4 zBMzGbhduG?xWK6q?(=pK2=qn1cDs(^cC}Jfbt)RLQejkfb2~K=ECz!OyS-XV<2neB=Q!RG*`wmx-bUErvaoqSMk9%eqR`&H5mg<&QxNp`#cV4y zxMii&#n#hwIvpW7e?;Fv4_1qfqJm;fMl*I>9)#B8!qR8WH1WWj2&QJ7zhL2sbDC-%KVBsP?B@P1fokx z$s8&w7E)Jt4yGuVEU&jC#*~g#GucRro4zRS?cayruNhtie?g28oghnQ(+{3FD3IrHlCMvEKy6!xW=ESpqc&zAh9Sn!QyR;79*uCT z%10li8P?&dE9&t1y>#_-pc8c{gWE(7TON6i_&O#K8K=Q*q8L|X-}WXTkmsnSb5Ohn z0`Yb2*wsdfe^iY;bIY1X7D@8Z-J=~8Cx|+V3a3T-77X;qyvJeHR)2oKcg*Jta}rS$ z2bFRaHHrLj%`JN|n+)XH#!iiqtodL1bmoHn#c#}ALzv&@M1P4KD^iM(b9$$7uZwdG8w{T zaG;H`e~Kh|QRHCcz>zUA*v;IVMPgLj?`nw{dM)N+oQ^sGbfN)Skx*3yDd@&%9R3D3 z)@ZA=Yx2%Sz(Vbp3GAkX2%WE}gX=+15}f1SkKX;tYliZmszAnMWSjmV(@pb`iU z3=1eqgzyVI?rHNwAs;>6S_)M`&{0}4D?;x^ik9m0YCc>?v2@wBn?I8D~gO>r|qXEOFYLr ze~zKk=_rq6=S;DrId^W?kwIe95D2Dt05chwMbgy+I=i+aE1F@ns9*{Py*2WiDcS%m zxklZ)^rVRhg23UUhhk2|NTPDso#ZlFAH;hfuEuE8vuI9zMlaP(S1bZx z^Y+-ODMBGBLR>bhgM$1biV8}S;%=bnf0=fxaom^$s)`&6(Ahma*K$l%^ z?%@OORyR~^qXvh~%=!;kM^>fLP*s(n6iB&`09`#D^!E2Kv9364Odu&nf6=`y za-3Bd!9WZHcYIm{()RdG9kMLZ-MtggJSa+wXAP9Pu0LICD?^TO=He#$U43lZ6+6Y4-|tIWghpfg?#%!!Sui0hOdy%? zLASR9w@2Fzie4lHBp0fxB;5{-e`DJGuI_Efie}|6ENH@@w~e~SXtYP1FXfPL)M+oM zDi@zQ165VovLklN2Y(s|ta*JN-dOVz1$kDMFDL_fg#Z8t14%?dRGOGEv7xGb=Z^Cc zMUjQ5tVXk{^$+9k*9g^vx!xCQ4&`Kp3`#5{$v+K6}+7L{a4ZP4C1Ux==_O zbNgx}UVif*`29X^zA}8$b>^k4$j7>Bg^iX4n6n*t)mg)UcBxV4=ZgWf_xe|{67Vl+fq6Qr;^ z+1T6-#vyxBckdo#S=)>*a8AS+LzKoNxL`209(V7pX8|g&y!BkniBVNm^Q28;A|HG9 z5BU6EzWSNvgQ0=L2Pkz6^>fbDO6EKmD4Z9?bwP7!XB%Xim?-=wt?!0*! zvBo7?CXh@~l@MLsf7=G@6(}y8j)-Qu897I(>u^~TBO{B?-;K+oy&*~B z_Mt(RVzcXWPHm#9%*opK-sH%!L$N1PQETWN?g0Ebd~LC=QpoC>c={g#phIOa=!ggM*TS>3}fieFnWFau0$S zcT9fut;4_d+2l`fBWU9QW&(v^a+_S7<%f!ju2}6Bd(!pWJ;>&iWcI{$L+2 zmzI!6ueVU>9De9Xvc{D{PpRFxakw0PZ(Y(Nx{fs!sGMslV5*>epesgf4Ti!8mkK0pPu?{ zWCtS=f7-i`^Sxi(&5>g*eDdOXeCCRiv)ivYgYftNx`FztLe{Kb$#4JkbAm~%r2G21 z`Tm1nqqVJtYc8D4Ma!pVG6^F}Z1fHsBq(X#C=OdS7IQ3DcNLQ%O7#gP>p3zkE_Fnm zqnJew->M9-`tdKY{JP(=X8lSI9^S_%FaIo)e+J&MhloppWwP_{=&LR{=;dsX}w(R(TKrn#SY~Z`MpUc%3%pNmg zO*RqT(l;J_l|MhX5m}Zg$S-2nw0RWf7ZZ{~bac0|b>~KWelJFYo^RZCrpARwR+43? zsI+x{Kp?0sY|4t}V2#P$G`M{oboOWle_@NMfb!zRF}#ObdsuwUe-9GeB z){&!u+jXFW`+xfe?`=7PKQOeA5Cnljr-h4`Pvsl8or%Sqqz_k0NbRAHb`0>XU%txf z_YdL=3~lBMf`HR*=Da0M{O1?XANUSPL1BV9YHDg@36DSU2>|@T5PO+S;`m`r-6HkOX4Huv))g?#iyM5&R&ya|l;JO-T!QP82NGKwl8%YJma zv0sj~J$}be7^R3oj|;O=&+I9wf1b>IV)RtAQOBIA;l7T49|BSzc1t+`HggH0pi7F- zowgc6gZtx(BBK)%+>@zUB)7wo0XdHYNBF2B-BGif%dneMKbqWWuN(eFy!)cuu5uU% z{n5zUlU00V*}8_0G80gf<8c26i6#^L%RkZyoX=>2i_hRr><4muDz(5o4f|e%OCeCNNy2LRKE+|1O2ZLAt{~&I6T2Mv!jL)o2cPw7vgd zV|AAk8TAksQcVl-00}yz+`njxgN@xNAkaz>A delta 18061 zcmW(+b3k8h8_unkmR(EBw%xMLWiPuebzS%07>NFK zh(B@?z&e3y<71bZ+;90yKKEG>4_~M+2C*ce=weXdif8Z|3MA_7oe+=ZrEUuqp(O0- zBua&pdBK8Y-#Z?ExGQDr$uY3pelV5fq;cI#UcBmIVtK4kM*J-ke4SzUHKtTM+_ z);`~NoUPfV_Vkg8_L6kQfmvm5ji2kcb>w2v155^tV9cCnE4bCz-HlD`cnzeMj4tp4| z0DNEEB|Eti#2k*&@}2PGg1?KVWRm6Ln#%V_3a&Fw_JTh7+qDfb0R;4PfbO)z*7lq++qI19C;o$r-m!Y6lS2&1AAx~R1(@OU`x5oN6A}M0LJ< z1XEenF>W;(HS9JuflFIvtAf51pjFk36M9)9gV@ijNhXQPf$Cs#QPH&Q zDtX7jV@GW68_aw%qK*$H|uVx0^As zseyxZthIVGSbk%7&i%r`wn+3WowLx=#wX(Y>x3* zC)@Ly-k56i_vaOg#A~!&-_)ehAxR+5+?hEoEJ3N@CxAZEeajIAF2U*0 zNvY9MwvqM5H4$ygFPJ6WdML9XzB~dk#r>WszNvAG`j(qfnz<+HqxPNdZo&Mmit=(p zpq|cv>5-M(dm{cCP)Y_Hms!EY)l8RSDRzMzWkyu^yAXVTx^%lx1K>#4_xodDuui%T zLhwgnBZ{|&km7r~rgn9_Vk5u3_~ieJ8(f6x-TCrsq{puV3+*8W1u?|^WhA7w&059` zMj2~!xX_)b(0e63A`QTpv8S^bJI`X*uv@Me%+KA3kZs~+m~el;(x zDJW*}E&&|@<7I7h&RjFH%8G3&EEQQ3TUvG-YvGpQW ztaq9FS_dEL|1Izi2XHM(DyN4n(J|@oCi5C1iF^!>6*ge}qG3&OhH=hWVWZO2{AYQB z_oHkfbato|xa3T+Vc;Lkr-`?DAj03-(bnrjce&cG@d;Qvr^hkyShX+^(x4juae46l2dCX9AZ*@!zRq9r@b z2AEAx9`S)IU*J&R{RwDTB{JpjJglBR*lkl)ceS4w>By^6{ z7Tg;`ukq9;Z5yR@f28Zc1A4k6zv7>{e_S@mbcBbeKiY2vZILow>~$RF3>Czz%euO{ z<_l1y?{-!%ogRcg>UWP@Y-l5(+$dzIoWQRk~z>gX@4XPm!=>iZBm|xLuz(rc^Ha1I{0P| zdjeoJJ3QEGx}K@0_?w|+V9L3vYnKr<&J6lRmuXOM#NpgYBN08yEbr!)-@jX~PExo5 zcQ`)p^Bv{4`e?G({B>(K>+Wk4<<4g=+C6v~VB$llzIY22Dac8Xc0*!{tO!U^3i>JI zN0 zi`BC)$i9FQCW`9I6PK`H->v($>3E!-X@iIm+*?OnI6Gq$20l`& zd!%kWbx2xP_2DO#zNM9o99XqNxG2fnuKK6!o*3#ERPTy>(P)j9%P^unhZZNG2L&l) zrM3k5XkVW63ryxt)6*b_T50-C1cT0C0Yg=O`8$Ml6xQ`qPR>q34&~eMj8e(ZUGE0- zp3H{#j)PoCzK}hlhzL*lym)ooanoBaFj!BI`qADzMx(v#{H?o1Q*z?mXyu zubjA_M6y_Lq2JQ_nTf(L6vG9EY8r!By5daPH}FP8jX#t97MK{sA93if`UB+ z1SBT*ixE7pgdbEtgod^NJwA|ic%QZdbZ-12H17wg@mWaleY$%GYkrPb@g#hHj4<4U zPcDNhRzVMZevgmv-m675T50uSOkm5wEeB)ksYD9B@9dv^4S(%&}*%|y8FdC9G2 zRZ6U=D9`0aGXe2}+H(B!q}@JMY=q#cEuYu(hmvfMa9Bb&-vVIh)oF`SG9C`f9|~gB zDSE|l(6ZK`HdU2ib<$;k7Xb;0L^A&3i*!60;p<$%{Jg5NvaYPQ80^;+I>YPTfv`Ap z{{2T-&c{0>12!;xtK;rR~ogWMy1 z4@AZIKj{;yGv>gZ@b+af`eMuO#Ett$PF z_lLD}FzFO}lfy`yw-@A?J!Eta9J(>)I6J5=f$rthkeI~^h>;^ihTnl z!8kEi>N?YN7(Jb5Zgu_lZ=Ej-HSiP(7Vx!u-@=Ow!Ck#oRg-8g0>AoqXKlS;LI#Me zzX#+b^~wXds_JSK!B0XG9zD!hp9C2@Yc@h*D?JZlgcfM|1I_yPpP@JqX(+V0n2HaGqR_3V5TT!FM-)HJL0dUn|S{CND~kF0cDoMF4W>3Lc86$)Yryo)Yz zNf_K!ixoHPGi}OCd?}bgw>7ayI>q5A%axg#xmWvl$H2)#g*(U`?uS7s&tohM0L97T z!XoANX||}DC(XSx8!P*~vbu8AhvJ|dANTIbADP;8>E-n60{{Eof6H{c-p%xb#f%8zxon^FS+i;K=0N$v zE9{rh15LNhg&%!X)AjLk3tYBccIKBTzxfF8xLz2q{k5?GkO^k>*^W<6cEf&TQ$GCt z+)~olCrZ(A2(u+f7!iHk9~n>Q%SdA}W42nPHJ!{HX|R|Jhp$Evi(9ETgGD0XzQfr5 z5U!?!vYUevTCR1!HT`w7KN4i03hT~G=;7(f<#LAI^>)DE{;*=k1-xcP;NTIN&IPk^ z;v;Rk?M2mXJ$^c`)|q@I`g|G!H@CJl^0E%S^g7~UWo4b7ofS4RA|)UoDCc>V&XXW? z4+KSWgh-FUpDYb93I4>#t3|`w%4SffPl^W44%W0bF`mYv+3Cgg`gmDVT^)z4_bxff zukP;o?gMC1mjiKtu)TVp^AhbW+PnE4*I9NRJ9oornqM#4 z=-ON_Kk~^WL0-r9{Bw%#OYcO}mXD{q%fewU=pcfbGB!~-G6z1O!kJ0`#(&u*Q;xJS zPrXusnd1Q<0nfJqXKPLJ&RPYT#*Eniz0YPz$8my#T=E6lJuvqOvRU5M*%>F9L0^wM zgv;E`Ew#~Vsm|`2IWTc%?w-RZ%&zngj#I`&+~S~U9Cj26wJ%43LBAz~{=fJY<3Y>% zx%EoXep-H6@O6QYYh?b%mZ_w3ek~`^&ahZ50<^zCKBpfhUa&Q)DuA^;?$Y*t>bh

S((t^Wc zM%R5_y&r_7$NAkC^8Y`g4r-N`{Bj^#qg|?n`q+|^k}%Zu+)Y+e=L%BvJmBhAfblz= zTPe#Sm3`AVVJd8R^dt##Y}yzr1Lm|kFRg-^;r3@~jSdgz-mVOD?#uRD&ii>~lZb(v zdmU$IeU`QjpCOy34O6~v(XC{X?azl9G8z0SFTt~z?<`v{VZ@rr*aS(nS`m&mG}Ptc zn={E|&J`@bdz3bH?p+rvHT$@ZfK26GzMBG25U#oG1YvE$ufyUml$Vt3&|^aVnnmPT zw-b(L-TdsIK!&ttyIqfZZtNF?3X2l*@7 zcW&ROn^faQE$0FSHt-`m&WrYgWNn#hX6XBa{QrL3m+Mw;Oi?*!96)0nDHG3A*i1Mr zN6!{?T$)~%Qg-C2Y0uBja?Jmi?RilvUNAG2n~<*h)vhGCZ$}waH04un!Sr>tS{%q1 z*zOw|os;IH2bgyp=R^x;{N)eF$Rv|P#Utcj5+V*6G)99F@le0M{_w}7)#!%C(ji{8 z{`=c%K@mD}!}jwq3gFnJWMfYT#?o-3Vq)rD&b3UOw0g~#4Hyk*cOpZ*sBo#=oA3pN z=BR>ZL8d?vyAG@o*m7RdUHnPr0GZ1TR|?ZSvSarY56Z3g{RsEIDf&K|X-HHHndx+B zKAc|!3uSyG0>|c}aVIjmpoCE5nl-c7tT0n_UDAx3)-II*nkk6P1xsD~5%Lt#A0Elv zNqXM*s!&1(E`GN*X~yh1*q?ON$;r{Qa`CiD*6EGIx{3p!Sdy0toM`HLw~^+=|6pWf zsi#G=L(He#d`hbEGT9}_L)U7w#5gBNuy2);j?bDBKT-l!(mIMVs+7Mf1)B9(b)JtG zXeUG9GHC#FP(@9rmIJ6t$7v?Ra0;Q=>b1Ms-!;rhvDgGnPfw$OV%v1}ZdOhr@&{6; zdF!LV?>)bqE#1El6b{>rM#T<)mjphar{LN`FuhN;x0plvb{^4q+PUP)d>uisleT)L zhP`k?MwMPw=03Cc;M;nV^RO{D6aCR<#~PPYDX>o_42OEl%k&33oyiKkTM}GHLIRfP zb$Q5+J8Mh&oe#q<$MW}Yjx6Bg`A1^yLPb&{ymCp3OYeP{)+Brggvo-GVVbNHH5+S# zg8Ei}I-3$yjnc)+uy=o$won!r>$dxv_;hX-i;$io;A~+U@o0yY)()N$*9T1dUoHBglh` zx?4~nGv|)q4*!eStk7jwJ}nS@Faww;9JsR`?P5iN)(;B?lQ7aij~hi_URD+-Loo-(Cp2+%3&)=`1?q$Z#C1g+XU><;ks2us{e=LfK-A;Ir3ZWQ@ z3J>#Vb(AGQMTYuSxU>SWqORED?MC*w3J8lqlX+vvdN^MjGK%3-OWvc2$nOh5X5r-g zxm0*e-bmVS*e3@S6-91#wzo}_6e+$JSo7)PBukEuRO@=VS=UHakXf7jjLfB$toTFj z2cjJwhb>gD)oerjDixn4JAf+>zTRB2&%*2*##Uk<{&@Y@W(^)_mHVm^2H-kRmqMCw z%l^R*7x41*wBUJ`@1LyXjEY7g=lAT`GqJc8ZD-8NQU+hHHIh(EONNu7St>2?NXyiu z82YGr^R%!y9oOZ8%__x!@tv93=?`_VcH)#cfBPr;oSUVq%w@L?<^vF~ z<;man7#;}}exz`sk*tM~7SQqOBg>&~M-%c%QKA>-)dI5ha%8BZ{FYilP&@rq?+=ee zMQv?(A$>jD-Pd&unv>uD#?xwfV-&U&K}yDRdOiK2E}e!@4VPCt0fPX&$iu?}th8on zndSRfV8C~O_p?aq$HdUhR@2XGz1G&&fB*g&*eSr;xr+nnj`+KA`I zh)854!tBb5q?uU>{q~#BZ@sotM3ebRpDru}7;9X2=@X+0t_}&0uh}AFu(|){xsLgD z@a^vJSEaT5XF?|~h*7S-TLfMv;#u?Ozy=VPUI4{J*qp&+l1?rzvkMCY%l{a6{=94| zWlRW>;ZL^EROr0H21rF_j}>JV(`_zwbaaSKGox8vSGV<$->`<9~)o-tQO-_W@?dYze-rHdUUZmel)w+%<%)&>Sf3??Zmq{2y8 zl?7-zLl=At=?mJr6pI%xs%*q#>P+)1K}%~o;9EYSf63-?yY-WU+q{faqkXs_BE4?DT`c*w(==H zPr3O)wPeD-?0lI*LcZ04mFgMe-UA0xF#v=H%nHP=t%^v!F~NMw-RsD2U!Lqab(hcO zbkUX^?U4P68NFAXOeZp-yW{#`oi2Z&L~`odMw+7W=SW#ti zuUwh*93Y(XIRM^SArF?Cz*iWKf(<>KnR9o5@(U*crjDG|CjZFtGB%RLuWKu{3d8~f zGL)-th!THR&WFV!Z}Jt%XGw9R3~IG%SEz4xUlpeGxVgD&c`}`xTy~texn?VnIR)+a zZIEb9X?k*rV$67pZ7M_KNxFr-S^#ra|E0$wC^-# z$>n478&RC6m;sR9lbk4RkIwV=&9kab!UoXwBY6|1WXwodhZXoJ_Ib)D=C6LIwp&R8k5&lfRVJXLM|7EE9)Xvkwu; zbFTCC?Yo|*6*7h`o-`AVDm*kc5uOkue5#Bo2$vSICKpU56O|N2 zFZh`s2r|88J-L!QhHzgW0O`hSB$LBL+H+sIQor(N2U(OPuia~g6GxT;pMnGQrm(>4 zSH3`h@JI?Y@q)T0b|ijD)Nf*n*Sd%XjjOe-mE@CrrbI?Q!Z0XBFLsV*jSeefj+!nn z=(Lz(%ENWzn~*KaYh!gfH(z8Kacg;WK#qWqgav})DZ@lbzvTmQU*c(hMOws;mgBBp zNYP?yEB3CoX>;j$PiD5jZ5e;~2ZiPj_q?}79Bx_G?>RM>l~(pzHiCc{lmW`PQ07m{ zF=o?x=#;kj(b4o&3R*&I`&4aN2v35eLRzm!6Ks;x=pORczAaizoQt{AuRWzVyO zoDTvO3sPiS*=|5xb6DDNJ4u_14UEC{?i_0Iuj_nea1*1u`D4d$TWt^z+Jo(B@q@ij zoD6l)ve6`sXPDj3!bFpZbyW3)3|)syqO59S1ZtABz%MiOx4E^kbtP}a7FtUQi(H$F zb$)Ga*y7)Z{`!ebF1(Vt5kK!0Cexr;%B<%1e9}HhRyLrqbSxM@o!Y2Bt!9yCP0R8m zibjWBhGL*Hm{f6)PiQ0UEcgY+OHW?2m@^-(-9$cK^x=;q}cQ!*rIU`0I8pGkmD>MBe5PSOuJc;m$OjA5Q9xc7E zpcCP6F0Ji3qUmZ&po_>gC}E4wiKuDg<=e;GG6TR>%;H!hftKcHeoI0AZCukVE8%_} zbZDdpTWhLjI3`5PJ3zWWJK*T*Op#x=BSe1N6|ig5eOrf%54SYdH3W4VqDnN zGs{I)|2d?;E9-CBo6239z9fzyL`5=l7Dk#@squgo*t>s`=Hxn+TjkB{d><9Y_YK5P zk_TmY@*3R{Zcc6Tt*vyP>;ART1cru(iNU%)`}TC)X2{Fm_;2qLyj~JGz!Y?KhwjpD z)7an9TPb8qidrDsSRTomSBxj!RT5~W@dbPWMPMKQcs^pfT(0&`p5uLiNg`e-AB9iO zZ1shNMJB}=Z@(Gj_Uc4o=qR=xV#STpap=i!z(Ay{D1K1}PnNH$Jlgiz1nRA`^LJ3? zayB6=8XArS^&zaC8*Okge_LQ!xSCB}Dg!^+; zliNqp(sHLGk9Jd~?0lm}C2NzDlKPUMkuuHwJ)Mf$f}tyC?OK*bl^Rq}YN$4vSlspX zw?T;z90EL3C*#=8J&Au@Ky%3T^?tT;{Yv13X`eCk^;Lnq11RpyCVmVY=RfL>?#c6x z>3`n*AoRRLuJk-&TWoJ+R@76}6tg-t1_3gq<&{0Eia~3-L2V>04W{+0e^v+rMZQA_ z34nLS#Kj4T2g71xbBY67iXO<9^3qc$f7a5toS9SfoCue6y}7c0^Kchkr;DQI@{>3d zJI=~f8S#8YlY!pTuR9LK3}MGqfx<#bYIOBgDhbpj=6}3YEr?M}-%STbIB=npX-`Zy z1|jh;)xkKKFG`A6i#%*h3(8FV4GPQ4!}ESYgAv%!dA+S7Ute1#sTfK`6PVrp&9dHX zO+eKC>v*NWu=bY_*xhjcZ7I+J1LrR&>x?l+(Zuw7@|jkTF@>wXxZYsW+UI60<3ktP>{}z zZ1YPC3hLS-fNqov*uT57QJ8bc*GDEBEtijkaUX6s1vukWeiLT;l7Jw`dKM3)s7OX$ zW+_JW!M$s8FKU3;cg4EbwtmNj%f3x_era)fecrdx7u_HldCCrj;J6v5^vY|KChOxtAHg_+?z5ikjNCgJn*T zmNqC|I)f0di+*VZIb3{PA#^dz(~~&Ayg2*l#~&#uLwr86U>NyTO42v?_p@yH#|G1o z&XfT5!NvqEI3qj+D7O523X=tn3XIZ_Qv@6vot>@MlJy70?fofawY3uw`6Xfq zXYhMUqBmoRc%-D2`vR}x8Sas(%P1`^PeQ{{Gn(*!(mehHr#}_3O#4_%j;X==vaQYzQBLiC(>z@5-vEdWbOYQ|FRZmM2df+S`1%KO~9m_p6ShwjnKS7C9PBN5ePiY z>+r`Zvszc`xi;0A8GQDvON-ah$`8wQ5QBPOmv(;nNpd`#D-XWrOthRJ}z}7&OLw7=Qy)|+=WKa zu6A64Q6!|z*i$hPr?C|zNVzIe=*8$`AhZZ3#jH1+6lqPP6Vdlo$9*{OC5t>kj`B{A zmsP2#a;i?)nuvuHab)+IO;cuMPTM^r<5!~nsWWNR_!82$Vk%M;Bb=uo1dxt~hfFnw`lJ5XY)RpjTJA^=RkMbyF~TuguG>tCob$T+RJ1mrq$|yT>oc zR1|R}FQ=_OZcOi7cybMmLQX{$>XjPt&gp(LpJ1y5-4lyv7n?dgf%O~hn;aV>XaDd& z>oU&r#)dDdVRxA|wG1WycT^iOg+Lb)ma-zzf_z*0%+KZ^}I71dUi zbH{a`3|WdZ_JM|<@#9Z!3Ue{ik2tg5gH+1JGmfPk42-h#?yB*H2!Xr20m+|+q305u zRGP=&%Z-m-_0|*Jn*6Rs}^T|CJ;v zT4uJ);~_`6t+eUm0O!pcE~YY8swVUa&!eM-ngm3TR?H>es8`dF1RZ)g} zP8nhV7A;^xHov99J@5t4bkUKh9B&yP7sRr(b_ zHG#6!d6inOb9Q>S7zf(80c$nH8AUuR+Dsm=K%|ao}4YQx+g%W?A@=&}(PAalGROy|qj%b#3j?j$5n#;wA z)waULUZ|H3H1AxAoZ$-2t+|b4a2qm0G%7|BtDcR^tK^&=9{X!oS1QLdHO8TxfYjWKhE}lK7dt0DKE?d_|E!IU&<#Yng zf51gQz{rb`{#6xW!jX73d5cmtHw;Toe(%v-gcK46xoxM;uaPIQuoQgr?% z_^oo2W7^u;aeixeCn>D~V!dHKo`B^a^wKVQm7C|bO<$YVl1wQUeonr`YEhVdYy2$=}^GAlIJWiOW~u6{nLUm#e2~RDT{l5W-?zHm@-(a-U!p6r$a3` z7%OzTn*OGx#q%Yo?<@k^AKqWkNQ610b_(m~m7e-m=!$jT!L{%eQdRe0(mK%*DU- z4`+Odse7mY%=0<{`WD5(BYFC7I8dar6IFMFbW?f1pQoj3s4}`hyXYb(bVSaP z7+^Q%$wJc7PPc1~ceM8tWW>ZRy4sjsmAs5YgKbW#sWJsd=QZa!{(=QTimPiMfK%Ul zlAA~rZ~gQFX-6Hooo^tM;2rB)FBMf)$L`V`ZyD- zxT)NI)HDDA`hfY?Wh6_AW9|`Sh)2u`d_KOTJ*=T`MfgF9)o-hwW0}cu9{dqFgPf_H z>Hdi-0?@|S8oiU>#zuTxmmA8-Yq#p?QzU+&Zn+XF#7*7}ox9t)Yse-nbzmxS?$Qwy z*sFYmxEj>{oW~I5Di+_x3%5E=x8puzZDKFx19Il#SS$GT9V`nQtG@bX_en?{e#h!~ zd$Fukqq)V*R*ctAVkn($_(5&r4o1MtmMM22)1ScLsP#KVoPaKOO4kk-DI`I)HqDui zdgvn<)X8kyHJ6rF{&ILEL-=CByP}-j4B=NOX>*wgvESs^h8|BArmog8(H>-GSi9`R z2FOs9PJ<_d7CS;ulpAou-qR1eYobMF|)c?H@MhEOFJV8`V}D?pK68OyHhY@zWDeR@bL(~zPOc%xuk($1^iOh7JF`@27W;MT#u@hA$+(vNS!t%)EnZYhum4KTm#1H9ER)1+j2q zK1zo{!+kKKXjz?MIZrjdH1_#jv8;YS@&F9vqbWq{?V5bkb0FH%bn(pH07V6O9yI9w zB_Evq>OHdDJSjPMvnj7l3)+9}`9(Iy7EjY2U#fnn!w7KM+yozj(kP|dU-UU4w252` z+{Z{ZW=krd7Tn$Yo~IRp%H$hxL})Vin0Q?B&l1LSbmgW@nKUb**s&cXS*m&Xrcg8u zN(||Zc$_vP?!H1zUH5`XDZo1G$-%?@yhL*>vlLwrm($7q1pe{31vE^i`$LfRYAfkC z;xh3fFk44!;3%=I-ue0OS^^|2KzuypET0xcJ9f6*A%9C)G~W0Ap2YD*f`pdq)PFr! zX@^JJ7T0;$ng%&276hd*_a->JsB#esMsTsB=*!Iaa$UAyOU6G90ag{PK}zA08?8oI zE^PtD$#SaMyKk#X#o`bj#@6fmWuq1|ZBAh7@O-bmV;Z!Wa=IRRns3w{AL0_}k%k zlvUZiOC0|>v}pXX0~G66)z|-9WjfK$6*NmD#G;a2aghc=Oy(NT1$U9!U_a4~$E9$lA)|fRNq&+b>f04~} z*)-vO8WJ!$H;FyyO+ZINim_YE`t0fHlR(wMVrjFjr3?WR0PfcX+@7Z7zftwI=S~rA zAFg%D9+8OJ{%qwYzu9$HQ?@{gau9lfaHJ`0BzPTR)7SDGB-*NX|S z%b~~Hf}HpCw7C+y8PBN$qa~e@M9D18$H$kq_L~>WkGJu!AaH9fEW!40LT?wA|_sH~## z@U>QW!y`6E(Q4Oa+VF5(FSaf%H#-v`^Szl7dOS^GBfx;tI%Ykb5XzU}UQv>`K!C=8 z5&Pdo`d}UT<-r;Gk+N!DCYAl*uA}DjpyS)Y^{eLf<>S$SvbepEgYt9UbP9taxpZF{ zI6bVRg_zdTVULcO3AtPe+HdR4Q6lNMqJ`P*ZhKStoDlMZyY9b-a@8N*Q9AE_pos|_ z1N|vm52VXIjn8kqoA$h$OqLbY_COo8g{5^-MOB98+~MqEd=+>GZX#)*%lY!KyL~rC zE>1KNPbBe|`9;dO9vCROv|3+fQ(D19v1kH;-jD2!ySob`gHq%e*UX<}G9tRr7}eC= zT5YNGOhL+~^fJD*NpfjSMD;7ZhxZk1&$9{Ai&G{LN!-&1mGWFH2AE@+^gsa4co76W z$`z?{fyebBqE#$eoFsvAqgHXf4YM-E#5C=cm|RF*+a?Qt^5TzY%V2}leaO!w%A<2nkgfN6bD^i;l~@)-(T?9c$lqsiCEn3cpLDb>ycx34x-;8A`C#68l7 zCLJM34eyO?C$NvWP~eiupuEui%hj-nT(+J}(yiSH8_Ew}z36Zc=$B|J*6>+(@=+xu zE69NoBCce)rX3kYpPU{hX0JdD*+}D5Qc5;y=SQ9~4?HQ?nqWUM;^{{^+ZfG+HnX@< zNnM|<2_3U>YDTVcMTazCO|$2wu~$ z>0j(WN@syl>aS4kfdM0Po$s4WUfGJcT5W6$O3*rIH6MUh`Cw*r`HT3}x_QO9Ah95z zuaE2a;j5oiA}Hu?m!bL&hSWn0lE(6Pr3g;P5M!5Bmi8XoeIE%AmgaL*i5eWlxLR6d z!(i9PI8`iZiwC(OV_|kCBfliUmzV2jjIPGIeE6mc)Y$7M-oV}$i>}@DwLL|*392R% z*<9Ly5x~0VxV+a`t`*MblM}p4eU7%`7Cj+ODfL`EE7V*;GWnQqK7ob zl;DuQGd=8U`D)*d7=>U^zY2P0$h7glhs%KSK5J$c)=D3n8?L0I7|~wxXVr8EQg<6! zT^?uI1oPumzkzfoMSpO4A?%+VWjI-BxeW}s0wdfeWeY0surIIIuSpXVqfAm#MZN;D zY690x151je0X@De#>I_gkU3jp+jasJ1ynA(xqk4j3 z1RDOt)647W2K3nQrB`HoPSiiX!T`!%@O0r;s6r@)axIaW5+BX)d~LNF7*!g7Ok;QM zSm+_P6)wS@JNu1Yd{qd8mcRb6XqPG5N2X-3P?Z%CCpxymmPJdASqFOiwC^@MBBE_XOO?xr$Xx-ik5dlTAlztFf zAr}$>?~1XI)Cfw^!r$JF{Xo%{z4<;r&wk+b-0)&PdgqQEMS*m8q1q7wA|h*9^hEN& z*ejw&!!64zESWOKp+s8rDQe{1Kgv&D-VTEQ>NJ2QtW;A$+ZnPJYDA0jc>@BIfiF@4 znZ`?0)ZMjkXjd;N~Mf5oGFa^d?HWMEt9IdsagOa>B zAyvmDDMV*kH}5@=Sx6<7gmXO_jHcA`XDI9CxK!6viufZe^JJW5A=@5pBogzngRB7- zYH=o9+4;iAVsPs9Bn`S2`EWvPCGk^bGurW8A$g%>3XX1KVr)S6XRSq!9i{KW$^3Z% zr~Ebd2^^fVZGMQGcT{oolR262r@hHmu@H*6<@wKs1o%G&jIjfA^Eji1&Y!s+z1UQH zPUfIWjUx*ht*B&XTWywISpjBDh{p0}SSW;8Hve&bog_hruY2saU3e8G8OeH%Co^QG zJ}r9|hakEX2nS|BLhrwtYfy)z919*Nq&mhpU9M!{?7lxaQ?)%!%{9*vQn>`hkPaJG zxm@-Y4ERcBAVotT}W!ireJs>rmZ;RVSVd`7=KubO5=K3{Y0JtT?@$watA>V{fDR zy``0;j}eqh-K3$Nk8NVj=^fxomZ<`7zRqx;9cyGEjjYn*kGjCxw>->97=wys+Qd^< zT_#T>zht*kBH2b``A6gYOrW%yi`i&? zl9}8>VSmlk%yk(@w0QpAfNYtN!UKZA&kCbjimfY!lb>uwa?+<{(f5zchmzHDWKdL+ zq7G#7Gx_RjU0=FG_&uzUzMBeUXbI42-RKSS768^OdD(_C_Bh51%zUe@j-+6Yn6%2U zs)TAVb`sM77t-qZz-J`h6(s!P&`$F>}f?HoO z$FI?$shnxmJa_40#7RI}*h=oPWc_H`G3(Xh?3?cz{oqy~zA-84d@*hL=|Lq~C9lPY zufWa5-LoeOlwVRswt7binPhs!H;=oi`wGzfgpR{=$^1d`kF9_6l1Uk45pKbIKOFH& zE6@8;LZZk}G5WalJ5?(8c^TNS9G`g$UA+)4o8!$UOdS7lhlQZ0Km9?Cgse+Cg~ByZo;avML9X=^#0At_;baUmQ}HJpC( z-;HK4fCkqzd}buRE}C;46f6BezeByFmi7}*zzAu>!C};l4`}N-$s)ut8rGWqLwj478lkKL_y8qhZSSLTG4H|j>_%YT4kf9Y=evW{~qU3LF0u51|%lF)*WQUcz8Znf1!BBDLQ zTt~*l&+WTlI%HX|ytNK)o-z9*P`?n@Op#}=a^igutxrsPn;`*|SBwbf5gYB1x_J|V zCe^QlF8%WOecrk^K!>%?_kgbdu*zagOUsOw2ed8Yb(T>`pcxrZTx7oo)f)RLY`rZv zDYDSI+N0VMpNIv^BW3-Uyw-QL)`VFuQBEs<;Kr`hnBYPbWmw% zD=kpv67;1X`U11(oyoSH8~Od8eu<2J5J4mo!Rzxd<{71L;27Wg*`0(VVLpG`eh{1$lj4u<)F|(@i{3$$T11|Gzf7WA8r9ss9gG$rYb2nkT(FWYqcA z_n)GvWfH5_tmGHJ{So0jM$)6BL)`P@74#nK;oJ+a;q+x6&;XT+>{R5I$&*@HaJ(Ne zqWF_mM=d6+TlIdae`Ylr)tCT|x0~`pq=cBuWRk4;CF^V2kE&&<)pRPzSs!%nDZ|fx zyM|jnaVD$QtmMFw}g!g^mCO&z~_mUb`kKY1$ zQZF-b--B;+$ITb8Zqr-z^zP%@D{o@%tc8hrtvE%*C~MZff5zj_{Ek4t&p8)d%cpPs zuY`>B2<_tkbqA~h;0ze*xb!&eUtjxXtEIrDFGN_B!Fy7wxf3Y__YQ={?()F zXl|X9AE;6a^yE}5sIq=mSe|(m=UVjQnl4z)FV&=3tR9Ds_NfO8VkFsgo zIzr(fR-1#He?I@8yyv28(l&w=3o6aW(CMRxe!ha0&;1rrj8a`y%iLKDsi~?X8i~?B zbd1g0*Aoc(F`F%1^ND}wqRT#A5KqO(O_pW6!+Q}UN_dpBdNL-9Tl1dL;3)pF9so=h z7mliKU7q@m9pLuQo~8iqfk$G1%agezlRAa*Yx4nBf5s1Lpvs4*p8ebT-7o*09b4Zf z97;|z7z_s76*Vk9=Q3{i%>Th+R^Jm*uttv+6t2-_2K#&Y{V%`C=Jl^C^Nq1e27>{o ztBS=--^WKkbr%k2Me1c`;Ym#{Azwd1uM&l0vDaX$XxDxDq-Y4=@Xi>|CR}y1Q>te( z+JEc-f46_`bp4(8c{8ta57nQt_$VAypv`N?q*>hg-9IA(;c$>c2XGY^=*9lo4*Ah6b^Fu;C3t)JMCRF(|IaVF1pud@FYb;guF_OogkR8 zI9t-MRROxeXa<7-vV<&0k)$vNgISZOG#{=If97<{HJH6Z>cZ@MoFLaF8u4SYD|2 zPx0SOsbBIof%$+c<)?}6Fm0X+=SQ8V(m_=UPkM4Gg{Oqpu~lIrxz|^9aFW@K3?_SH zf7WkV*U9N+X(lx{b0-;8sywM*RGjwIHi7x@QwmQL4XR?SOtA_`?WZ)8nmN~16mvNw zTzZG8*D2Nn{?7t>M!Wg+kt$DmCQ#>|6AJY7@5zLOd8kn+V~0(sxhdTQmLKRU*A>N6 z;h@T!r=o$bUMICI&7^7-7Rp@K&t)<&-oNP%h1oOGg*-Q$8J*NBKkz zYBVoA(IzmT@;rFb6M8P5h6iK&S@d}L7yXOMP5&3~dNp+~^gD0>0000 @@ -135,10 +135,10 @@ inkscape:collect="always" xlink:href="#linearGradient1157" id="linearGradient1159" - x1="103.25241" - y1="162.52734" + x1="96.024963" + y1="163.18439" x2="96.445732" - y2="116.77643" + y2="120.55441" gradientUnits="userSpaceOnUse" /> image/svg+xml - + @@ -265,7 +265,7 @@ λ + style="font-style:italic;fill:#c5a83a;fill-opacity:1;stroke-width:0.9375px;">λ λ + style="font-style:italic;fill:#c5a83a;fill-opacity:1;stroke-width:0.9375px;">λ λ + style="font-style:italic;fill:#c5a83a;fill-opacity:1;stroke-width:0.93749994px;">λ Date: Sun, 2 May 2021 17:36:30 +0300 Subject: [PATCH 091/832] mark a TODO --- unpythonic/syntax/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 3efc1b85..bad1c75d 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -78,6 +78,8 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. +# TODO: `aif`: `it` should only exist in the `then` and `otherwise` parts + # TODO: upgrade let_syntax block, expr into `mcpyrate` magic variables # TODO: also kw() in unpythonic.syntax.prefix From f35ce21ad2f841a11d65b4536769f3389c907550 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 17:43:47 +0300 Subject: [PATCH 092/832] `aif`: `it` should only exist in the `then` and `otherwise` parts --- CHANGELOG.md | 1 + unpythonic/syntax/__init__.py | 2 -- unpythonic/syntax/ifexprs.py | 25 ++++++++++++++----------- 3 files changed, 15 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 74d0a910..3adc83b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand **Fixed**: - Make `callsite_filename` ignore our call helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`). +- In `aif`, `it` is now only valid in the `then` and `otherwise` parts, as it should. --- diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index bad1c75d..3efc1b85 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -78,8 +78,6 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: `aif`: `it` should only exist in the `then` and `otherwise` parts - # TODO: upgrade let_syntax block, expr into `mcpyrate` magic variables # TODO: also kw() in unpythonic.syntax.prefix diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index 5551e40e..38e506d4 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -56,22 +56,25 @@ def aif(tree, *, syntax, expander, **kw): _aif_level = NestingLevelTracker() def _aif(tree, bindings_of_it): - with _aif_level.changed_by(+1): - # expand any `it` inside the `aif` (thus confirming those uses are valid) - def expand_it(tree): - return MacroExpander(bindings_of_it, dyn._macro_expander.filename).visit(tree) + # expand any `it` inside the `aif` (thus confirming those uses are valid) + def expand_it(tree): + return MacroExpander(bindings_of_it, dyn._macro_expander.filename).visit(tree) + # careful here: `it` is only valid in the `then` and `otherwise` parts. + test, then, otherwise = tree.elts + test = _implicit_do(test) + with _aif_level.changed_by(+1): name_of_it = list(bindings_of_it.keys())[0] expanded_it = expand_it(q[n[name_of_it]]) - tree = expand_it(tree) + then = _implicit_do(expand_it(then)) + otherwise = _implicit_do(expand_it(otherwise)) - test, then, otherwise = [_implicit_do(x) for x in tree.elts] - let_bindings = q[(a[expanded_it], a[test])] - let_body = q[a[then] if a[expanded_it] else a[otherwise]] - # We use a hygienic macro reference to `let[]` in the output, - # so that the expander can expand it later. - return q[h[let][a[let_bindings]][a[let_body]]] + let_bindings = q[(a[expanded_it], a[test])] + let_body = q[a[then] if a[expanded_it] else a[otherwise]] + # We use a hygienic macro reference to `let[]` in the output, + # so that the expander can expand it later. + return q[h[let][a[let_bindings]][a[let_body]]] @namemacro def it(tree, *, syntax, **kw): From 5eabcee66d4caf8fc02b82ac8480b52b19f887e4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 2 May 2021 17:54:33 +0300 Subject: [PATCH 093/832] update comment --- unpythonic/syntax/letdo.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index d03faee8..a7c22578 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -830,6 +830,8 @@ def _do(tree): # # TODO: We have to be careful with nested `do`, though - some local definitions may belong to # TODO: nested invocations. So perhaps we need to expand `do`, `do0`, `local` and `delete` here. + # TODO: Even that doesn't help, e.g.: do[..., let[...][[local[x << 42], ...]]] + # TODO: Here the `let` has an implicit `do`, which should be treated separately. with _do_level.changed_by(+1): tree = dyn._macro_expander.visit(tree) From 75bfebdd8ee228ab186cb7f5aa9349f2bbe186f0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:38:21 +0300 Subject: [PATCH 094/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 3efc1b85..34f55fa4 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -120,6 +120,8 @@ # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... +# TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. + # Re-exports - macro interfaces from .autocurry import autocurry # noqa: F401 from .autoref import autoref # noqa: F401 From 6415b2ce1459bc1c240ece9da1e10af04977b567 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:38:52 +0300 Subject: [PATCH 095/832] extract utility `is_unexpanded_expr_macro` --- unpythonic/syntax/lazify.py | 23 ++++---------------- unpythonic/syntax/nameutil.py | 40 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 22 deletions(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index b9432bc5..3ea7fced 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -5,16 +5,18 @@ from ast import (Lambda, FunctionDef, AsyncFunctionDef, Call, Name, Attribute, Starred, keyword, List, Tuple, Dict, Set, Subscript, Load) +from functools import partial from mcpyrate.quotes import macros, q, a, h # noqa: F401 from mcpyrate.astfixers import fix_ctx -from mcpyrate.quotes import capture_as_macro, is_captured_macro, is_captured_value, lookup_macro +from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.walkers import ASTTransformer from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, isx, getname, is_decorator, wrapwith) from .letdoutil import islet, isdo, ExpandedLetView +from .nameutil import is_unexpanded_expr_macro from ..lazyutil import Lazy, passthrough_lazy_args, force, force1, maybe_force_args from ..dynassign import dyn @@ -490,24 +492,7 @@ def _lazy(tree): _expanded_lazy_name = "Lazy" _our_lazy = capture_as_macro(lazy) def _lazyrec(tree): - def is_unexpanded_lazy(tree): - if not type(tree) is Subscript: - return False - if isx(tree.value, _unexpanded_lazy_name): - return True - - # hygienic captures and as-imports - key = is_captured_macro(tree) - if key: - name_node = lookup_macro(key) - elif type(tree) is Name: - name_node = tree - else: - return False - macrofunction = dyn._macro_expander.isbound(name_node.id) - if macrofunction is lazy: # does the macro binding (in the current expander) point to our macro definition? - return True - return False + is_unexpanded_lazy = partial(is_unexpanded_expr_macro, lazy, dyn._macro_expander) # This helper doesn't need to recurse, so we don't need `ASTTransformer` here. def transform(tree): diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index c2307e4b..b27d3a3e 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -5,12 +5,13 @@ with a unified API. """ -__all__ = ["isx", "getname"] +__all__ = ["isx", "getname", "is_unexpanded_expr_macro"] -from ast import Name, Attribute +from ast import Name, Attribute, Subscript +import sys from mcpyrate.core import Done -from mcpyrate.quotes import is_captured_value +from mcpyrate.quotes import is_captured_macro, is_captured_value, lookup_macro def isx(tree, x, accept_attr=True): """Test whether tree is a reference to the name ``x`` (str). @@ -96,3 +97,36 @@ def getname(tree, accept_attr=True): if accept_attr and type(tree) is Attribute: return tree.attr return None + +# TODO: This utility really wants to live in `mcpyrate`, as part of a macro destructuring subsystem. +# TODO: It needs to be made more general, to detect also macro invocations with args. +def is_unexpanded_expr_macro(macrofunction, expander, tree): + """Check whether `tree` is an expr macro invocation bound to `macrofunction` in `expander`. + + This accounts for hygienic macro captures and as-imports. + + If there is a match, return the subscript slice, i.e. the tree that would be passed + to the macro function by the expander if the macro was expanded normally. + + **CAUTION**: This function doesn't currently support detecting macros that + take macro arguments. + """ + if not type(tree) is Subscript: + return False + + # hygienic captures and as-imports + key = is_captured_macro(tree.value) + if key: + name_node = lookup_macro(key) + elif type(tree.value) is Name: + name_node = tree.value + else: + return False + + macro = expander.isbound(name_node.id) + if macro is macrofunction: + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + return tree.slice + else: + return tree.slice.value + return False From 87e4440e5aba277d0d822ee31a247eb718d4d144 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:39:19 +0300 Subject: [PATCH 096/832] compactify comment --- unpythonic/syntax/letdo.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index a7c22578..87b086a4 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -904,11 +904,10 @@ def _do0(tree): if type(tree) not in (Tuple, List): raise SyntaxError("do0 body: expected a sequence of comma-separated expressions") # pragma: no cover elts = tree.elts - # Use `local[]` as a hygienically captured macro. + # Use `local[]` and `do[]` as hygienically captured macros. newelts = [q[a[_our_local][_do0_result << a[elts[0]]]], # noqa: F821, local[] defines it inside the do[]. *elts[1:], q[_do0_result]] # noqa: F821 - # Use `do[]` as a hygienically captured macro. return q[a[_our_do][t[newelts]]] # do0[] is also just a do[] def _implicit_do(tree): From 674e5f0d1f45c1b1871965c4154888ada228e02b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:39:29 +0300 Subject: [PATCH 097/832] handle `local`, `delete` manually at top level of `do` The macros are now just stubs that raise `SyntaxError`. This handling makes them correctly report an error (at macro expansion time) if they appear anywhere but at the top level of a `do`. --- unpythonic/syntax/letdo.py | 62 +++++++++----------------------------- 1 file changed, 14 insertions(+), 48 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 87b086a4..3b88a76b 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -31,9 +31,7 @@ from mcpyrate.quotes import macros, q, u, n, a, t, h # noqa: F401 from mcpyrate import gensym, parametricmacro -from mcpyrate.markers import ASTMarker from mcpyrate.quotes import capture_as_macro, is_captured_value -from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer from ..dynassign import dyn @@ -43,7 +41,7 @@ from .letdoutil import (isenvassign, UnexpandedEnvAssignView, UnexpandedLetView, canonize_bindings) -from .nameutil import getname +from .nameutil import getname, is_unexpanded_expr_macro from .scopeanalyzer import scoped_transform # -------------------------------------------------------------------------------- @@ -627,7 +625,7 @@ def local(tree, *, syntax, **kw): """ if syntax != "expr": raise SyntaxError("local is an expr macro only") # pragma: no cover - return _local(tree) + raise SyntaxError("local[] is only valid at the top level of a do[] or do0[]") # pragma: no cover, not meant to hit the expander def delete(tree, *, syntax, **kw): """[syntax] Delete a previously declared local name in a "do". @@ -648,7 +646,7 @@ def delete(tree, *, syntax, **kw): """ if syntax != "expr": raise SyntaxError("delete is an expr macro only") # pragma: no cover - return _delete(tree) + raise SyntaxError("delete[] is only valid at the top level of a do[] or do0[]") # pragma: no cover, not meant to hit the expander def do(tree, *, syntax, expander, **kw): """[syntax, expr] Stuff imperative code into an expression position. @@ -796,63 +794,31 @@ def do0(tree, *, syntax, expander, **kw): # -------------------------------------------------------------------------------- # Syntax transformers -_do_level = NestingLevelTracker() # for checking validity of local[] and delete[] - -# Use `mcpyrate` ASTMarkers, so that the expander can do the dirty work of -# detecting macro invocations. Our `do[]` macro then only needs to detect -# instances of the appropriate markers. -class UnpythonicLetDoMarker(ASTMarker): - """AST marker related to unpythonic's let/do subsystem.""" -class UnpythonicDoLocalMarker(UnpythonicLetDoMarker): - """AST marker for local variable definitions in a `do` context.""" -class UnpythonicDoDeleteMarker(UnpythonicLetDoMarker): - """AST marker for local variable deletion in a `do` context.""" - -def _local(tree): # syntax transformer - if _do_level.value < 1: - raise SyntaxError("local[] is only valid within a do[] or do0[]") # pragma: no cover - return UnpythonicDoLocalMarker(tree) - -def _delete(tree): # syntax transformer - if _do_level.value < 1: - raise SyntaxError("delete[] is only valid within a do[] or do0[]") # pragma: no cover - return UnpythonicDoDeleteMarker(tree) - def _do(tree): if type(tree) not in (Tuple, List): raise SyntaxError("do body: expected a sequence of comma-separated expressions") # pragma: no cover, let's not test the macro expansion errors. - # Handle nested `local[]`/`delete[]`. This will also expand any other nested macro invocations. - # TODO: If we want to make `do` an outside-in macro, instantiate another expander here and register - # TODO: only the `local` and `delete` transformers to it - grabbing them from the current expander's - # TODO: bindings to respect as-imports. (Expander instances are cheap in `mcpyrate`.) - # TODO: Grep the `unpythonic` codebase (and `mcpyrate` demos) for `MacroExpander` to see how. - # - # TODO: We have to be careful with nested `do`, though - some local definitions may belong to - # TODO: nested invocations. So perhaps we need to expand `do`, `do0`, `local` and `delete` here. - # TODO: Even that doesn't help, e.g.: do[..., let[...][[local[x << 42], ...]]] - # TODO: Here the `let` has an implicit `do`, which should be treated separately. - with _do_level.changed_by(+1): - tree = dyn._macro_expander.visit(tree) - e = gensym("e") envset = q[n[f"{e}._set"]] # use internal _set to allow new definitions envdel = q[n[f"{e}.pop"]] + islocaldef = partial(is_unexpanded_expr_macro, local, dyn._macro_expander) + isdelete = partial(is_unexpanded_expr_macro, delete, dyn._macro_expander) + def find_localdefs(tree): class LocaldefCollector(ASTTransformer): def transform(self, tree): if is_captured_value(tree): return tree # don't recurse! - if isinstance(tree, UnpythonicDoLocalMarker): - expr = tree.body + expr = islocaldef(tree) + if expr: if not isenvassign(expr): raise SyntaxError("local[...] takes exactly one expression of the form 'name << value'") # pragma: no cover view = UnexpandedEnvAssignView(expr) self.collect(view.name) - # e.g. `x << 21`; preserve the original expr to make the assignment occur. - return self.visit(expr) # handle nested local[] (e.g. from `do0[local[y << 5],]`) - return self.generic_visit(tree) + view.value = self.visit(view.value) # nested local[] (e.g. from `do0[local[y << 5],]`) + return expr # e.g. `x << 21`; preserve the original expr to make the assignment occur. + return tree # don't recurse! c = LocaldefCollector() tree = c.visit(tree) return tree, c.collected @@ -861,13 +827,13 @@ class DeleteCollector(ASTTransformer): def transform(self, tree): if is_captured_value(tree): return tree # don't recurse! - if isinstance(tree, UnpythonicDoDeleteMarker): - expr = tree.body + expr = isdelete(tree) + if expr: if type(expr) is not Name: raise SyntaxError("delete[...] takes exactly one name") # pragma: no cover self.collect(expr.id) return q[a[envdel](u[expr.id])] # -> e.pop(...) - return self.generic_visit(tree) + return tree # don't recurse! c = DeleteCollector() tree = c.visit(tree) return tree, c.collected From cd6dd3e3334307c8a91b4d2220d7027e90604799 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:42:41 +0300 Subject: [PATCH 098/832] oops, fix item level --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3adc83b4..cdff0f1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,7 +36,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. - - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. +- Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. **Fixed**: From d9d3907ae2d066639fb8a50ac50187b4b0366800 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:54:00 +0300 Subject: [PATCH 099/832] update changelog --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index cdff0f1d..361a47cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). - CI: Test coverage improved to 94%. +- Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) **Breaking changes**: @@ -33,6 +34,9 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import. Now you **must** import also the macro `f` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `f` is currently bound to. - **Rename the `curry` macro** to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. +- The `do[]` and `do0[]` macros now expand outside-in. The main differences from a user perspective are: + - Any source code captures (such as those performed by `test[]`) show the expanded output of `do` and `do0`, because that's what they receive. + - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. - Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. From 36281359db14b1746a4179f5c4d4787700f61c37 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 01:59:57 +0300 Subject: [PATCH 100/832] mark a TODO --- unpythonic/syntax/testingtools.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 9ac26f54..db03d422 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -424,6 +424,8 @@ def warn(tree, *, syntax, expander, **kw): # noqa: F811 # the function `unpythonic.conditions.error`, because a macro invocation # is an `ast.Subscript`, whereas a function call is an `ast.Call`. # TODO: Maybe these lists should be public, autoref already uses the list of functions. +# TODO: We should use `unpythonic.syntax.nameutil.is_unexpanded_expr_macro` to detect +# TODO: macro invocations, to respect as-imports. But it needs some bells and whistles first. _test_asserter_names = ["test", "test_signals", "test_raises", "error", "fail", "warn"] _test_function_names = ["unpythonic_assert", "unpythonic_assert_signals", From a5436812c7834eca5860c75d153074cb1705bb55 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 02:06:32 +0300 Subject: [PATCH 101/832] document difference between let and walrus-assignment --- doc/macros.md | 2 ++ unpythonic/syntax/__init__.py | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index ee7ffe4d..77ec2781 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -213,6 +213,8 @@ let[(x, y), # v0.12.0+ #### Notes +The main difference of the `let` family to Python's own named expressions (a.k.a. walrus operator, added in Python 3.8) is that `x := 42` does not create a scope, but `let[(x, 42)][...]` does. The walrus operator assigns to the name `x` in the scope it appears in, whereas in the `let` expression, the `x` only exists in that expression. + ``let`` and ``letrec`` expand into the ``unpythonic.lispylet`` constructs, implicitly inserting the necessary boilerplate: the ``lambda e: ...`` wrappers, quoting variable names in definitions, and transforming ``x`` to ``e.x`` for all ``x`` declared in the bindings. Assignment syntax ``x << 42`` transforms to ``e.set('x', 42)``. The implicit environment parameter ``e`` is actually named using a gensym, so lexically outer environments automatically show through. ``letseq`` expands into a chain of nested ``let`` expressions. Nesting utilizes an inside-out macro expansion order: diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 34f55fa4..21912f92 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -85,8 +85,6 @@ # TODO: Now that `unpythonic` provides dialects, update `mcpyrate` docs. -# TODO: `let` constructs: document difference to Python 3.8 walrus operator (`let` creates a scope, `:=` doesn't) - # TODO: `make_dynvar` needs to be better advertised in the docs. A workflow example would also be nice. # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? From f8e1e9c311a9f1534febbd751fece5c4dbbc1e5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 02:35:20 +0300 Subject: [PATCH 102/832] improve dynassign docs, especially advertise `make_dynvar` --- doc/features.md | 27 ++++++++++++++++++--------- unpythonic/syntax/__init__.py | 2 -- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/doc/features.md b/doc/features.md index d6159ba4..a8508abd 100644 --- a/doc/features.md +++ b/doc/features.md @@ -284,14 +284,17 @@ Like global variables, but better-behaved. Useful for sending some configuration There's a singleton, `dyn`: ```python -from unpythonic import dyn +from unpythonic import dyn, make_dynvar + +make_dynvar(c=17) # top-level default value def f(): # no "a" in lexical scope here assert dyn.a == 2 def g(): - with dyn.let(a=2, b="foo"): + with dyn.let(a=2, b="foo", c=42): assert dyn.a == 2 + assert dyn.c == 42 f() @@ -301,17 +304,24 @@ def g(): # now "a" has reverted to its previous value assert dyn.a == 2 + assert dyn.c == 17 # "c" has reverted to its default value print(dyn.b) # AttributeError, dyn.b no longer exists g() ``` -Dynamic variables are set using `with dyn.let(...)`. There is no `set`, `<<`, unlike in the other `unpythonic` environments. +Dynamic variables (a.k.a. *dynvars*) are created using `with dyn.let(k0=v0, ...)`. The syntax is in line with the nature of the assignment, which is in effect *for the dynamic extent* of the `with`. Exiting the `with` block pops the dynamic environment stack. Inner dynamic environments shadow outer ones. + +The point of dynamic assignment is that dynvars are seen also by code that is outside the lexical scope where the `with dyn.let` resides. The use case is to avoid a function parameter definition cascade, when you need to pass some information through several layers that don't care about it. This is especially useful for passing "background" information, such as plotter settings in scientific visualization, or the macro expander instance in metaprogramming. + +To give a dynvar a top-level default value, use ``make_dynvar(k0=v0, ...)``. Usually this is done at the top-level scope of the module for which that dynvar is meaningful. Each dynvar, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). -**Changed in v0.14.2.** *To bring this in line with [SRFI-39](https://srfi.schemers.org/srfi-39/srfi-39.html), `dyn` now supports rebinding, using assignment syntax such as `dyn.x = 42`. For an atomic mass-update, see `dyn.update`. Rebinding occurs in the closest enclosing dynamic environment that has the target name bound. If the name is not bound in any dynamic environment, ``AttributeError`` is raised.* +To rebind existing dynvars, use `dyn.k = v`, or `dyn.update(k0=v0, ...)`. Rebinding occurs in the closest enclosing dynamic environment that has the target name bound. If the name is not bound in any dynamic environment (including the top-level one), ``AttributeError`` is raised. -**CAUTION**: Use rebinding of dynamic variables carefully, if at all. Stealth updates of dynamic variables defined in an enclosing dynamic extent can destroy any chance of statically reasoning about the code. +**CAUTION**: Use rebinding of dynvars carefully, if at all. Stealth updates of dynvars defined in an enclosing dynamic extent can destroy any chance of statically reasoning about the code. -The values of dynamic variables remain bound for the dynamic extent of the `with` block. Exiting the `with` block then pops the stack. Inner dynamic scopes shadow outer ones. Dynamic variables are seen also by code that is outside the lexical scope where the `with dyn.let` resides. +There is no `set` function or `<<` operator, unlike in the other `unpythonic` environments. + +**Changed in v0.14.2.** *To bring this in line with [SRFI-39](https://srfi.schemers.org/srfi-39/srfi-39.html), `dyn` now supports rebinding, using assignment syntax such as `dyn.x = 42`, and the function `dyn.update(x=42, y=17, ...)`.*

Each thread has its own dynamic scope stack. There is also a global dynamic scope for default values, shared between threads. @@ -322,9 +332,7 @@ The source of the copy is always the main thread mainly because Python's `thread Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (after the global definition has been executed).
-To create a dynvar and set its default value, use ``make_dynvar``. Each dynamic variable, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). - -See also the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a live view to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. ``dyn`` also provides the ``collections.abc.Mapping`` API. +For more details, see the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a *live view* to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. ``dyn`` also provides the ``collections.abc.Mapping`` API. To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, so you can do things like: @@ -346,6 +354,7 @@ On Common Lisp's special variables, see [Practical Common Lisp by Peter Seibel]( So what we have in `dyn` is almost exactly like Common Lisp's special variables, except we are missing convenience features such as `setf` and a smart `let` that auto-detects whether a variable is lexical or dynamic (if the name being bound is already in scope). + ## Containers We provide some additional containers. diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 21912f92..da8ac61f 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -85,8 +85,6 @@ # TODO: Now that `unpythonic` provides dialects, update `mcpyrate` docs. -# TODO: `make_dynvar` needs to be better advertised in the docs. A workflow example would also be nice. - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. From 7ada2cae9f3b72109a936f7ddc16a998c6d2be50 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 02:36:17 +0300 Subject: [PATCH 103/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index da8ac61f..bc545272 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -94,7 +94,7 @@ # TODO: macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" -# TODO: `isx` and `getname` from `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead +# TODO: something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead # TODO: `mcpyrate` does not auto-expand macros in quasiquoted code. # - Consider when we should expand macros in quoted code and when not From b31a676e0fc3efeaa5bd3853569d76bd65d972a3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 02:40:44 +0300 Subject: [PATCH 104/832] update lispylet note --- doc/features.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index a8508abd..efb74423 100644 --- a/doc/features.md +++ b/doc/features.md @@ -181,7 +181,7 @@ This has been fixed in Python 3.6, see [PEP 468](https://www.python.org/dev/peps **NOTE**: This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use. Below is the documentation for the raw API. -If you need **guaranteed left-to-right initialization** of `letrec` bindings in Pythons older than 3.6, there is also an alternative implementation for all the `let` constructs, with positional syntax and more parentheses. The only difference is the syntax; the behavior is identical with the default implementation. +The `lispylet` module was originally created to allow guaranteed left-to-right initialization of `letrec` bindings in Pythons older than 3.6, hence the positional syntax and more parentheses. The only difference is the syntax; the behavior is identical with the other implementation. As of 0.15, the main role of `lispylet` is to act as the run-time backend for the `let` family of macros. These constructs are available in the top-level `unpythonic` namespace, with the ``ordered_`` prefix: ``ordered_let``, ``ordered_letrec``, ``ordered_dlet``, ``ordered_dletrec``, ``ordered_blet``, ``ordered_bletrec``. @@ -192,7 +192,7 @@ from unpythonic.lispylet import * # override the default "let" implementation letrec((('a', 1), ('b', lambda e: - e.a + 1)), # may refer to any bindings above it in the same letrec, also in Python < 3.6 + e.a + 1)), # may refer to any bindings above it in the same letrec lambda e: e.b) # --> 2 @@ -208,8 +208,6 @@ letrec((("evenp", lambda e: The syntax is `let(bindings, body)` (respectively `letrec(bindings, body)`), where `bindings` is `((name, value), ...)`, and `body` is like in the default variants. The same rules concerning `name` and `value` apply. -The let macros internally use this *lispylet* implementation. - ### ``env``: the environment From 80204c29c09c94898345744e4f1e309d9458d00b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 02:54:46 +0300 Subject: [PATCH 105/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index bc545272..3cf8e923 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -83,8 +83,6 @@ # TODO: let_syntax block, expr: syntactic consistency: change parentheses to brackets -# TODO: Now that `unpythonic` provides dialects, update `mcpyrate` docs. - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. From 487bad23b255603969399e794a144f4e27fb80da Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 02:55:27 +0300 Subject: [PATCH 106/832] remove done TODO --- unpythonic/syntax/__init__.py | 1 - 1 file changed, 1 deletion(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 3cf8e923..3d2e9389 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -86,7 +86,6 @@ # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. -# TODO: Test the q[t[...]] implementation in do0[] # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. From 364272f0762dff87e35d47d6432352d1aa470dcb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 03:05:57 +0300 Subject: [PATCH 107/832] fix error message wording --- unpythonic/syntax/ifexprs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index 38e506d4..ed2a7de5 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -86,7 +86,7 @@ def it(tree, *, syntax, **kw): if syntax != "name": raise SyntaxError("`it` is a name macro only") if _aif_level.value < 1: - raise SyntaxError("`it` may only appear within an `aif[...]`") + raise SyntaxError("`it` may only appear in the 'then' and 'otherwise' parts of an `aif[...]`") return tree # -------------------------------------------------------------------------------- From 619d1049f3fb036f36b28aab84be1966ac55db5b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 03:09:31 +0300 Subject: [PATCH 108/832] move comment --- unpythonic/syntax/nameutil.py | 67 ++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 33 deletions(-) diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index b27d3a3e..109cc7c1 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -13,6 +13,40 @@ from mcpyrate.core import Done from mcpyrate.quotes import is_captured_macro, is_captured_value, lookup_macro +# Here hygienic captures only come from `unpythonic.syntax` (unless there are +# also user-defined macros), and we use from-imports and bare names for anything +# `q[h[]]`'d; but any references that appear explicitly in the user code may use +# either bare `somename` or `unpythonic.somename`. +# +# TODO: How about `unpythonic.somemodule.somename`? Currently not detected. +# +# Note that in `mcpyrate`, a hygienic capture can contain the value of an +# arbitrary expression, which does not need to be bound to a name. In that +# case the "name" will be the unparsed source code of the expression. See +# the implementation of `mcpyrate.quotes.h`. That's harmless here since +# an expression won't produce an exact match on the name. +# +# Here we're mainly interested in the case where we have captured the value +# a name had at the use site of `h[]`, and even then, we just look at the name, +# not the actual value. +# +# TODO: Let's look at the value, not just the name. Requires changes to use sites, +# TODO: because currently `isx` doesn't know about the value the caller wants to +# TODO: check against. +# +# TODO: For our use cases, that value is usually a syntax transformer function +# TODO: defined somewhere in `unpythonic.syntax`, so we can use things like +# TODO: `q[h[letter]]` or `q[h[dof]]` in the let/do constructs to ensure that +# TODO: the workhorses resolve correctly at the use site, and still be able +# TODO: to detect the expanded forms of those constructs in the AST. +# +# TODO: The run-time value can be obtained at this end by +# TODO: `value = mcpyrate.quotes.lookup_value(key)`, +# TODO: provided that `key and (key[1] is not None)`. +# TODO: If the second element of the key is `None`, it means that +# TODO: program execution hasn't yet reached the point where the +# TODO: actual value capture triggers for that particular use of `h[]`. + def isx(tree, x, accept_attr=True): """Test whether tree is a reference to the name ``x`` (str). @@ -37,39 +71,6 @@ def isx(tree, x, accept_attr=True): """ if isinstance(tree, Done): return isx(tree.body, x, accept_attr=accept_attr) - # Here hygienic captures only come from `unpythonic.syntax` (unless there are - # also user-defined macros), and we use from-imports and bare names for anything - # `q[h[]]`'d; but any references that appear explicitly in the user code may use - # either bare `somename` or `unpythonic.somename`. - # - # TODO: How about `unpythonic.somemodule.somename`? Currently not detected. - # - # Note that in `mcpyrate`, a hygienic capture can contain the value of an - # arbitrary expression, which does not need to be bound to a name. In that - # case the "name" will be the unparsed source code of the expression. See - # the implementation of `mcpyrate.quotes.h`. That's harmless here since - # an expression won't produce an exact match on the name. - # - # Here we're mainly interested in the case where we have captured the value - # a name had at the use site of `h[]`, and even then, we just look at the name, - # not the actual value. - # - # TODO: Let's look at the value, not just the name. Requires changes to use sites, - # TODO: because currently `isx` doesn't know about the value the caller wants to - # TODO: check against. - # - # TODO: For our use cases, that value is usually a syntax transformer function - # TODO: defined somewhere in `unpythonic.syntax`, so we can use things like - # TODO: `q[h[letter]]` or `q[h[dof]]` in the let/do constructs to ensure that - # TODO: the workhorses resolve correctly at the use site, and still be able - # TODO: to detect the expanded forms of those constructs in the AST. - # - # TODO: The run-time value can be obtained at this end by - # TODO: `value = mcpyrate.quotes.lookup_value(key)`, - # TODO: provided that `key and (key[1] is not None)`. - # TODO: If the second element of the key is `None`, it means that - # TODO: program execution hasn't yet reached the point where the - # TODO: actual value capture triggers for that particular use of `h[]`. key = is_captured_value(tree) # AST -> (name, frozen_value) or False if key: name, frozen_value = key From 38e28c05df78601ea69bcf364b94a807240c56bd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 03:09:39 +0300 Subject: [PATCH 109/832] implement `isx` in terms of `getname` --- unpythonic/syntax/nameutil.py | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index 109cc7c1..70bffe94 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -69,16 +69,9 @@ def isx(tree, x, accept_attr=True): - ``x`` as an attribute (if ``accept_attr=True``) """ - if isinstance(tree, Done): - return isx(tree.body, x, accept_attr=accept_attr) - key = is_captured_value(tree) # AST -> (name, frozen_value) or False - if key: - name, frozen_value = key - ismatch = x if callable(x) else lambda name: name == x - return ((type(tree) is Name and ismatch(tree.id)) or - (key and ismatch(name)) or - (accept_attr and type(tree) is Attribute and ismatch(tree.attr))) + thename = getname(tree, accept_attr=accept_attr) + return thename is not None and ismatch(thename) def getname(tree, accept_attr=True): """The cousin of ``isx``. From 4b3eb874a00a7c09d981798994060d916cebe780 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 03:14:30 +0300 Subject: [PATCH 110/832] terminology --- unpythonic/syntax/letdoutil.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index c968f937..e2300ff6 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -303,7 +303,7 @@ class UnexpandedLetView: let[body, where((k0, v0), ...)] # haskelly expression, inverted Lispy expressions are supported also using the old parenthesis syntax - to pass macro parameters:: + to pass macro arguments:: let((k0, v0), ...)[body] # lispy expression From 715e0e6e31d713fbbc7869fe80e4fafb6a9115c4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 03:14:41 +0300 Subject: [PATCH 111/832] Python 3.9+: deprecation tag for parenthesis syntax for macro arguments --- unpythonic/syntax/letdoutil.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index e2300ff6..b351efda 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -127,7 +127,7 @@ def islet(tree, expanded=True): s = tree.value.id if any(s == x for x in deconames): return ("decorator", s) - if type(tree) is Call and type(tree.func) is Name: # up to Python 3.8: parenthesis syntax for decorator macros + if type(tree) is Call and type(tree.func) is Name: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 s = tree.func.id if any(s == x for x in deconames): return ("decorator", s) @@ -150,7 +150,7 @@ def islet(tree, expanded=True): s = macro.value.id if any(s == x for x in exprnames): return ("lispy_expr", s) - elif type(macro) is Call and type(macro.func) is Name: # alternative parenthesis syntax to pass macro arguments + elif type(macro) is Call and type(macro.func) is Name: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 s = macro.func.id if any(s == x for x in exprnames): return ("lispy_expr", s) @@ -376,7 +376,7 @@ def _theexpr_ref(self): def _getbindings(self): t = self._type if t == "decorator": # bare Subscript, dlet[...], blet[...] - if type(self._tree) is Call: # up to Python 3.8: parenthesis syntax for decorator macros + if type(self._tree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 return canonize_bindings(self._tree.args) # Subscript as decorator (Python 3.9+) if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. @@ -391,7 +391,8 @@ def _getbindings(self): theargs = self._tree.value.slice.elts else: theargs = self._tree.value.slice.value.elts - # Call inside a Subscript, (let(...))[...], parenthesis syntax to pass macro arguments + # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + # Call inside a Subscript, (let(...))[...] else: # type(self._tree.value) is Call: theargs = self._tree.value.args return canonize_bindings(theargs) @@ -404,7 +405,7 @@ def _getbindings(self): def _setbindings(self, newbindings): t = self._type if t == "decorator": - if type(self._tree) is Call: # up to Python 3.8: parenthesis syntax for decorator macros + if type(self._tree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 self._tree.args = newbindings return # Subscript as decorator (Python 3.9+) @@ -419,7 +420,8 @@ def _setbindings(self, newbindings): self._tree.value.slice.elts = newbindings else: self._tree.value.slice.value.elts = newbindings - # Call inside a Subscript, (let(...))[...], parenthesis syntax to pass macro arguments + # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + # Call inside a Subscript, (let(...))[...] else: # type(self._tree.value) is Call: self._tree.value.args = newbindings else: From fefbd95516c71ac798af8d6adf0f43792a8a26cd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 04:08:14 +0300 Subject: [PATCH 112/832] oops --- unpythonic/syntax/ifexprs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index ed2a7de5..f53da7de 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -45,7 +45,7 @@ def aif(tree, *, syntax, expander, **kw): raise SyntaxError("aif is an expr macro only") # Detect the name(s) of `it` at the use site (this accounts for as-imports) - macro_bindings = extract_bindings(dyn._macro_expander.bindings, it) + macro_bindings = extract_bindings(expander.bindings, it) if not macro_bindings: raise SyntaxError("The use site of `aif` must macro-import `it`, too.") From 603c88649a4226251400942980f665ad40e67f40 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 04:08:24 +0300 Subject: [PATCH 113/832] `aif`: fix `it` binding issue --- unpythonic/syntax/ifexprs.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index f53da7de..15f6f637 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -45,6 +45,8 @@ def aif(tree, *, syntax, expander, **kw): raise SyntaxError("aif is an expr macro only") # Detect the name(s) of `it` at the use site (this accounts for as-imports) + # TODO: We don't know which binding this particular use site uses. + # TODO: For now, we hack this by making `it` always rename itself to literal `it`. macro_bindings = extract_bindings(expander.bindings, it) if not macro_bindings: raise SyntaxError("The use site of `aif` must macro-import `it`, too.") @@ -64,6 +66,8 @@ def expand_it(tree): test, then, otherwise = tree.elts test = _implicit_do(test) with _aif_level.changed_by(+1): + # TODO: We don't know which binding this particular use site uses. + # TODO: For now, we hack this by making `it` always rename itself to literal `it`. name_of_it = list(bindings_of_it.keys())[0] expanded_it = expand_it(q[n[name_of_it]]) @@ -82,12 +86,15 @@ def it(tree, *, syntax, **kw): Inside an `aif` body, evaluates to the value of the test result. Anywhere else, is considered a syntax error. + + **CAUTION**: Currently cannot be as-imported; must be imported + without renaming. """ if syntax != "name": raise SyntaxError("`it` is a name macro only") if _aif_level.value < 1: raise SyntaxError("`it` may only appear in the 'then' and 'otherwise' parts of an `aif[...]`") - return tree + return q[it] # always rename to literal `it` # -------------------------------------------------------------------------------- From 07f884d1b2b7bd796657df0e7760e37a78fa3120 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 04:08:42 +0300 Subject: [PATCH 114/832] `continuations`: output `aif` in boolop as a hygienically captured macro --- unpythonic/syntax/tailtools.py | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index df4fbaab..ded28538 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -22,17 +22,17 @@ from mcpyrate import gensym from mcpyrate.markers import ASTMarker -from mcpyrate.quotes import is_captured_value +from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer, ASTVisitor from .astcompat import getconstant, NameConstant +from .ifexprs import aif, it from .util import (isx, isec, detect_callec, detect_lambda, has_tco, sort_lambda_decorators, suggest_decorator_index, ExpandedContinuationsMarker, wrapwith, isexpandedmacromarker) from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView -from .ifexprs import _aif, it as aif_it from ..dynassign import dyn from ..it import uniqify @@ -40,6 +40,16 @@ from ..tco import trampolined, jump from ..lazyutil import passthrough_lazy_args +# In `continuations`, we use `aif` and `it` as hygienically captured macros. +# Note the difference between `aif[..., it, ...]` and `q[a[_our_aif][..., a[_our_it], ...]]`. +# +# If `it` is bound in the current expander, even *mentioning* it outside an `aif` is a syntax error, by design. +# +# When constructing a quasiquoted tree that invokes `aif[]`, we can splice in a hygienic reference to `it` +# as `a[_our_it]` without even having the macro bound in the expander that expands *this* module. +_our_aif = capture_as_macro(aif) +_our_it = capture_as_macro(it) + # -------------------------------------------------------------------------------- # Macro interface @@ -1254,18 +1264,15 @@ def transform(tree): else: op_of_others = tree.values[0] if type(tree.op) is Or: - # or(data1, ..., datan, tail) --> it if any(others) else tail - tree = _aif(Tuple(elts=[op_of_others, - transform_data(Name(id="it")), - transform(tree.values[-1])]), # tail-call item - {'it': aif_it}) + # or(data1, ..., datan, tail) --> aif[any(others), it, tail] + tree = q[a[_our_aif][a[op_of_others], + a[transform_data(_our_it)], + a[transform(tree.values[-1])]]] # tail-call item elif type(tree.op) is And: # and(data1, ..., datan, tail) --> tail if all(others) else False - fal = q[False] - fal = copy_location(fal, tree) - tree = IfExp(test=op_of_others, - body=transform(tree.values[-1]), - orelse=transform_data(fal)) + tree = q[a[transform(tree.values[-1])] + if a[op_of_others] + else a[transform_data(q[False])]] else: # cannot happen raise SyntaxError(f"unknown BoolOp type {tree.op}") # pragma: no cover else: # optimization: BoolOp, no call or compound in tail position --> treat as single data item From 733d0759346eb3d2c15e67759749a350c18110fd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 04:09:39 +0300 Subject: [PATCH 115/832] make the dummy expander error out on any attribute access --- unpythonic/syntax/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 3d2e9389..43e70045 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -147,6 +147,6 @@ # We use `dyn` to pass the `expander` parameter to the macro implementations. class _NoExpander: - def visit(self, tree): + def __getattr__(self, k): # Make the dummy error out whenever we attempt to do anything with it. raise NotImplementedError("Macro expander instance has not been set in `dyn`.") make_dynvar(_macro_expander=_NoExpander()) From 3762f99e19e6c1f5d2e6ecd157ea6c749f71a8b3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 3 May 2021 04:13:46 +0300 Subject: [PATCH 116/832] update changelog --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 361a47cd..20c359f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,8 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). - CI: Test coverage improved to 94%. -- Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) +- Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. The `continuations` macro also outputs a hygienically captured `aif` when transforming an `or` expression that occurs in tail position. + - This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) **Breaking changes**: From fa4a5c4406ce8d788d01626eac4631ebbb0c9fa6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:03:28 +0300 Subject: [PATCH 117/832] add `with expand_testing_macros_first`, which does exactly that. --- unpythonic/syntax/__init__.py | 3 +- unpythonic/syntax/testingtools.py | 47 +++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 43e70045..03763ba1 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -135,7 +135,8 @@ continuations, call_cc) from .testingtools import (the, test, # noqa: F401 test_signals, test_raises, - fail, error, warn) + fail, error, warn, + expand_testing_macros_first) # Re-exports - regular code from .dbg import dbgprint_block, dbgprint_expr # noqa: F401, re-export for re-use in a decorated variant. diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index db03d422..cb542869 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -7,12 +7,15 @@ __all__ = ["the", "test", "test_signals", "test_raises", "fail", "error", "warn", + "expand_testing_macros_first", "isunexpandedtestmacro", "isexpandedtestmacro", "istestmacro"] from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym, parametricmacro, unparse +from mcpyrate.expander import MacroExpander from mcpyrate.quotes import is_captured_value +from mcpyrate.utils import extract_bindings from mcpyrate.walkers import ASTTransformer from ast import Tuple, Subscript, Name, Call, copy_location, Compare, arg, Return, parse, Expr, AST @@ -417,6 +420,50 @@ def warn(tree, *, syntax, expander, **kw): # noqa: F811 with dyn.let(_macro_expander=expander): return _warn_expr(tree) +# TODO: There's also `quicklambda`. Maybe add a general utility for this kind of thing to `mcpyrate.metatools`? +def expand_testing_macros_first(tree, *, syntax, expander, **kw): + """[syntax, block] Force testing framework macros to expand first. + + Usage:: + + with expand_testing_macros_first: + ... + + This is useful if you have your own block macro that expands outside in and + does some code-walking transformations, and some tests inside such a block. + Expanding the test macros first allows the test framework to capture the + unexpanded source code for error reporting. + + As an example, consider:: + + with your_block_macro: + test[expr] + + In this case, if `your_block_macro` expands outside-in, it will transform the + `expr` inside the `test[expr] before `test` even sees the AST. If the test + fails or errors, the error message will contain the expanded version of `expr`, + not the original one. Now, if we change the example to:: + + with expand_testing_macros_first: + with your_block_macro: + test[expr] + + In this case, `expand_testing_macros_first` arranges things so that `test[expr]` + expands first (even if `your_block_macro` expands outside-in), so it will see + the original, unexpanded AST. + + This does imply that `your_block_macro` will then receive the expanded form of + `test[expr]` as input, but that's macros for you. Macros don't compose, after all. + """ + if syntax != "block": + raise SyntaxError("expand_testing_macros_first is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("expand_testing_macros_first does not take an asname") + + testing_macros = [test, test_signals, test_raises, error, fail, warn] + macro_bindings = extract_bindings(expander.bindings, *testing_macros) + return MacroExpander(macro_bindings, filename=expander.filename).visit(tree) + # ----------------------------------------------------------------------------- # Helpers for other macros to detect uses of the ones we defined here. From 545355ff78a5444ce3f2a34246e0d35417d36612 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:05:34 +0300 Subject: [PATCH 118/832] validate asname in block macros --- unpythonic/syntax/autocurry.py | 2 ++ unpythonic/syntax/autoref.py | 2 ++ unpythonic/syntax/dbg.py | 2 ++ unpythonic/syntax/lambdatools.py | 8 ++++++++ unpythonic/syntax/lazify.py | 2 ++ unpythonic/syntax/letsyntax.py | 4 ++++ unpythonic/syntax/nb.py | 2 ++ unpythonic/syntax/prefix.py | 2 ++ unpythonic/syntax/tailtools.py | 6 ++++++ unpythonic/syntax/testingtools.py | 6 ++++++ 10 files changed, 36 insertions(+) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index bcd3c9f5..7b2d2ea4 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -65,6 +65,8 @@ def add3(a, b, c): """ if syntax != "block": raise SyntaxError("autocurry is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("autocurry does not take an asname") tree = expander.visit(tree) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index cfad6d1f..612608c9 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -127,6 +127,8 @@ def autoref(tree, *, args, syntax, expander, **kw): raise SyntaxError("autoref requires an argument, the object to be auto-referenced") target = kw.get("optional_vars", None) + if target and type(target) is not Name: # tuples not accepted + raise SyntaxError("with autoref[...] as ... takes at most one asname") with dyn.let(_macro_expander=expander): return _autoref(block_body=tree, args=args, asname=target) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 5e629b39..06cb4ce3 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -100,6 +100,8 @@ def dbg(tree, *, args, syntax, expander, **kw): """ if syntax not in ("expr", "block"): raise SyntaxError("dbg is an expr and block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("dbg (block mode) does not take an asname") tree = expander.visit(tree) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 479d0e65..de369cb2 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -65,6 +65,8 @@ def multilambda(tree, *, syntax, expander, **kw): """ if syntax != "block": raise SyntaxError("multilambda is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("multilambda does not take an asname") # Expand outside in. # multilambda should expand first before any let[], do[] et al. that happen @@ -116,6 +118,8 @@ def namedlambda(tree, *, syntax, expander, **kw): """ if syntax != "block": raise SyntaxError("namedlambda is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("namedlambda does not take an asname") # Two-pass macro. We pass in the expander to allow the macro to decide when to recurse. with dyn.let(_macro_expander=expander): @@ -187,6 +191,8 @@ def quicklambda(tree, *, syntax, expander, **kw): """ if syntax != "block": raise SyntaxError("quicklambda is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("quicklambda does not take an asname") # This macro expands outside in. # @@ -220,6 +226,8 @@ def foo(n): """ if syntax != "block": raise SyntaxError("envify is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("envify does not take an asname") # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 3ea7fced..e3ac482d 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -411,6 +411,8 @@ def doit(): """ if syntax != "block": raise SyntaxError("lazify is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("lazify does not take an asname") # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 40173800..62e6f714 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -128,6 +128,8 @@ def let_syntax(tree, *, args, syntax, expander, **kw): """ if syntax not in ("expr", "block"): raise SyntaxError("let_syntax is an expr and block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("let_syntax (block mode) does not take an asname") tree = expander.visit(tree) @@ -159,6 +161,8 @@ def abbrev(tree, *, args, syntax, expander, **kw): """ if syntax not in ("expr", "block"): raise SyntaxError("abbrev is an expr and block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("abbrev (block mode) does not take an asname") # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. diff --git a/unpythonic/syntax/nb.py b/unpythonic/syntax/nb.py index c2fb2a4d..7c688f6c 100644 --- a/unpythonic/syntax/nb.py +++ b/unpythonic/syntax/nb.py @@ -38,6 +38,8 @@ def nb(tree, *, args, syntax, **kw): """ if syntax != "block": raise SyntaxError("nb is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("nb does not take an asname") # Expand outside in. This macro is so simple and orthogonal the # ordering doesn't matter. This is cleaner. diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 9f52a797..6e46a7e1 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -95,6 +95,8 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 """ if syntax != "block": raise SyntaxError("prefix is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("prefix does not take an asname") # Expand outside in. Any nested macros should get clean standard Python, # not having to worry about tuples possibly denoting function calls. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index ded28538..f60f488b 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -112,6 +112,8 @@ def g(x): """ if syntax != "block": raise SyntaxError("autoreturn is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("autoreturn does not take an asname") # Expand outside in. Any nested macros should get clean standard Python, # not having to worry about implicit "return" statements. @@ -216,6 +218,8 @@ def result(ec): """ if syntax != "block": raise SyntaxError("tco is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("tco does not take an asname") # Two-pass macro. with dyn.let(_macro_expander=expander): @@ -617,6 +621,8 @@ def result(ec): """ if syntax != "block": raise SyntaxError("continuations is a block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("continuations does not take an asname") # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index cb542869..9bea2b7f 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -240,6 +240,8 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 """ if syntax not in ("expr", "block"): raise SyntaxError("test is an expr and block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("test (block mode) does not take an asname") # Two-pass macros. with dyn.let(_macro_expander=expander): @@ -290,6 +292,8 @@ def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 """ if syntax not in ("expr", "block"): raise SyntaxError("test_signals is an expr and block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("test_signals (block mode) does not take an asname") # Two-pass macros. with dyn.let(_macro_expander=expander): @@ -339,6 +343,8 @@ def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 """ if syntax not in ("expr", "block"): raise SyntaxError("test_raises is an expr and block macro only") + if syntax == "block" and kw['optional_vars'] is not None: + raise SyntaxError("test_raises (block mode) does not take an asname") with dyn.let(_macro_expander=expander): if syntax == "expr": From 9080c7af301b506fc489cb0f164e2af4527392c9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:05:57 +0300 Subject: [PATCH 119/832] document `with autoref[...] as ...` --- unpythonic/syntax/autoref.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 612608c9..25a8007f 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -97,6 +97,22 @@ def autoref(tree, *, args, syntax, expander, **kw): b c + The macro argument of `with autoref[...]` is an arbitrary expression that, + at run time, evaluates to the object instance to be autoreferenced. + + At the beginning of the block, the expression given as the macro argument + is implicitly assigned to a gensymmed variable, and then always used from + there, to ensure that the expression is evaluated only once. If you want to + explicitly name the variable instead of allowing `autoref` to gensym it, + use `with autoref[...] as ...`:: + + with autoref[e] as the_e: + a + b + c + + (Explicit naming can be useful for debugging.) + The transformation is applied in ``Load`` context only. ``Store`` and ``Del`` are not redirected. From 3bce9261103a009195b3d26547f661b7f729b294 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:06:15 +0300 Subject: [PATCH 120/832] use quasiquotes --- unpythonic/syntax/autoref.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 25a8007f..581e34aa 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -3,7 +3,7 @@ __all__ = ["autoref"] -from ast import (Name, Assign, Load, Call, Lambda, With, Constant, arg, +from ast import (Name, Load, Call, Lambda, With, Constant, arg, Attribute, Subscript, Store, Del) from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 @@ -301,7 +301,8 @@ def transform(self, tree): 'lazy', 'lazyrec', 'maybe_force_args', # lazify subsystem # the test framework subsystem 'callsite_filename', 'returns_normally'] + _test_function_names - newbody = [Assign(targets=[q[n[o]]], value=args[0])] + with q as newbody: + n[o] = a[args[0]] for stmt in block_body: newbody.append(AutorefTransformer(referents=always_skip + [o]).visit(stmt)) From 1e15b27af70c1a9de27dd22a7382f996a03b7f8a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:06:27 +0300 Subject: [PATCH 121/832] improve naming --- unpythonic/syntax/tailtools.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index f60f488b..df4ea885 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -1051,7 +1051,7 @@ def transform_callcc(owner, body): body = before + make_continuation(owner, callcc, contbody=after) return body # TODO: improve error reporting for stray call_cc[] invocations - class StrayChecker(ASTVisitor): + class StrayCallccChecker(ASTVisitor): def examine(self, tree): if iscallcc(tree): raise SyntaxError("call_cc[...] only allowed at the top level of a def or async def, or at the top level of the block; must appear as an expr or an assignment RHS") # pragma: no cover @@ -1098,7 +1098,7 @@ def transform(self, tree): block_body = CallccTransformer().visit(block_body) # inside defs # Validate. Each call_cc[] reached by the transformer was in a syntactically correct # position and has now been eliminated. Any remaining ones indicate syntax errors. - StrayChecker().visit(block_body) + StrayCallccChecker().visit(block_body) # set up the default continuation that just returns its args # (the top-level "cc" is only used for continuations created by call_cc[] at the top level of the block) From 444e1fa114c4f617be247dc8f9696237b068a464 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:06:33 +0300 Subject: [PATCH 122/832] remove unused import --- unpythonic/syntax/prefix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 6e46a7e1..ce4556b5 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -6,7 +6,7 @@ __all__ = ["prefix", "q", "u", "kw"] -from ast import Name, Call, Starred, Tuple, Load, Subscript +from ast import Call, Starred, Tuple, Load, Subscript import sys from mcpyrate.quotes import macros, q, u, a, t # noqa: F811, F401 From f69de8d7b3148d3970790bf597a28579db8a4321 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:07:15 +0300 Subject: [PATCH 123/832] use quasiquotes --- unpythonic/syntax/letdo.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 3b88a76b..fabb9d7e 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -860,8 +860,9 @@ def transform(self, tree): lines.append(expr) # CAUTION: letdoutil.py depends on the literal name "dof" to detect expanded do forms. # Also, the views depend on the exact AST structure. - thecall = q[h[dof]()] - thecall.args = lines + # AST-unquoting a `list` of ASTs in the arguments position of a quasiquoted call + # unpacks it into positional arguments. + thecall = q[h[dof](a[lines])] return thecall _our_local = capture_as_macro(local) From 202462241585e9df96b920b79a2e274d270a89c6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:07:56 +0300 Subject: [PATCH 124/832] check for stray local/delete manually in do/do0 This finally gives us nice compile-time error messages in a standardized format no matter where the stray `local[]` or `delete[]` appears. --- unpythonic/syntax/letdo.py | 46 +++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index fabb9d7e..cf69a320 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -32,14 +32,14 @@ from mcpyrate import gensym, parametricmacro from mcpyrate.quotes import capture_as_macro, is_captured_value -from mcpyrate.walkers import ASTTransformer +from mcpyrate.walkers import ASTTransformer, ASTVisitor from ..dynassign import dyn from ..lispylet import _let as letf, _dlet as dletf, _blet as bletf from ..misc import namelambda from ..seq import do as dof -from .letdoutil import (isenvassign, UnexpandedEnvAssignView, +from .letdoutil import (isdo, isenvassign, UnexpandedEnvAssignView, UnexpandedLetView, canonize_bindings) from .nameutil import getname, is_unexpanded_expr_macro from .scopeanalyzer import scoped_transform @@ -625,7 +625,7 @@ def local(tree, *, syntax, **kw): """ if syntax != "expr": raise SyntaxError("local is an expr macro only") # pragma: no cover - raise SyntaxError("local[] is only valid at the top level of a do[] or do0[]") # pragma: no cover, not meant to hit the expander + raise SyntaxError("local[] is only valid at the top level of a do[] or do0[]") # pragma: no cover def delete(tree, *, syntax, **kw): """[syntax] Delete a previously declared local name in a "do". @@ -646,7 +646,7 @@ def delete(tree, *, syntax, **kw): """ if syntax != "expr": raise SyntaxError("delete is an expr macro only") # pragma: no cover - raise SyntaxError("delete[] is only valid at the top level of a do[] or do0[]") # pragma: no cover, not meant to hit the expander + raise SyntaxError("delete[] is only valid at the top level of a do[] or do0[]") # pragma: no cover def do(tree, *, syntax, expander, **kw): """[syntax, expr] Stuff imperative code into an expression position. @@ -805,7 +805,7 @@ def _do(tree): islocaldef = partial(is_unexpanded_expr_macro, local, dyn._macro_expander) isdelete = partial(is_unexpanded_expr_macro, delete, dyn._macro_expander) - def find_localdefs(tree): + def transform_localdefs(tree): class LocaldefCollector(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -817,12 +817,12 @@ def transform(self, tree): view = UnexpandedEnvAssignView(expr) self.collect(view.name) view.value = self.visit(view.value) # nested local[] (e.g. from `do0[local[y << 5],]`) - return expr # e.g. `x << 21`; preserve the original expr to make the assignment occur. + return expr # `local[x << 21]` --> `x << 21`; compiling *that* makes the env-assignment occur. return tree # don't recurse! c = LocaldefCollector() tree = c.visit(tree) return tree, c.collected - def find_deletes(tree): + def transform_deletes(tree): class DeleteCollector(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -832,24 +832,50 @@ def transform(self, tree): if type(expr) is not Name: raise SyntaxError("delete[...] takes exactly one name") # pragma: no cover self.collect(expr.id) - return q[a[envdel](u[expr.id])] # -> e.pop(...) + return q[a[envdel](u[expr.id])] # `delete[x]` --> `e.pop('x')` return tree # don't recurse! c = DeleteCollector() tree = c.visit(tree) return tree, c.collected + def check_strays(ismatch, tree): + class StrayHelperMacroChecker(ASTVisitor): + def examine(self, tree): + if is_captured_value(tree): + return # don't recurse! + elif isdo(tree, expanded=False): + return # don't recurse! + elif ismatch(tree): + # Expand the stray helper macro invocation, to trigger its `SyntaxError` + # with a useful message, and *make the expander generate a use site traceback*. + # + # (If we just `raise` here directly, the expander won't see the use site + # of the `local[]` or `delete[]`, but just that of the `do[]`.) + dyn._macro_expander.visit(tree) + self.generic_visit(tree) + StrayHelperMacroChecker().visit(tree) + check_stray_localdefs = partial(check_strays, islocaldef) + check_stray_deletes = partial(check_strays, isdelete) + names = [] lines = [] for j, expr in enumerate(tree.elts, start=1): # Despite the recursion, this will not trigger false positives for nested do[] expressions, # because do[] is a second-pass macro, so they expand from inside out. - expr, newnames = find_localdefs(expr) - expr, deletednames = find_deletes(expr) + expr, newnames = transform_localdefs(expr) + expr, deletednames = transform_deletes(expr) if newnames and deletednames: raise SyntaxError("a do-item may have only local[] or delete[], not both") # pragma: no cover if newnames: if any(x in names for x in newnames): raise SyntaxError("local names must be unique in the same do") # pragma: no cover + + # Before transforming any further, check that there are no local[] or delete[] further in, where + # they don't belong. This allows the error message to show the *untransformed* source code for + # the erroneous invocation. + check_stray_localdefs(expr) + check_stray_deletes(expr) + # The envassignment transform (LHS) needs the updated bindings, whereas # the name transform (RHS) should use the previous bindings, so that any # changes to bindings take effect starting from the **next** do-item. From 45d3922407375ccbf95b22495c3e97ff96687966 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 01:23:16 +0300 Subject: [PATCH 125/832] add troubleshooting section --- README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/README.md b/README.md index a5f77847..623f6abb 100644 --- a/README.md +++ b/README.md @@ -689,6 +689,24 @@ Not working as advertised? Missing a feature? Documentation needs improvement? While `unpythonic` is intended as a serious tool for improving productivity as well as for teaching, right now my work priorities mean that it's developed and maintained on whatever time I can spare for it. Thus getting a response may take a while, depending on which project I happen to be working on. +## Troubleshooting + +### Cannot import the name `macros`? + +Could be a stale bytecode cache that Python thinks is still valid. This can happen especially if you first accidentally run `python3 some_macro_program.py`, and only then realize the invocation should have been `macropython some_macro_program.py`. + +The invocation with bare Python may compile to bytecode successfully and write the bytecode cache, but there is indeed no run-time object named `macros`, so the program will crash at that point. When the program is run again via `macropython`, the loader sees the bytecode cache, and because its `mtime` (as compared to the `.py` file) suggests it's up to date, the `.py` file is not automatically recompiled. + +Try clearing the bytecode caches in the affected directory with `macropython -c .`; this will force a recompile of the `.py` files the next time they are loaded. Then run normally, with `macropython some_macro_program.py`. + + +### I'm hacking a macro inside a module in `unpythonic.syntax` and my changes don't take? + +As of `mcpyrate` 3.4.0, macro re-exports, as done by `unpythonic.syntax.__init__`, may confuse the macro-dependency analyzer that determines bytecode cache validity. It only looks at the macro-import dependency graph, not the full dependency graph. I might change this in the future, but doing so will make it a lot slower than it needs to be in most circumstances. + +Try clearing the bytecode cache in `unpythonic/syntax/`; this will force a recompile. + + ## License All original code is released under the 2-clause [BSD license](LICENSE.md). From ac858afcdeff3b3b2120a20a8e9e6fc6ae703bff Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 02:02:25 +0300 Subject: [PATCH 126/832] remove outdated comment --- unpythonic/syntax/letsyntax.py | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 62e6f714..9ad2bca4 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -194,27 +194,7 @@ def __call__(self, tree, **kw): # make `block` look like a macro def _let_syntax_expr(bindings, body): # bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) body = _implicit_do(body) # support the extra bracket syntax - if not bindings: - # Optimize out a `let_syntax` with no bindings. The macro layer cannot trigger - # this case, because our syntaxes always require at least one binding. - # So this check is here just to protect against use with no bindings directly - # from other syntax transformers, which in theory could attempt anything. - # - # TODO: update this comment for mcpyrate - # The reason the macro layer never calls us with no bindings is technical. - # In the macro interface, with no bindings, the macro's `args` are `()` - # whether it was invoked as `let_syntax()[...]` or just `let_syntax[...]`. - # Thus, there is no way to distinguish, in the macro layer, between these - # two. We can't use `UnexpandedLetView` to do the dirty work of AST - # analysis, because the macro expander does too much automatically: in the macro - # layer, `tree` is only the part inside the brackets. So we really - # can't see whether the part outside the brackets was a Call with no - # arguments, or just a Name - both cases get treated exactly the same, - # as a macro invocation with empty `args`. - # - # The latter form, `let_syntax[...]`, is used by the haskelly syntax - # `let_syntax[(...) in ...]`, `let_syntax[..., where(...)]` - and in - # these cases, both the bindings and the body reside inside the brackets. + if not bindings: # Optimize out a `let_syntax` with no bindings. return body # pragma: no cover names_seen = set() From e99bec6b7fdb96f6914d77b57eb7b8f3a8470de3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 02:02:38 +0300 Subject: [PATCH 127/832] add comment --- unpythonic/syntax/letsyntax.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 9ad2bca4..435eb0b1 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -192,6 +192,9 @@ def __call__(self, tree, **kw): # make `block` look like a macro # -------------------------------------------------------------------------------- # Syntax transformers +# let_syntax[...][...] +# let_syntax[(...) in ...] +# let_syntax[..., where(...)] def _let_syntax_expr(bindings, body): # bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) body = _implicit_do(body) # support the extra bracket syntax if not bindings: # Optimize out a `let_syntax` with no bindings. From a0ae0454c305bdb43fb005b699e2422de34556c4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 02:02:54 +0300 Subject: [PATCH 128/832] make _implicit_do expand into a hygienic macro invocation --- unpythonic/syntax/letdo.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index cf69a320..98110d37 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -355,6 +355,8 @@ def _let_expr_impl(bindings, body, mode): # invocations in both bindings and body. # # But apply the implicit `do` (extra bracket syntax) first. + # (It is important we expand at least that immediately after, to resolve its local variables, + # because those may have the same lexical names as some of the let-bindings.) body = _implicit_do(body) body = dyn._macro_expander.visit(body) if not bindings: @@ -920,4 +922,4 @@ def _implicit_do(tree): The outer brackets enable multiple-expression mode, and the inner brackets are then interpreted as a list. """ - return _do(tree) if type(tree) is List else tree + return q[a[_our_do][t[tree.elts]]] if type(tree) is List else tree From ee34947141b0a405df6c6b444b1941c812197636 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 02:16:15 +0300 Subject: [PATCH 129/832] update docstrings --- unpythonic/syntax/letdoutil.py | 3 +++ unpythonic/syntax/util.py | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index b351efda..1a049d8f 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -254,6 +254,9 @@ class UnexpandedEnvAssignView: the env-assignment is expanded away (so, before the ``let[]`` or ``do[]`` containing it is expanded away). + This handles `mcpyrate.core.Done` `ASTMarker`s in the name position transparently, + to accommodate for expanded `mcpyrate.namemacro`s. + **Attributes**: ``name``: the name of the variable, as a str. diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 93b825ce..2e7eda76 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -124,8 +124,8 @@ def is_decorator(tree, fname): We detect: - - ``Name``, ``Attribute`` or a `mcpyrate` hygienic capture matching - the given ``fname`` (non-parametric decorator), and + - ``Name``, ``Attribute``, a `mcpyrate.core.Done`, or a `mcpyrate` + hygienic capture matching the given ``fname`` (non-parametric decorator), and - ``Call`` whose ``.func`` matches the above rule (parametric decorator). """ From 9837a427676ec5db6a5211ad91c1a75564f10d0e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 02:33:29 +0300 Subject: [PATCH 130/832] update test --- unpythonic/syntax/tests/test_autoref.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tests/test_autoref.py b/unpythonic/syntax/tests/test_autoref.py index 3e379840..e630e0c0 100644 --- a/unpythonic/syntax/tests/test_autoref.py +++ b/unpythonic/syntax/tests/test_autoref.py @@ -62,12 +62,19 @@ def runtests(): test[c == 17] # # Explicit asname optimizes lookups also in nested autoref blocks. - # # TODO: how to test? For now, just "with step_expansion" this and eyeball the result. + # # TODO: To test this, we need to use run-time compiler access and look at the AST. + # # TODO: See how `mcpyrate` does its tests. + # # TODO: For now, just "with step_expansion" this and eyeball the result. # with autoref[env(a=1, b=2)] as e1: # e1 # just e1, no autoref lookup - # with autoref[env(c=3, d=4)] as e2: + # with autoref[env(c=3, d=4, e1=None)] as e2: # e2 # just e2 # e1 # just e1 (special handling; already inserted lookup is removed by the outer block when it expands) + # But this special case we can test easily: + with autoref[env(a=1, b=2)] as e1: + # Place a key "e1" into our second env so that a spurious lookup for that triggers an error. + with autoref[env(c=3, d=4, e1=None)] as e2: + test[isinstance(e1, env)] # just e1, no lookup with testset("attributes and subscripts"): e2 = env(x=e, s=[1, 2, 3]) From 93e7aea12358b57697908140db96798c150beca5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 02:34:37 +0300 Subject: [PATCH 131/832] terminology: as-part vs. asname --- unpythonic/syntax/autocurry.py | 2 +- unpythonic/syntax/autoref.py | 4 ++-- unpythonic/syntax/dbg.py | 2 +- unpythonic/syntax/lambdatools.py | 8 ++++---- unpythonic/syntax/lazify.py | 2 +- unpythonic/syntax/letsyntax.py | 4 ++-- unpythonic/syntax/nb.py | 2 +- unpythonic/syntax/prefix.py | 2 +- unpythonic/syntax/tailtools.py | 6 +++--- unpythonic/syntax/testingtools.py | 8 ++++---- 10 files changed, 20 insertions(+), 20 deletions(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 7b2d2ea4..580bf655 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -66,7 +66,7 @@ def add3(a, b, c): if syntax != "block": raise SyntaxError("autocurry is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("autocurry does not take an asname") + raise SyntaxError("autocurry does not take an as-part") tree = expander.visit(tree) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 581e34aa..30f2beb8 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -47,7 +47,7 @@ # x # --> (lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))) # x.a # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x")))).a # x[s] # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))))[s] -# o # --> o (can only occur if an asname is supplied) +# o # --> o (can only occur if an as-part is supplied) # with ExpandedAutorefMarker("p"): # x # --> (lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))) # x.a # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))).a @@ -144,7 +144,7 @@ def autoref(tree, *, args, syntax, expander, **kw): target = kw.get("optional_vars", None) if target and type(target) is not Name: # tuples not accepted - raise SyntaxError("with autoref[...] as ... takes at most one asname") + raise SyntaxError("with autoref[...] as ... takes at most one name in the as-part") with dyn.let(_macro_expander=expander): return _autoref(block_body=tree, args=args, asname=target) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 06cb4ce3..c035bc5c 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -101,7 +101,7 @@ def dbg(tree, *, args, syntax, expander, **kw): if syntax not in ("expr", "block"): raise SyntaxError("dbg is an expr and block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("dbg (block mode) does not take an asname") + raise SyntaxError("dbg (block mode) does not take an as-part") tree = expander.visit(tree) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index de369cb2..e0e6b206 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -66,7 +66,7 @@ def multilambda(tree, *, syntax, expander, **kw): if syntax != "block": raise SyntaxError("multilambda is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("multilambda does not take an asname") + raise SyntaxError("multilambda does not take an as-part") # Expand outside in. # multilambda should expand first before any let[], do[] et al. that happen @@ -119,7 +119,7 @@ def namedlambda(tree, *, syntax, expander, **kw): if syntax != "block": raise SyntaxError("namedlambda is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("namedlambda does not take an asname") + raise SyntaxError("namedlambda does not take an as-part") # Two-pass macro. We pass in the expander to allow the macro to decide when to recurse. with dyn.let(_macro_expander=expander): @@ -192,7 +192,7 @@ def quicklambda(tree, *, syntax, expander, **kw): if syntax != "block": raise SyntaxError("quicklambda is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("quicklambda does not take an asname") + raise SyntaxError("quicklambda does not take an as-part") # This macro expands outside in. # @@ -227,7 +227,7 @@ def foo(n): if syntax != "block": raise SyntaxError("envify is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("envify does not take an asname") + raise SyntaxError("envify does not take an as-part") # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index e3ac482d..a0ec1c32 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -412,7 +412,7 @@ def doit(): if syntax != "block": raise SyntaxError("lazify is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("lazify does not take an asname") + raise SyntaxError("lazify does not take an as-part") # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 435eb0b1..7c85d90c 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -129,7 +129,7 @@ def let_syntax(tree, *, args, syntax, expander, **kw): if syntax not in ("expr", "block"): raise SyntaxError("let_syntax is an expr and block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("let_syntax (block mode) does not take an asname") + raise SyntaxError("let_syntax (block mode) does not take an as-part") tree = expander.visit(tree) @@ -162,7 +162,7 @@ def abbrev(tree, *, args, syntax, expander, **kw): if syntax not in ("expr", "block"): raise SyntaxError("abbrev is an expr and block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("abbrev (block mode) does not take an asname") + raise SyntaxError("abbrev (block mode) does not take an as-part") # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. diff --git a/unpythonic/syntax/nb.py b/unpythonic/syntax/nb.py index 7c688f6c..a18f70e3 100644 --- a/unpythonic/syntax/nb.py +++ b/unpythonic/syntax/nb.py @@ -39,7 +39,7 @@ def nb(tree, *, args, syntax, **kw): if syntax != "block": raise SyntaxError("nb is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("nb does not take an asname") + raise SyntaxError("nb does not take an as-part") # Expand outside in. This macro is so simple and orthogonal the # ordering doesn't matter. This is cleaner. diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index ce4556b5..945e476e 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -96,7 +96,7 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 if syntax != "block": raise SyntaxError("prefix is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("prefix does not take an asname") + raise SyntaxError("prefix does not take an as-part") # Expand outside in. Any nested macros should get clean standard Python, # not having to worry about tuples possibly denoting function calls. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index df4ea885..8f132141 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -113,7 +113,7 @@ def g(x): if syntax != "block": raise SyntaxError("autoreturn is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("autoreturn does not take an asname") + raise SyntaxError("autoreturn does not take an as-part") # Expand outside in. Any nested macros should get clean standard Python, # not having to worry about implicit "return" statements. @@ -219,7 +219,7 @@ def result(ec): if syntax != "block": raise SyntaxError("tco is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("tco does not take an asname") + raise SyntaxError("tco does not take an as-part") # Two-pass macro. with dyn.let(_macro_expander=expander): @@ -622,7 +622,7 @@ def result(ec): if syntax != "block": raise SyntaxError("continuations is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("continuations does not take an asname") + raise SyntaxError("continuations does not take an as-part") # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 9bea2b7f..24b9407c 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -241,7 +241,7 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax not in ("expr", "block"): raise SyntaxError("test is an expr and block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("test (block mode) does not take an asname") + raise SyntaxError("test (block mode) does not take an as-part") # Two-pass macros. with dyn.let(_macro_expander=expander): @@ -293,7 +293,7 @@ def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax not in ("expr", "block"): raise SyntaxError("test_signals is an expr and block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("test_signals (block mode) does not take an asname") + raise SyntaxError("test_signals (block mode) does not take an as-part") # Two-pass macros. with dyn.let(_macro_expander=expander): @@ -344,7 +344,7 @@ def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax not in ("expr", "block"): raise SyntaxError("test_raises is an expr and block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("test_raises (block mode) does not take an asname") + raise SyntaxError("test_raises (block mode) does not take an as-part") with dyn.let(_macro_expander=expander): if syntax == "expr": @@ -464,7 +464,7 @@ def expand_testing_macros_first(tree, *, syntax, expander, **kw): if syntax != "block": raise SyntaxError("expand_testing_macros_first is a block macro only") if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("expand_testing_macros_first does not take an asname") + raise SyntaxError("expand_testing_macros_first does not take an as-part") testing_macros = [test, test_signals, test_raises, error, fail, warn] macro_bindings = extract_bindings(expander.bindings, *testing_macros) From ee9e18b6237ec38765bebd1885ac65718f861476 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 03:27:10 +0300 Subject: [PATCH 132/832] let_syntax block, expr: syntactic consistency: brackets --- unpythonic/syntax/__init__.py | 2 - unpythonic/syntax/letsyntax.py | 91 ++++++++++++++--------- unpythonic/syntax/tests/test_letsyntax.py | 8 +- 3 files changed, 60 insertions(+), 41 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 03763ba1..2bf6919b 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -81,8 +81,6 @@ # TODO: upgrade let_syntax block, expr into `mcpyrate` magic variables # TODO: also kw() in unpythonic.syntax.prefix -# TODO: let_syntax block, expr: syntactic consistency: change parentheses to brackets - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 7c85d90c..f6176862 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -8,12 +8,15 @@ from mcpyrate.quotes import macros, q, a # noqa: F401 -from ast import (Name, Call, Starred, Expr, With, - FunctionDef, AsyncFunctionDef, ClassDef, Attribute) +from ast import Name, Call, Subscript, Tuple, Starred, Expr, With from copy import deepcopy +from functools import partial +import sys from mcpyrate import parametricmacro +from mcpyrate.markers import ASTMarker from mcpyrate.quotes import is_captured_value +from mcpyrate.utils import rename from mcpyrate.walkers import ASTTransformer from .letdo import _implicit_do, _destructure_and_apply_let @@ -22,7 +25,6 @@ # -------------------------------------------------------------------------------- # Macro interface -# TODO: change the block() construct to block[], for syntactic consistency @parametricmacro def let_syntax(tree, *, args, syntax, expander, **kw): """[syntax, expr/block] Introduce local **syntactic** bindings. @@ -45,11 +47,11 @@ def let_syntax(tree, *, args, syntax, expander, **kw): with let_syntax: with block as xs: # capture a block of statements - bare name ... - with block(a, ...) as xs: # capture a block of statements - template + with block[a, ...] as xs: # capture a block of statements - template ... with expr as x: # capture a single expression - bare name ... - with expr(a, ...) as x: # capture a single expression - template + with expr[a, ...] as x: # capture a single expression - template ... body0 ... @@ -86,13 +88,12 @@ def let_syntax(tree, *, args, syntax, expander, **kw): Templates support only positional arguments, with no default values. Even in block templates, parameters are always expressions (because they - use the function-call syntax at the use site). + use the subscript syntax at the use site). - In the body of the ``let_syntax``, a template is used like a function call. - Just like in an actual function call, when the template is substituted, + In the body of the ``let_syntax``, a template is used like an expr macro. + Just like in an actual macro invocation, when the template is substituted, any instances of its formal parameters on its RHS get replaced by the - argument values from the "call" site; but ``let_syntax`` performs this - at macro-expansion time. + argument values from the invocation site. Note each instance of the same formal parameter gets a fresh copy of the corresponding argument value. @@ -223,11 +224,11 @@ def register_bindings(): # with let_syntax: # with block as xs: # ... -# with block(a, ...) as xs: +# with block[a, ...] as xs: # ... # with expr as x: # ... -# with expr(a, ...) as x: +# with expr[a, ...] as x: # ... # body0 # ... @@ -278,6 +279,11 @@ def isbinding(tree): ctxmanager = tree.items[0].context_expr if type(ctxmanager) is Name and ctxmanager.id == mode: return mode, "barename" + # expr[...], block[...] + if type(ctxmanager) is Subscript and type(ctxmanager.value) is Name and ctxmanager.value.id == mode: + return mode, "template" + # expr(...), block(...) + # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 if type(ctxmanager) is Call and type(ctxmanager.func) is Name and ctxmanager.func.id == mode: return mode, "template" return False @@ -298,10 +304,30 @@ def isbinding(tree): # ----------------------------------------------------------------------------- +def _get_subscript_args(tree): + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + theslice = tree.slice + else: + theslice = tree.slice.value + if type(theslice) not in (Tuple, Name): + raise SyntaxError("expected [a0, ...]") + if type(theslice) is Name: + args = [theslice.id] + else: # Tuple + args = [a.id for a in theslice.elts] + return args + +# x --> "x", [] +# f[a, b, c] --> "f", ["a", "b", "c"] +# f(a, b, c) --> "f", ["a", "b", "c"] def _analyze_lhs(tree): if type(tree) is Name: # bare name name = tree.id args = [] + elif type(tree) is Subscript and type(tree.value) is Name: # template f[x, ...] + name = tree.value.id + args = _get_subscript_args(tree) + # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 elif type(tree) is Call and type(tree.func) is Name: # template f(x, ...) name = tree.func.id if any(type(a) is Starred for a in tree.args): # *args (Python 3.5+) @@ -338,47 +364,42 @@ def subst(): return self.generic_visit(tree) return Splicer().visit(tree) - # if the new value is also bare name, perform the substitution (now as a string) - # also in the name part of def and similar, to support human intuition of "renaming" - # TODO: use `mcpyrate.utils.rename`, it was designed for things like this? + # If the new value is also bare name, perform the substitution (now as a string) + # also in the name part of def and similar, to support human intuition of "renaming". if type(value) is Name: - newname = value.id - def splice_barestring(tree): - class BarestringSplicer(ASTTransformer): - def transform(self, tree): - if is_captured_value(tree): - return tree # don't recurse! - if type(tree) in (FunctionDef, AsyncFunctionDef, ClassDef): - if tree.name == name: - tree.name = newname - elif type(tree) is Attribute: - if tree.attr == name: - tree.attr = newname - return self.generic_visit(tree) - return BarestringSplicer().visit(tree) - postproc = splice_barestring + postproc = partial(rename, name, value.id) else: postproc = lambda x: x return postproc(splice(tree)) def _substitute_barenames(barenames, tree): - for name, _, value, mode in barenames: + for name, _noformalparams, value, mode in barenames: tree = _substitute_barename(name, value, tree, mode) return tree def _substitute_templates(templates, tree): for name, formalparams, value, mode in templates: def isthisfunc(tree): - return type(tree) is Call and type(tree.func) is Name and tree.func.id == name + if type(tree) is Subscript and type(tree.value) is Name and tree.value.id == name: + return True + # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + if type(tree) is Call and type(tree.func) is Name and tree.func.id == name: + return True + return False def subst(tree): - theargs = tree.args + if type(tree) is Subscript: + theargs = _get_subscript_args(tree) + elif type(tree) is Call: + theargs = tree.args + else: + assert False if len(theargs) != len(formalparams): raise SyntaxError(f"let_syntax template '{name}' expected {len(formalparams)} arguments, got {len(theargs)}") # pragma: no cover # make a fresh deep copy of the RHS to avoid destroying the template. - tree = deepcopy(value) # expand the f itself in f(x, ...) + tree = deepcopy(value) # expand the f itself in f[x, ...] or f(x, ...) for k, v in zip(formalparams, theargs): # expand the x, ... in the expanded form of f - # can't put statements in a Call, so always treat args as expressions. + # can't put statements in a Subscript or in a Call, so always treat args as expressions. tree = _substitute_barename(k, v, tree, "expr") return tree def splice(tree): diff --git a/unpythonic/syntax/tests/test_letsyntax.py b/unpythonic/syntax/tests/test_letsyntax.py index 60ef58cd..8531d498 100644 --- a/unpythonic/syntax/tests/test_letsyntax.py +++ b/unpythonic/syntax/tests/test_letsyntax.py @@ -86,11 +86,11 @@ class Silly: test[snd == 5] with let_syntax: - with block(a, b, c) as makeabc: # block template - parameters are expressions # noqa: F821, `let_syntax` defines `a`, `b`, `c` when we call `makeabc`. + with block[a, b, c] as makeabc: # block template - parameters are expressions # noqa: F821, `let_syntax` defines `a`, `b`, `c` when we call `makeabc`. lst = [a, b, c] # noqa: F821 makeabc(3 + 4, 2**3, 3 * 3) test[lst == [7, 8, 9]] - with expr(n) as nth: # single-expression template # noqa: F821, `let_syntax` defines `n` when we call `nth`. + with expr[n] as nth: # single-expression template # noqa: F821, `let_syntax` defines `n` when we call `nth`. lst[n] # noqa: F821 test[nth(2) == 9] @@ -160,10 +160,10 @@ def alias(): with let_syntax: # in this example, both substitutions are templates, so they must be # defined in the same order they are meant to be applied. - with block(a) as twice: # noqa: F821 + with block[a] as twice: # noqa: F821 a # noqa: F821 a # noqa: F821 - with block(x, y, z) as appendxyz: # noqa: F821 + with block[x, y, z] as appendxyz: # noqa: F821 lst += [x, y, z] # noqa: F821 lst = [] # template substitution invoked in a parameter From 99440a25ad320c153269a6e8b66796243da5ef4e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 03:33:25 +0300 Subject: [PATCH 133/832] update doc --- doc/macros.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 77ec2781..748559ec 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -377,19 +377,19 @@ y = let_syntax[[print(f(2)), # works as a block macro with let_syntax: - with block(a, b, c) as makeabc: # capture a block of statements + with block[a, b, c] as makeabc: # capture a block of statements lst = [a, b, c] makeabc(3 + 4, 2**3, 3 * 3) assert lst == [7, 8, 9] - with expr(n) as nth: # capture a single expression + with expr[n] as nth: # capture a single expression lst[n] assert nth(2) == 9 with let_syntax: - with block(a) as twice: + with block[a] as twice: a a - with block(x, y, z) as appendxyz: + with block[x, y, z] as appendxyz: lst += [x, y, z] lst = [] twice(appendxyz(7, 8, 9)) @@ -443,14 +443,14 @@ When used as an expr macro, all bindings are registered first, and then the body The ``abbrev`` macro is otherwise exactly like ``let_syntax``, but it expands in the first pass (outside in). Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the ``abbrev`` itself expands before any macros invoked in its body. This allows things like: ```python -abbrev[(a, ast_literal)][ - a[tree1] if a[tree2] else a[tree3]] +abbrev[(m, macrowithverylongname)][ + m[tree1] if m[tree2] else m[tree3]] # v0.12.0+ -abbrev[((a, ast_literal)) in - a[tree1] if a[tree2] else a[tree3]] -abbrev[a[tree1] if a[tree2] else a[tree3], - where((a, ast_literal))] +abbrev[((m, macrowithverylongname)) in + m[tree1] if m[tree2] else m[tree3]] +abbrev[m[tree1] if m[tree2] else m[tree3], + where((m, macrowithverylongname))] ``` which can be useful when writing macros. From fe2f7cc4162178cf2f877979239c49acf7cca891 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 03:33:40 +0300 Subject: [PATCH 134/832] update test --- unpythonic/syntax/tests/test_letsyntax.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_letsyntax.py b/unpythonic/syntax/tests/test_letsyntax.py index 8531d498..240297dd 100644 --- a/unpythonic/syntax/tests/test_letsyntax.py +++ b/unpythonic/syntax/tests/test_letsyntax.py @@ -46,6 +46,7 @@ def verylongfunctionname(x=1): # templates # - positional parameters only, no default values + # TODO: updating this to use bracket syntax requires changes to `_destructure_and_apply_let`. y = let_syntax((f(a), verylongfunctionname(2 * a)))[[ # noqa: F821 f(2), # noqa: F821 f(3)]] # noqa: F821 @@ -147,7 +148,7 @@ def alias(): with block as append456: lst += [4, 5, 6] # template - applied before any barenames - with block(a) as twice: # noqa: F821 + with block[a] as twice: # noqa: F821 a # noqa: F821 a # noqa: F821 lst = [] From 4d8eaa6328701ff11a75704617fbf282f45c2379 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 03:46:42 +0300 Subject: [PATCH 135/832] allow brackets in let_syntax template definitions in expr mode --- doc/macros.md | 28 +++++++++++------------ unpythonic/syntax/letdoutil.py | 5 ++-- unpythonic/syntax/letsyntax.py | 12 ++++------ unpythonic/syntax/tests/test_letsyntax.py | 6 ++--- 4 files changed, 25 insertions(+), 26 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 748559ec..f5b6e59f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -356,9 +356,9 @@ y = let_syntax[(f, verylongfunctionname)][[ # extra brackets: implicit do in bo f(5)]] assert y == 5 -y = let_syntax[(f(a), verylongfunctionname(2*a))][[ # template with formal parameter "a" - print(f(2)), - f(3)]] +y = let_syntax[(f[a], verylongfunctionname(2*a))][[ # template with formal parameter "a" + print(f[2]), + f[3]]] assert y == 6 # v0.12.0+ @@ -368,12 +368,12 @@ y = let_syntax[((f, verylongfunctionname)) in y = let_syntax[[print(f()), f(5)], where((f, verylongfunctionname))] -y = let_syntax[((f(a), verylongfunctionname(2*a))) in - [print(f(2)), - f(3)]] -y = let_syntax[[print(f(2)), - f(3)], - where((f(a), verylongfunctionname(2*a)))] +y = let_syntax[((f[a], verylongfunctionname(2*a))) in + [print(f[2]), + f[3]]] +y = let_syntax[[print(f[2]), + f[3]], + where((f[a], verylongfunctionname(2*a)))] # works as a block macro with let_syntax: @@ -401,13 +401,13 @@ After macro expansion completes, ``let_syntax`` has zero runtime overhead; it co
There are two kinds of substitutions: ->*Bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are currently **not** supported.) +>*Bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are **not** supported.) > ->When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, ``f(a)`` is a template with a formal parameter ``a``. But when used as a block macro, the formal parameters are declared on the ``block`` or ``expr`` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use ``with block as ...:`` or ``with expr as ...:`` with no arguments. +>When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, ``f[a]`` is a template with a formal parameter ``a``. But when used as a block macro, the formal parameters are declared on the ``block`` or ``expr`` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use ``with block as ...:`` or ``with expr as ...:`` with no macro arguments. > ->In the body of ``let_syntax``, a bare name substitution is invoked by name (just like a variable). A template substitution is invoked like a function call. Just like in an actual function call, when the template is substituted, any instances of its formal parameters in the definition get replaced by the argument values from the "call" site; but ``let_syntax`` performs this at macro-expansion time, and the "value" is a snippet of code. +>In the body of ``let_syntax``, a bare name substitution is invoked by name (just like a variable). A template substitution is invoked like an expr macro. Any instances of the formal parameters of the template get replaced by the argument values from the use site, at macro expansion time. > ->Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each ``a`` in the body of ``twice`` separately expands to a copy of whatever code was given as the positional argument ``a``. +>Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each ``a`` in the body of ``twice`` separately expands to a copy of whatever code was given as the macro argument ``a``. > >When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. (The single expression can be an explicit ``do[]`` if multiple expressions are needed.) When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. > @@ -430,7 +430,7 @@ After macro expansion completes, ``let_syntax`` has zero runtime overhead; it co > - If the bindings are ``((x, y), (y, z))``, then an ``x`` at the use site transforms to ``z``. So does a ``y`` at the use site. > - But if the bindings are ``((y, z), (x, y))``, then an ``x`` at the use site transforms to ``y``, and only an explicit ``y`` at the use site transforms to ``z``. > ->Even in block templates, arguments are always expressions, because invoking a template uses the function-call syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above. +>Even in block templates, arguments are always expressions, because invoking a template uses the subscript syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above.

diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 1a049d8f..14ab2eed 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -24,7 +24,6 @@ def where(*bindings): dof_name = "dof" # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. currycall_name = "currycall" # output of ``unpythonic.syntax.curry`` -# TODO: switch from call to subscript in name position for let_syntax templates. def canonize_bindings(elts, allow_call_in_name_position=False): # public as of v0.14.3+ """Wrap a single binding without container into a length-1 `list`. @@ -42,13 +41,15 @@ def canonize_bindings(elts, allow_call_in_name_position=False): # public as of allow_call_in_name_position: used by let_syntax to allow template definitions; in the call, the "function" is the template name, and the positional "parameters" are the template parameters (which may then appear in the template body). + (Despite the name, this recognizes `Subscript` too, to support brackets.) """ def isname(x): # The `Done` may be produced by expanded `@namemacro`s. return type(x) is Name or (isinstance(x, Done) and isname(x.body)) def iskey(x): return (isname(x) or - (allow_call_in_name_position and type(x) is Call and isname(x.func))) + (allow_call_in_name_position and ((type(x) is Call and isname(x.func)) or + (type(x) is Subscript and isname(x.value))))) if len(elts) == 2 and iskey(elts[0]): return [Tuple(elts=elts)] # TODO: `mcpyrate`: just `q[t[elts]]`? if all((type(b) is Tuple and len(b.elts) == 2 and iskey(b.elts[0])) for b in elts): diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index f6176862..1b88ae34 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -309,12 +309,10 @@ def _get_subscript_args(tree): theslice = tree.slice else: theslice = tree.slice.value - if type(theslice) not in (Tuple, Name): - raise SyntaxError("expected [a0, ...]") - if type(theslice) is Name: - args = [theslice.id] - else: # Tuple - args = [a.id for a in theslice.elts] + if type(theslice) is Tuple: + args = theslice.elts + else: + args = [theslice] return args # x --> "x", [] @@ -326,7 +324,7 @@ def _analyze_lhs(tree): args = [] elif type(tree) is Subscript and type(tree.value) is Name: # template f[x, ...] name = tree.value.id - args = _get_subscript_args(tree) + args = [a.id for a in _get_subscript_args(tree)] # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 elif type(tree) is Call and type(tree.func) is Name: # template f(x, ...) name = tree.func.id diff --git a/unpythonic/syntax/tests/test_letsyntax.py b/unpythonic/syntax/tests/test_letsyntax.py index 240297dd..8e067955 100644 --- a/unpythonic/syntax/tests/test_letsyntax.py +++ b/unpythonic/syntax/tests/test_letsyntax.py @@ -47,9 +47,9 @@ def verylongfunctionname(x=1): # templates # - positional parameters only, no default values # TODO: updating this to use bracket syntax requires changes to `_destructure_and_apply_let`. - y = let_syntax((f(a), verylongfunctionname(2 * a)))[[ # noqa: F821 - f(2), # noqa: F821 - f(3)]] # noqa: F821 + y = let_syntax((f[a], verylongfunctionname(2 * a)))[[ # noqa: F821 + f[2], # noqa: F821 + f[3]]] # noqa: F821 test[evaluations == 8] test[y == 6] From 57f91f4b89629674564a648de9e322d48ed7ee1e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 04:04:21 +0300 Subject: [PATCH 136/832] make `let_syntax` `expr` and `block` error out in incorrect position --- unpythonic/syntax/__init__.py | 3 +- unpythonic/syntax/letsyntax.py | 50 +++++++++++++---------- unpythonic/syntax/tests/test_letsyntax.py | 4 +- 3 files changed, 31 insertions(+), 26 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 2bf6919b..1e0a9704 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -78,8 +78,7 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: upgrade let_syntax block, expr into `mcpyrate` magic variables -# TODO: also kw() in unpythonic.syntax.prefix +# TODO: upgrade kw() in unpythonic.syntax.prefix into `mcpyrate` magic variable? # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 1b88ae34..a9692781 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -14,9 +14,9 @@ import sys from mcpyrate import parametricmacro -from mcpyrate.markers import ASTMarker +from mcpyrate.expander import MacroExpander from mcpyrate.quotes import is_captured_value -from mcpyrate.utils import rename +from mcpyrate.utils import rename, extract_bindings from mcpyrate.walkers import ASTTransformer from .letdo import _implicit_do, _destructure_and_apply_let @@ -132,12 +132,22 @@ def let_syntax(tree, *, args, syntax, expander, **kw): if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("let_syntax (block mode) does not take an as-part") - tree = expander.visit(tree) + # Expand other inner macros now, but not `with expr` or `with block`; + # we'll transform those away manually. + expr_and_block_bindings = extract_bindings(expander.bindings, expr, block) + other_bindings = {k: v for k, v in expander.bindings.items() if k not in expr_and_block_bindings} + tree = MacroExpander(other_bindings, filename=expander.filename).visit(tree) if syntax == "expr": - return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, allow_call_in_name_position=True) + tree = _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, + allow_call_in_name_position=True) else: # syntax == "block": - return _let_syntax_block(block_body=tree) + tree = _let_syntax_block(block_body=tree) + + # Now, we can make incorrectly placed `with expr` and `with block` error out. + # (With nested `with let_syntax` invocations only the outermost one will do this, + # but that should be fine.) + return MacroExpander(expr_and_block_bindings, filename=expander.filename).visit(tree) @parametricmacro def abbrev(tree, *, args, syntax, expander, **kw): @@ -172,23 +182,19 @@ def abbrev(tree, *, args, syntax, expander, **kw): else: return _let_syntax_block(block_body=tree) -# TODO: convert to mcpyrate magic variable -class expr: - """[syntax] Magic identifier for ``with expr:`` inside a ``with let_syntax:``.""" - def __repr__(self): # in case one of these ends up somewhere at runtime - return "" # pragma: no cover - def __call__(self, tree, **kw): # make `expr` look like a macro - pass -expr = expr() - -# TODO: convert to mcpyrate magic variable -class block: - """[syntax] Magic identifier for ``with block:`` inside a ``with let_syntax:``.""" - def __repr__(self): # in case one of these ends up somewhere at runtime - return "" # pragma: no cover - def __call__(self, tree, **kw): # make `block` look like a macro - pass -block = block() +@parametricmacro +def expr(tree, *, syntax, **kw): + """[syntax, block] ``with expr:`` inside a ``with let_syntax:``.""" + if syntax != "block": + raise SyntaxError("`expr` is a block macro only") + raise SyntaxError("`expr` is only valid at the top level of a block-mode `let_syntax` or `abbrev`") # pragma: no cover, not intended to hit the expander + +@parametricmacro +def block(tree, *, syntax, **kw): + """[syntax, block] ``with block:`` inside a ``with let_syntax:``.""" + if syntax != "block": + raise SyntaxError("`block` is a block macro only") + raise SyntaxError("`block` is only valid at the top level of a block-mode `let_syntax` or `abbrev`") # pragma: no cover, not intended to hit the expander # -------------------------------------------------------------------------------- # Syntax transformers diff --git a/unpythonic/syntax/tests/test_letsyntax.py b/unpythonic/syntax/tests/test_letsyntax.py index 8e067955..26b1c8ad 100644 --- a/unpythonic/syntax/tests/test_letsyntax.py +++ b/unpythonic/syntax/tests/test_letsyntax.py @@ -15,8 +15,8 @@ from ...syntax import macros, test, test_raises # noqa: F401 from ...test.fixtures import session, testset -from ...syntax import macros, let_syntax, abbrev # noqa: F401, F811 -from ...syntax import block, expr, where +from ...syntax import macros, let_syntax, abbrev, block, expr # noqa: F401, F811 +from ...syntax import where def runtests(): with testset("expression variant"): From f1428dec22097c87726e08e510024f684b9795f5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:07:37 +0300 Subject: [PATCH 137/832] update changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20c359f6..68cc77b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) together with a kitchen-sink language extension macro package such as `unpythonic`. - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) -- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), and `it` (for `aif[]`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. +- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), and `with expr`/`with block` (for `let_syntax`/`abbrev`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. - `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. - Python 3.8 and 3.9 support added. From 0cb60eedadcda5fb4eacc5aeb008f48969f0dc9f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:08:10 +0300 Subject: [PATCH 138/832] pass filename to ast.parse in setup.py --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 478cbf8c..573ce4ff 100644 --- a/setup.py +++ b/setup.py @@ -40,7 +40,7 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa with open(init_py_path) as f: for line in f: if line.startswith("__version__"): - module = ast.parse(line) + module = ast.parse(line, filename=init_py_path) expr = module.body[0] assert isinstance(expr, ast.Assign) v = expr.value From 6933219a7d013a38c23e7ca5f33d639d7e769520 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:08:26 +0300 Subject: [PATCH 139/832] mark some TODOs --- unpythonic/syntax/letdoutil.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 14ab2eed..9aa5be50 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -67,6 +67,8 @@ def isenvassign(tree): # The `Done` may be produced by expanded `@namemacro`s. return type(tree.left) is Name or (isinstance(tree.left, Done) and type(tree.body) is Name) +# TODO: This would benefit from macro destructuring in the expander. +# TODO: See https://github.com/Technologicat/mcpyrate/issues/3 def islet(tree, expanded=True): """Test whether tree is a ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]``, or ``abbrev[]``. @@ -202,6 +204,8 @@ def maybeiscontentofletwhere(tree): return "where_expr" return False # invalid syntax for haskelly let +# TODO: This would benefit from macro destructuring in the expander. +# TODO: See https://github.com/Technologicat/mcpyrate/issues/3 def isdo(tree, expanded=True): """Detect whether tree is a ``do[]`` or ``do0[]``. From 7f578076d22cbecc8b5d4cf1e7de32e25333e14c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:08:44 +0300 Subject: [PATCH 140/832] add is_unexpanded_block_macro --- unpythonic/syntax/nameutil.py | 49 +++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index 70bffe94..45636e96 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -5,9 +5,10 @@ with a unified API. """ -__all__ = ["isx", "getname", "is_unexpanded_expr_macro"] +__all__ = ["isx", "getname", + "is_unexpanded_expr_macro", "is_unexpanded_block_macro"] -from ast import Name, Attribute, Subscript +from ast import Name, Attribute, Subscript, Call import sys from mcpyrate.core import Done @@ -107,13 +108,14 @@ def is_unexpanded_expr_macro(macrofunction, expander, tree): """ if not type(tree) is Subscript: return False + maybemacro = tree.value # hygienic captures and as-imports - key = is_captured_macro(tree.value) + key = is_captured_macro(maybemacro) if key: name_node = lookup_macro(key) - elif type(tree.value) is Name: - name_node = tree.value + elif type(maybemacro) is Name: + name_node = maybemacro else: return False @@ -124,3 +126,40 @@ def is_unexpanded_expr_macro(macrofunction, expander, tree): else: return tree.slice.value return False + + +# TODO: This utility really wants to live in `mcpyrate`, as part of a macro destructuring subsystem. +# TODO: It needs to be made more general, to detect if there are several macros in the same `with`. +def is_unexpanded_block_macro(macrofunction, expander, tree): + """Check whether `tree` is an expr macro invocation bound to `macrofunction` in `expander`. + + This accounts for hygienic macro captures and as-imports. + + If there is a match, return the subscript slice, i.e. the tree that would be passed + to the macro function by the expander if the macro was expanded normally. + + **CAUTION**: This function doesn't currently support detecting macros that + take macro arguments. + """ + if not type(tree) is Subscript: + return False + maybemacro = tree.value + + # discard args if any + if type(maybemacro) is Subscript: + maybemacro = maybemacro.value + # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + elif type(maybemacro) is Call: + maybemacro = maybemacro.func + + # hygienic captures and as-imports + key = is_captured_macro(maybemacro) + if key: + name_node = lookup_macro(key) + elif type(maybemacro) is Name: + name_node = maybemacro + else: + return False + + macro = expander.isbound(name_node.id) + return macro is macrofunction From 8b8881800c29d152c18ebdd3fe8ae4949a025356 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:08:56 +0300 Subject: [PATCH 141/832] improve docstring and comment --- unpythonic/syntax/prefix.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 945e476e..1d6e38b4 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -76,11 +76,12 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 specified by multiple ``kw`` operators, the rightmost definition wins. **Note**: Python itself prohibits having repeated named args in the **same** - ``kw`` operator, because it uses the function call syntax. If you get a - `SyntaxError: keyword argument repeated` with no useful traceback, - check any recent ``kw`` operators you have added in prefix blocks. + ``kw`` operator, because it uses the function call syntax. If you try to pass + the same named arg multiple times, as of 0.15, you should get a + `SyntaxError: keyword argument repeated` with a traceback. - A ``kw(...)`` operator in a quoted tuple (not a function call) is an error. + A ``kw(...)`` operator in a quoted tuple (i.e. a tuple that does not not + represent a function call) is an error. Current limitations: @@ -222,6 +223,6 @@ def transform(self, tree): self.withstate(thecall, quotelevel=quotelevel) return self.visit(thecall) - # This is a first-pass macro. Any nested macros should get clean standard Python, + # This is a outside-in macro. Any nested macros should get clean standard Python, # not having to worry about tuples possibly denoting function calls. return PrefixTransformer(quotelevel=0).visit(block_body) From ef307bac8d96c1ffdaa5d17532686eefc12548d3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:38:39 +0300 Subject: [PATCH 142/832] mark some TODOs --- unpythonic/syntax/nameutil.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index 45636e96..826b4b47 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -86,7 +86,7 @@ def getname(tree, accept_attr=True): if type(tree) is Name: return tree.id key = is_captured_value(tree) # AST -> (name, frozen_value) or False - if key: + if key: # TODO: Python 3.8+: use walrus assignment here name, frozen_value = key return name if accept_attr and type(tree) is Attribute: @@ -112,7 +112,7 @@ def is_unexpanded_expr_macro(macrofunction, expander, tree): # hygienic captures and as-imports key = is_captured_macro(maybemacro) - if key: + if key: # TODO: Python 3.8+: use walrus assignment here name_node = lookup_macro(key) elif type(maybemacro) is Name: name_node = maybemacro @@ -154,7 +154,7 @@ def is_unexpanded_block_macro(macrofunction, expander, tree): # hygienic captures and as-imports key = is_captured_macro(maybemacro) - if key: + if key: # TODO: Python 3.8+: use walrus assignment here name_node = lookup_macro(key) elif type(maybemacro) is Name: name_node = maybemacro From 91953db0ce0d9a8c9d98d580df4c190a73416706 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:39:59 +0300 Subject: [PATCH 143/832] fix is_unexpanded_block_macro --- unpythonic/syntax/nameutil.py | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index 826b4b47..caf43993 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -8,7 +8,7 @@ __all__ = ["isx", "getname", "is_unexpanded_expr_macro", "is_unexpanded_block_macro"] -from ast import Name, Attribute, Subscript, Call +from ast import Name, Attribute, Subscript, Call, With import sys from mcpyrate.core import Done @@ -94,7 +94,9 @@ def getname(tree, accept_attr=True): return None # TODO: This utility really wants to live in `mcpyrate`, as part of a macro destructuring subsystem. -# TODO: It needs to be made more general, to detect also macro invocations with args. +# TODO: It needs to be made more general: +# - detect also macro invocations that have macro arguments +# - destructure macro arguments, if any def is_unexpanded_expr_macro(macrofunction, expander, tree): """Check whether `tree` is an expr macro invocation bound to `macrofunction` in `expander`. @@ -119,31 +121,35 @@ def is_unexpanded_expr_macro(macrofunction, expander, tree): else: return False + # extract the expr macro = expander.isbound(name_node.id) if macro is macrofunction: if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - return tree.slice + body = tree.slice else: - return tree.slice.value + body = tree.slice.value + return body return False # TODO: This utility really wants to live in `mcpyrate`, as part of a macro destructuring subsystem. -# TODO: It needs to be made more general, to detect if there are several macros in the same `with`. +# TODO: It needs to be made more general: +# - detect if there are several macros in the same `with` +# - destructure macro arguments, if any +# - destructure as-part, if any def is_unexpanded_block_macro(macrofunction, expander, tree): """Check whether `tree` is an expr macro invocation bound to `macrofunction` in `expander`. This accounts for hygienic macro captures and as-imports. - If there is a match, return the subscript slice, i.e. the tree that would be passed - to the macro function by the expander if the macro was expanded normally. - - **CAUTION**: This function doesn't currently support detecting macros that - take macro arguments. + **CAUTION**: This function doesn't currently support several macros in the same `with`. """ - if not type(tree) is Subscript: + if not type(tree) is With: return False - maybemacro = tree.value + ctxmanager = tree.items[0].context_expr + # optvars = tree.items[0].optional_vars # as-part + # body = tree.body + maybemacro = ctxmanager # discard args if any if type(maybemacro) is Subscript: @@ -163,3 +169,5 @@ def is_unexpanded_block_macro(macrofunction, expander, tree): macro = expander.isbound(name_node.id) return macro is macrofunction + +# TODO: We might also need a utility to detect decorator macros. From 75f971df4b8bc39f54b53589345de892bedb56bb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 13:40:18 +0300 Subject: [PATCH 144/832] let_syntax/abbrev: show unexpanded source code in expr/block errors A.k.a. helper macro expansion order acrobatics, vol XII. --- unpythonic/syntax/letdo.py | 2 +- unpythonic/syntax/letsyntax.py | 111 ++++++++++++++++++++++++--------- 2 files changed, 81 insertions(+), 32 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 98110d37..fd79e391 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -841,7 +841,7 @@ def transform(self, tree): return tree, c.collected def check_strays(ismatch, tree): - class StrayHelperMacroChecker(ASTVisitor): + class StrayHelperMacroChecker(ASTVisitor): # TODO: refactor this? def examine(self, tree): if is_captured_value(tree): return # don't recurse! diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index a9692781..32aced86 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -14,14 +14,16 @@ import sys from mcpyrate import parametricmacro -from mcpyrate.expander import MacroExpander from mcpyrate.quotes import is_captured_value -from mcpyrate.utils import rename, extract_bindings -from mcpyrate.walkers import ASTTransformer +from mcpyrate.utils import rename +from mcpyrate.walkers import ASTTransformer, ASTVisitor from .letdo import _implicit_do, _destructure_and_apply_let +from .nameutil import is_unexpanded_block_macro from .util import eliminate_ifones +from ..dynassign import dyn + # -------------------------------------------------------------------------------- # Macro interface @@ -132,22 +134,13 @@ def let_syntax(tree, *, args, syntax, expander, **kw): if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("let_syntax (block mode) does not take an as-part") - # Expand other inner macros now, but not `with expr` or `with block`; - # we'll transform those away manually. - expr_and_block_bindings = extract_bindings(expander.bindings, expr, block) - other_bindings = {k: v for k, v in expander.bindings.items() if k not in expr_and_block_bindings} - tree = MacroExpander(other_bindings, filename=expander.filename).visit(tree) - if syntax == "expr": - tree = _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, + _let_syntax_expr_inside_out = partial(_let_syntax_expr, expand_inside=True) + return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr_inside_out, allow_call_in_name_position=True) else: # syntax == "block": - tree = _let_syntax_block(block_body=tree) - - # Now, we can make incorrectly placed `with expr` and `with block` error out. - # (With nested `with let_syntax` invocations only the outermost one will do this, - # but that should be fine.) - return MacroExpander(expr_and_block_bindings, filename=expander.filename).visit(tree) + with dyn.let(_macro_expander=expander): + return _let_syntax_block(block_body=tree, expand_inside=True) @parametricmacro def abbrev(tree, *, args, syntax, expander, **kw): @@ -178,9 +171,12 @@ def abbrev(tree, *, args, syntax, expander, **kw): # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. if syntax == "expr": - return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr, allow_call_in_name_position=True) + _let_syntax_expr_outside_in = partial(_let_syntax_expr, expand_inside=False) + return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr_outside_in, + allow_call_in_name_position=True) else: - return _let_syntax_block(block_body=tree) + with dyn.let(_macro_expander=expander): + return _let_syntax_block(block_body=tree, expand_inside=False) @parametricmacro def expr(tree, *, syntax, **kw): @@ -199,10 +195,19 @@ def block(tree, *, syntax, **kw): # -------------------------------------------------------------------------------- # Syntax transformers -# let_syntax[...][...] -# let_syntax[(...) in ...] -# let_syntax[..., where(...)] -def _let_syntax_expr(bindings, body): # bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) +# let_syntax[(lhs, rhs), ...][body] +# let_syntax[(lhs, rhs), ...][[body0, ...]] +# let_syntax[((lhs, rhs), ...) in body] +# let_syntax[((lhs, rhs), ...) in [body0, ...]] +# let_syntax[body, where((lhs, rhs), ...)] +# let_syntax[[body0, ...], where((lhs, rhs), ...)] +# +# This transformer takes destructured input, with the bindings subform +# and the body already extracted, and supplied separately. +# +# bindings: sequence of ast.Tuple: (k1, v1), (k2, v2), ..., (kn, vn) +# expand_inside: if True, expand inside-out. If False, expand outside-in. +def _let_syntax_expr(bindings, body, *, expand_inside): body = _implicit_do(body) # support the extra bracket syntax if not bindings: # Optimize out a `let_syntax` with no bindings. return body # pragma: no cover @@ -220,6 +225,9 @@ def register_bindings(): target = templates if args else barenames target.append((name, args, value, "expr")) + if expand_inside: + bindings = dyn._macro_expander.visit(bindings) + body = dyn._macro_expander.visit(body) register_bindings() body = _substitute_templates(templates, body) body = _substitute_barenames(barenames, body) @@ -239,11 +247,33 @@ def register_bindings(): # body0 # ... # -def _let_syntax_block(block_body): +# expand_inside: if True, expand inside-out. If False, expand outside-in. +def _let_syntax_block(block_body, *, expand_inside): + is_let_syntax = partial(is_unexpanded_block_macro, let_syntax, dyn._macro_expander) + is_abbrev = partial(is_unexpanded_block_macro, abbrev, dyn._macro_expander) + is_expr_declaration = partial(is_unexpanded_block_macro, expr, dyn._macro_expander) + is_block_declaration = partial(is_unexpanded_block_macro, block, dyn._macro_expander) + is_helper_macro = lambda tree: is_expr_declaration(tree) or is_block_declaration(tree) + def check_strays(ismatch, tree): + class StrayHelperMacroChecker(ASTVisitor): # TODO: refactor this? + def examine(self, tree): + if is_captured_value(tree): + return # don't recurse! + elif is_let_syntax(tree) or is_abbrev(tree): + return # don't recurse! + elif ismatch(tree): + # Expand the stray helper macro invocation, to trigger its `SyntaxError` + # with a useful message, and *make the expander generate a use site traceback*. + # + # (If we just `raise` here directly, the expander won't see the use site + # of the `with expr` or `with block`, but just that of the `do[]`.) + dyn._macro_expander.visit(tree) + self.generic_visit(tree) + StrayHelperMacroChecker().visit(tree) + check_stray_blocks_and_exprs = partial(check_strays, is_helper_macro) + names_seen = set() - templates = [] - barenames = [] - def register_binding(withstmt, mode, kind): + def destructure_binding(withstmt, mode, kind): assert mode in ("block", "expr") assert kind in ("barename", "template") ctxmanager = withstmt.items[0].context_expr @@ -275,8 +305,8 @@ def register_binding(withstmt, mode, kind): raise SyntaxError("'with expr:' expected an expression body, got a statement") # pragma: no cover value = theexpr.value # discard Expr wrapper in definition names_seen.add(name) - target = templates if args else barenames - target.append((name, args, value, mode)) + + return name, args, value, mode def isbinding(tree): for mode in ("block", "expr"): @@ -294,14 +324,33 @@ def isbinding(tree): return mode, "template" return False + templates = [] + barenames = [] new_block_body = [] for stmt in block_body: + # `let_syntax` mode (expand_inside): respect lexical scoping of nested `let_syntax`/`abbrev` + expanded = False + if expand_inside and (is_let_syntax(stmt) or is_abbrev(stmt)): + stmt = dyn._macro_expander.visit(stmt) + expanded = True + stmt = _substitute_templates(templates, stmt) stmt = _substitute_barenames(barenames, stmt) binding_data = isbinding(stmt) if binding_data: - register_binding(stmt, *binding_data) + name, args, value, mode = destructure_binding(stmt, *binding_data) + + check_stray_blocks_and_exprs(value) # before expanding it! + if expand_inside and not expanded: + value = dyn._macro_expander.visit(value) + + target = templates if args else barenames + target.append((name, args, value, mode)) else: + check_stray_blocks_and_exprs(stmt) # before expanding it! + if expand_inside and not expanded: + stmt = dyn._macro_expander.visit(stmt) + new_block_body.append(stmt) new_block_body = eliminate_ifones(new_block_body) if not new_block_body: @@ -362,7 +411,7 @@ def subst(): return tree elif isthisname(tree): if mode == "block": - raise SyntaxError("cannot substitute a block into expression position") # pragma: no cover + raise SyntaxError(f"cannot substitute block '{name}' into expression position") # pragma: no cover tree = subst() return self.generic_visit(tree) return self.generic_visit(tree) @@ -418,7 +467,7 @@ def transform(self, tree): return tree elif isthisfunc(tree): if mode == "block": - raise SyntaxError("cannot substitute a block into expression position") # pragma: no cover + raise SyntaxError(f"cannot substitute block '{name}' into expression position") # pragma: no cover tree = subst(tree) return self.generic_visit(tree) return self.generic_visit(tree) From a92db45739b7bcc872feceddd6c1b09730eed567 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 14:20:10 +0300 Subject: [PATCH 145/832] remove the local `ansicolor` module It was only used internally by the testing framework, and that framework depends on `mcpyrate` anyway (since the `test` constructs are macros), so we can just use `mcpyrate.colorizer`, to DRY out the colorization stuff. This needs an upgrade to `mcpyrate`, because `colorama` 0.4.4 is missing a couple of styles, particularly ITALIC. There's new code in `mcpyrate` master that injects the missing styles (if indeed missing) when the colorizer loads. --- unpythonic/syntax/__init__.py | 4 - unpythonic/test/ansicolor.py | 95 ------------------ unpythonic/test/fixtures.py | 184 ++++++++++++++++++++-------------- 3 files changed, 106 insertions(+), 177 deletions(-) delete mode 100644 unpythonic/test/ansicolor.py diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 1e0a9704..58848b52 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -104,10 +104,6 @@ # TODO: grep codebase for "0.15", may have some pending interface changes that don't have their own GitHub issue (e.g. parameter ordering of `unpythonic.it.window`) -# TODO: ansicolor: `mcpyrate` already depends on Colorama anyway (and has a *nix-only fallback capability). -# TODO: `unpythonic` only needs the colorizer in the *macro-enabled* test framework; so we don't really need -# TODO: to provide our own colorizer; we can use the one from `mcpyrate`. (It would be different if regular code needed it.) - # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... # TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. diff --git a/unpythonic/test/ansicolor.py b/unpythonic/test/ansicolor.py deleted file mode 100644 index f7312975..00000000 --- a/unpythonic/test/ansicolor.py +++ /dev/null @@ -1,95 +0,0 @@ -# -*- coding: utf-8; -*- -"""ANSI color support for *nix terminals. - -For a serious library that does this sort of thing in a cross-platform way, -see Colorama: - https://github.com/tartley/colorama -""" - -# TODO: We could also use Colorama (which also works on Windows), but that's one more dependency. -# TODO: Maybe this module should live in unpythonic.net, though we don't currently use it there. - -from enum import Enum - -__all__ = ["TC", "colorize"] - -class TC(Enum): - """Terminal colors, via ANSI escape sequences. - - This uses the terminal app palette (16 colors), so e.g. LIGHTGREEN may actually - be blue, depending on the user's color scheme. - - The colors are listed in palette order. - - See: - https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors - https://github.com/tartley/colorama - """ - # For grepping: \33 octal is \x1b hex. - RESET = '\33[0m' # return to normal state, ending colorization - RESETSTYLE = '\33[22m' # return to normal brightness - RESETFG = '\33[39m' - RESETBG = '\33[49m' - - # styles - BRIGHT = '\33[1m' # a.k.a. bold - DIM = '\33[2m' - ITALIC = '\33[3m' - URL = '\33[4m' # underline plus possibly a special color (depends on terminal app) - BLINK = '\33[5m' - BLINK2 = '\33[6m' # same effect as BLINK? - SELECTED = '\33[7m' - - # foreground colors - BLACK = '\33[30m' - RED = '\33[31m' - GREEN = '\33[32m' - YELLOW = '\33[33m' - BLUE = '\33[34m' - MAGENTA = '\33[35m' - CYAN = '\33[36m' - WHITE = '\33[37m' - LIGHTBLACK = '\33[90m' - LIGHTRED = '\33[91m' - LIGHTGREEN = '\33[92m' - LIGHTYELLOW = '\33[93m' - LIGHTBLUE = '\33[94m' - LIGHTMAGENTA = '\33[95m' - LIGHTCYAN = '\33[96m' - LIGHTWHITE = '\33[97m' - - # background colors - BLACKBG = '\33[40m' - REDBG = '\33[41m' - GREENBG = '\33[42m' - YELLOWBG = '\33[43m' - BLUEBG = '\33[44m' - MAGENTABG = '\33[45m' - CYANBG = '\33[46m' - WHITEBG = '\33[47m' - -def colorize(s, *colors): - """Colorize string `s` for ANSI terminal display. Reset color at end of `s`. - - For available `colors`, see the `TC` enum. - - Usage:: - - colorize("I'm new here", TC.GREEN) - colorize("I'm bold and bluetiful", TC.BRIGHT, TC.BLUE) - - Each entry can also be a `tuple` (arbitrarily nested), which is useful - for defining compound styles:: - - BRIGHT_BLUE = (TC.BRIGHT, TC.BLUE) - ... - colorize("I'm bold and bluetiful, too", BRIGHT_BLUE) - """ - def get_ansi_color_sequence(c): # recursive, so each entry can be a tuple. - if isinstance(c, tuple): - return "".join(get_ansi_color_sequence(elt) for elt in c) - if not isinstance(c, TC): - raise TypeError(f"Expected a TC instance, got {type(c)} with value {repr(c)}") # pragma: no cover - return c.value - return f"{get_ansi_color_sequence(colors)}{s}{get_ansi_color_sequence(TC.RESET)}" diff --git a/unpythonic/test/fixtures.py b/unpythonic/test/fixtures.py index b47d24c2..39de74f3 100644 --- a/unpythonic/test/fixtures.py +++ b/unpythonic/test/fixtures.py @@ -116,12 +116,22 @@ import threading import sys +# The testing framework depends on `mcpyrate` anyway, because the test +# constructs are macros. +# +# This regular-code module depends on `mcpyrate`'s colorizer, but since +# `unpythonic.test` is not auto-loaded, it's fine. +# +# Using `Bunch` is debatable, since we have `env`, and `Bunch` is essentially +# just a stripped-down version of that. But `mcpyrate` uses `Bunch` for storing +# config constants, so meh - let's just use the same approach here for consistency. +from mcpyrate.bunch import Bunch +from mcpyrate.colorizer import Fore, Style, colorize + from ..conditions import handlers, find_restart, invoke from ..collections import box, unbox from ..symbol import sym -from .ansicolor import TC, colorize - __all__ = ["session", "testset", "terminate", "returns_normally", "catch_signals", @@ -297,13 +307,46 @@ def maybe_colorize(s, *colors): If color is disabled (`TestConfig.use_color` is falsey), then no-op, i.e. return the original `s` as-is. - See `unpythonic.test.ansicolor.colorize` for details. + See `mcpyrate.colorizer.colorize` for details. """ if not TestConfig.use_color: return s return colorize(s, *colors) -class TestConfig: +# We instantiate this later, since the instance lives inside `TestConfig` anyway. +class ColorScheme(Bunch): + """The color scheme for terminal output in `unpythonic`'s testing framework. + + This is just a bunch of constants. To change the colors, simply assign new + values to them. Changes take effect immediately for any new output. + + To replace the whole color scheme at once, fill in a suitable `Bunch`, and + then use the `replace` method. If you need to get the names of all settings + programmatically, call the `keys` method. + + Don't replace the color scheme object itself. + + See `Fore`, `Back` and `Style` in `mcpyrate.colorizer` for valid values. + To make a compound style, place the values into a tuple. + + The defaults are designed to fit the "Solarized" (Zenburn-like) theme + of `gnome-terminal`, with "Show bold text in bright colors" set to OFF. + But they work also with "Tango", and indeed with most themes. + """ + def __init__(self): + super().__init__() + + self.HEADING = Fore.LIGHTBLUE_EX + self.PASS = Fore.GREEN + self.FAIL = Fore.LIGHTRED_EX + self.ERROR = Fore.YELLOW + self.WARNING = Fore.YELLOW + self.GREYED_OUT = (Style.DIM, self.HEADING) + # These colors are used for the pass percentage. + self.SUMMARY_OK = Fore.GREEN + self.SUMMARY_NOTOK = Fore.YELLOW # more readable than red on a dark background, yet stands out. + +class TestConfig(Bunch): """Global settings for the testing utilities. This is just a bunch of constants. @@ -318,7 +361,7 @@ class TestConfig: Default is `True`. `postproc`: Exception -> None; optional. Default None (no postproc). `indent_per_level`: How many indent to indent per nesting level of `testset`. - `CS`: The color scheme. + `ColorScheme`: The color scheme. The optional `postproc` is a custom callback for examining failures and errors. `TestConfig.postproc` sets the default that is used when no other @@ -334,45 +377,30 @@ class TestConfig: If you want a failure in a particular testset to abort the whole unit, you can use `terminate` as your `postproc`. """ - # It is overwhelmingly common that tests are invoked from a single thread, - # so by default, all threads share the same printer. (It is not worth - # complicating the common use case here to cater for the rare use case.) - # - # However, if you want different printers in different threads, that can - # be done. As `printer`, use a `Shim` that contains a `ThreadLocalBox`. - # In each thread, place in that box a custom object that has a `__call__` - # method that takes the same args `print` does. Because `Shim` redirects - # all attribute accesses, it will redirect the lookup of `__call__` - # (it doesn't have its own `__call__`, so it assumes the client wants to - # call the thing that is inside the box), and hence that method will then - # be used for printing. - # - # TODO: This is subject to change later if I figure out a better design - # TODO: that conveniently caters for *both* the common and rare use cases. - printer = partial(print, file=sys.stderr) - use_color = True - postproc = None - indent_per_level = 2 - - class CS: - """The color scheme. - - See the `unpythonic.test.ansicolor.TC` enum for valid values. To make a - compound style, place the values into a tuple. - - The defaults are designed to fit the "Solarized" (Zenburn-like) theme - of `gnome-terminal`, with "Show bold text in bright colors" set to OFF. - But they should work with most color schemes. - """ - HEADING = TC.LIGHTBLUE - PASS = TC.GREEN - FAIL = TC.LIGHTRED - ERROR = TC.YELLOW - WARNING = TC.YELLOW - GREYED_OUT = (TC.DIM, HEADING) - # These colors are used for the pass percentage. - SUMMARY_OK = TC.GREEN - SUMMARY_NOTOK = TC.YELLOW # more readable than red on a dark background, yet stands out. + def __init__(self): + super().__init__() + + # It is overwhelmingly common that tests are invoked from a single thread, + # so by default, all threads share the same printer. (It is not worth + # complicating the common use case here to cater for the rare use case.) + # + # However, if you want different printers in different threads, that can + # be done. As `printer`, use a `Shim` that contains a `ThreadLocalBox`. + # In each thread, place in that box a custom object that has a `__call__` + # method that takes the same args `print` does. Because `Shim` redirects + # all attribute accesses, it will redirect the lookup of `__call__` + # (it doesn't have its own `__call__`, so it assumes the client wants to + # call the thing that is inside the box), and hence that method will then + # be used for printing. + # + # TODO: This is subject to change later if I figure out a better design + # TODO: that conveniently caters for *both* the common and rare use cases. + self.printer = partial(print, file=sys.stderr) + self.use_color = True + self.postproc = None + self.indent_per_level = 2 + self.ColorScheme = ColorScheme() +TestConfig = TestConfig() # type: ignore[assignment, misc] def describe_exception(exc): """Return a human-readable (possibly multi-line) description of exception `exc`. @@ -390,7 +418,7 @@ def describe_instance(instance): if instance.__traceback__ is not None: snippets.append(maybe_colorize("\nTraceback (most recent call last):\n" + - "".join(format_tb(instance.__traceback__)), TC.DIM)) + "".join(format_tb(instance.__traceback__)), Style.DIM)) msg = str(instance) if msg: @@ -445,32 +473,32 @@ def summarize(runs, fails, errors, warns): # In techni... ANSI color: snippets = [] - color = TestConfig.CS.PASS if passes else TestConfig.CS.GREYED_OUT - snippets.extend([maybe_colorize("Pass", TC.BRIGHT, color), + color = TestConfig.ColorScheme.PASS if passes else TestConfig.ColorScheme.GREYED_OUT + snippets.extend([maybe_colorize("Pass", Style.BRIGHT, color), " ", maybe_colorize(f"{passes}", color), - maybe_colorize(", ", TestConfig.CS.HEADING)]) - color = TestConfig.CS.FAIL if fails else TestConfig.CS.GREYED_OUT - snippets.extend([maybe_colorize("Fail", TC.BRIGHT, color), + maybe_colorize(", ", TestConfig.ColorScheme.HEADING)]) + color = TestConfig.ColorScheme.FAIL if fails else TestConfig.ColorScheme.GREYED_OUT + snippets.extend([maybe_colorize("Fail", Style.BRIGHT, color), " ", maybe_colorize(f"{fails}", color), - maybe_colorize(", ", TestConfig.CS.HEADING)]) - color = TestConfig.CS.ERROR if errors else TestConfig.CS.GREYED_OUT - snippets.extend([maybe_colorize("Error", TC.BRIGHT, color), + maybe_colorize(", ", TestConfig.ColorScheme.HEADING)]) + color = TestConfig.ColorScheme.ERROR if errors else TestConfig.ColorScheme.GREYED_OUT + snippets.extend([maybe_colorize("Error", Style.BRIGHT, color), " ", maybe_colorize(f"{errors}", color), - maybe_colorize(", ", TestConfig.CS.HEADING)]) - color = TestConfig.CS.HEADING if runs else TestConfig.CS.GREYED_OUT - snippets.extend([maybe_colorize("Total", TC.BRIGHT, color), + maybe_colorize(", ", TestConfig.ColorScheme.HEADING)]) + color = TestConfig.ColorScheme.HEADING if runs else TestConfig.ColorScheme.GREYED_OUT + snippets.extend([maybe_colorize("Total", Style.BRIGHT, color), " ", maybe_colorize(f"{runs}", color)]) - color = TestConfig.CS.SUMMARY_OK if passes == runs else TestConfig.CS.SUMMARY_NOTOK + color = TestConfig.ColorScheme.SUMMARY_OK if passes == runs else TestConfig.ColorScheme.SUMMARY_NOTOK snippets.extend([" ", - maybe_colorize(f"({int(pass_percentage)}% pass)", TC.BRIGHT, color)]) + maybe_colorize(f"({int(pass_percentage)}% pass)", Style.BRIGHT, color)]) if warns > 0: - color = TestConfig.CS.WARNING + color = TestConfig.ColorScheme.WARNING snippets.extend([" ", - maybe_colorize(f"+ {warns} Warn", TC.BRIGHT, color)]) + maybe_colorize(f"+ {warns} Warn", Style.BRIGHT, color)]) return "".join(snippets) class TestSessionExit(Exception): @@ -482,7 +510,7 @@ def terminate(exc=None): # the parameter is ignored this can be used as a `postproc`, if you want a failure in a particular testset to abort the session. """ - TestConfig.printer(maybe_colorize("** TERMINATING SESSION", TC.BRIGHT, TestConfig.CS.HEADING)) + TestConfig.printer(maybe_colorize("** TERMINATING SESSION", Style.BRIGHT, TestConfig.ColorScheme.HEADING)) raise TestSessionExit def returns_normally(expr): @@ -550,11 +578,11 @@ def session(name=None): if _threadlocals.nesting_level > 0: raise RuntimeError("A test `session` cannot be nested inside a `testset`.") - title = maybe_colorize("SESSION", TC.BRIGHT, TestConfig.CS.HEADING) + title = maybe_colorize("SESSION", Style.BRIGHT, TestConfig.ColorScheme.HEADING) if name is not None: - title += maybe_colorize(f" '{name}'", TC.ITALIC, TestConfig.CS.HEADING) - TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.CS.HEADING) + - maybe_colorize("BEGIN", TC.BRIGHT, TestConfig.CS.HEADING)) + title += maybe_colorize(f" '{name}'", Style.ITALIC, TestConfig.ColorScheme.HEADING) + TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.ColorScheme.HEADING) + + maybe_colorize("BEGIN", Style.BRIGHT, TestConfig.ColorScheme.HEADING)) # We are paused when the user triggers the exception; `contextlib` detects the # exception and re-raises it into us. @@ -569,8 +597,8 @@ def session(name=None): except TestSessionExit: pass - TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.CS.HEADING) + - maybe_colorize("END", TC.BRIGHT, TestConfig.CS.HEADING)) + TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.ColorScheme.HEADING) + + maybe_colorize("END", Style.BRIGHT, TestConfig.ColorScheme.HEADING)) # We use a stack for postprocs so that the local overrides can be nested. _threadlocals.postproc_stack = deque() @@ -605,9 +633,9 @@ def makeindent(level): title = f"{indent}Testset" if name is not None: - title += maybe_colorize(f" '{name}'", TC.ITALIC) - TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.CS.HEADING) + - maybe_colorize("BEGIN", TC.BRIGHT, TestConfig.CS.HEADING)) + title += maybe_colorize(f" '{name}'", Style.ITALIC) + TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.ColorScheme.HEADING) + + maybe_colorize("BEGIN", Style.BRIGHT, TestConfig.ColorScheme.HEADING)) def print_and_proceed(condition): # The assert helpers in `unpythonic.syntax.testingtools` signal only @@ -615,13 +643,13 @@ def print_and_proceed(condition): # inside the test expression. if isinstance(condition, TestFailure): msg = maybe_colorize(f"{errmsg_indent}FAIL: ", - TC.BRIGHT, TestConfig.CS.FAIL) + str(condition) + Style.BRIGHT, TestConfig.ColorScheme.FAIL) + str(condition) elif isinstance(condition, TestError): msg = maybe_colorize(f"{errmsg_indent}ERROR: ", - TC.BRIGHT, TestConfig.CS.ERROR) + str(condition) + Style.BRIGHT, TestConfig.ColorScheme.ERROR) + str(condition) elif isinstance(condition, TestWarning): msg = maybe_colorize(f"{errmsg_indent}WARNING: ", - TC.BRIGHT, TestConfig.CS.WARNING) + str(condition) + Style.BRIGHT, TestConfig.ColorScheme.WARNING) + str(condition) # So any other signal must come from another source. else: if not _threadlocals.catch_uncaught_signals[0]: @@ -630,7 +658,7 @@ def print_and_proceed(condition): _update(tests_run, +1) _update(tests_errored, +1) msg = maybe_colorize(f"{errmsg_indent}Testset received signal outside test[]: ", - TC.BRIGHT, TestConfig.CS.ERROR) + describe_exception(condition) + Style.BRIGHT, TestConfig.ColorScheme.ERROR) + describe_exception(condition) TestConfig.printer(msg) # the custom callback @@ -670,7 +698,7 @@ def print_and_proceed(condition): _update(tests_run, +1) _update(tests_errored, +1) msg = maybe_colorize(f"{errmsg_indent}Testset terminated by exception outside test[]: ", - TC.BRIGHT, TestConfig.CS.ERROR) + Style.BRIGHT, TestConfig.ColorScheme.ERROR) msg += describe_exception(err) TestConfig.printer(msg) finally: @@ -685,8 +713,8 @@ def print_and_proceed(condition): errors = e2 - e1 warns = w2 - w1 - msg = (maybe_colorize(f"{title} ", TestConfig.CS.HEADING) + - maybe_colorize("END", TC.BRIGHT, TestConfig.CS.HEADING) + - maybe_colorize(": ", TestConfig.CS.HEADING) + + msg = (maybe_colorize(f"{title} ", TestConfig.ColorScheme.HEADING) + + maybe_colorize("END", Style.BRIGHT, TestConfig.ColorScheme.HEADING) + + maybe_colorize(": ", TestConfig.ColorScheme.HEADING) + summarize(runs, fails, errors, warns)) TestConfig.printer(msg) From 9e8a2c99d44d6f64701a13f28c416a51d92a5769 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 14:33:05 +0300 Subject: [PATCH 146/832] oops, fix testset name coloring --- unpythonic/test/fixtures.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/test/fixtures.py b/unpythonic/test/fixtures.py index 39de74f3..486e8d66 100644 --- a/unpythonic/test/fixtures.py +++ b/unpythonic/test/fixtures.py @@ -633,7 +633,7 @@ def makeindent(level): title = f"{indent}Testset" if name is not None: - title += maybe_colorize(f" '{name}'", Style.ITALIC) + title += maybe_colorize(f" '{name}'", Style.ITALIC, TestConfig.ColorScheme.HEADING) TestConfig.printer(maybe_colorize(f"{title} ", TestConfig.ColorScheme.HEADING) + maybe_colorize("BEGIN", Style.BRIGHT, TestConfig.ColorScheme.HEADING)) From 428d654503922013f358591ffd5144f579d70ea1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:44:36 +0300 Subject: [PATCH 147/832] prefix: make q, u, kw error at compile time in invalid positions --- CHANGELOG.md | 2 +- unpythonic/dialects/listhell.py | 3 +- unpythonic/syntax/__init__.py | 2 - unpythonic/syntax/prefix.py | 51 +++++++++++++++----------- unpythonic/syntax/tests/test_prefix.py | 3 +- 5 files changed, 33 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 68cc77b6..9010308c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) together with a kitchen-sink language extension macro package such as `unpythonic`. - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) -- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), and `with expr`/`with block` (for `let_syntax`/`abbrev`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. +- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), `with expr`/`with block` (for `let_syntax`/`abbrev`), and `q`/`u`/`kw` (for `prefix`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. - `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. - Python 3.8 and 3.9 support added. diff --git a/unpythonic/dialects/listhell.py b/unpythonic/dialects/listhell.py index f26b1dae..35ece7d4 100644 --- a/unpythonic/dialects/listhell.py +++ b/unpythonic/dialects/listhell.py @@ -17,9 +17,8 @@ class Listhell(Dialect): def transform_ast(self, tree): # tree is an ast.Module with q as template: __lang__ = "Listhell" # noqa: F841, just provide it to user code. - from unpythonic.syntax import macros, prefix, autocurry # noqa: F401, F811 + from unpythonic.syntax import macros, prefix, q, u, kw, autocurry # noqa: F401, F811 # Auxiliary syntax elements for the macros - from unpythonic.syntax import q, u, kw # noqa: F401 from unpythonic import apply # noqa: F401 from unpythonic import composerc as compose # compose from Right, Currying # noqa: F401 with prefix, autocurry: diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 58848b52..dca4a782 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -78,8 +78,6 @@ # However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. # Cleanups can be done in a future release. -# TODO: upgrade kw() in unpythonic.syntax.prefix into `mcpyrate` magic variable? - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 1d6e38b4..d67765e6 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -11,6 +11,7 @@ from mcpyrate.quotes import macros, q, u, a, t # noqa: F811, F401 +from mcpyrate import namemacro from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer @@ -103,27 +104,35 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 # not having to worry about tuples possibly denoting function calls. return _prefix(block_body=tree) -# Note the exported "q" and "u" are ours, but the "q" and "u" we use in this -# module are macros. The "q" and "u" we define here are regular run-time objects, -# namely the stubs for the "q" and "u" markers used within a `prefix` block. -class q: # noqa: F811 - """[syntax] Quote operator. Only meaningful in a tuple in a prefix block.""" - def __repr__(self): # in case one of these ends up somewhere at runtime # pragma: no cover - return "" -q = q() - -class u: # noqa: F811 - """[syntax] Unquote operator. Only meaningful in a tuple in a prefix block.""" - def __repr__(self): # in case one of these ends up somewhere at runtime # pragma: no cover - return "" -u = u() - -# TODO: Think of promoting this error to compile macro expansion time. -# TODO: Difficult to do, because we shouldn't probably hijack the name "kw" (so no name macro), -# TODO: and it can't be invoked like an expr macro, because the whole point is to pass arguments by name. -def kw(**kwargs): - """[syntax] Pass-named-args operator. Only meaningful in a tuple in a prefix block.""" - raise RuntimeError("kw(...) only meaningful inside a tuple in a prefix block") # pragma: no cover +# Note the exported "q" and "u" are ours (namely the stubs for the "q" and "u" +# operators compiled away by `prefix`), but the "q[]" we use as a macro in +# this module is the quasiquote operator from `mcpyrate.quotes`. +# +# This `def` doesn't overwrite the macro `q`, because the `def` runs at run time. +# The expander does not try to expand this `q` as a macro, because `def q(...)` +# is not a valid macro invocation even when the name `q` has been imported as a macro. +@namemacro +def q(tree, *, syntax, **kw): # noqa: F811 + """[syntax, name] Quote operator. Only meaningful in a tuple inside a prefix block.""" + if syntax != "name": + raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") + raise SyntaxError("q (unpythonic.syntax.prefix.q) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander + +@namemacro +def u(tree, *, syntax, **kw): # noqa: F811 + """[syntax, name] Unquote operator. Only meaningful in a tuple inside a prefix block.""" + if syntax != "name": + raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") + raise SyntaxError("q (unpythonic.syntax.prefix.q) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander + +# TODO: This isn't a perfect solution, because there is no "call" macro kind. +# TODO: We currently trigger the error on any appearance of the name `kw` outside a valid context. +@namemacro +def kw(tree, *, syntax, **kw): # noqa: F811 + """[syntax] Pass-named-args operator. Only meaningful in a tuple inside a prefix block.""" + if syntax != "name": + raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is a name macro only") + raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander # -------------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tests/test_prefix.py b/unpythonic/syntax/tests/test_prefix.py index 5bf306c4..5c164ec7 100644 --- a/unpythonic/syntax/tests/test_prefix.py +++ b/unpythonic/syntax/tests/test_prefix.py @@ -6,8 +6,7 @@ from ...syntax import macros, test, test_raises, the # noqa: F401 from ...test.fixtures import session, testset, returns_normally -from ...syntax import macros, prefix, autocurry, let, do # noqa: F401, F811 -from ...syntax import q, u, kw +from ...syntax import macros, prefix, q, u, kw, autocurry, let, do # noqa: F401, F811 from ...fold import foldr from ...fun import composerc as compose, apply From 379a45386ad94eb69d4e497b1d76ff2a66fbfd07 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:45:14 +0300 Subject: [PATCH 148/832] update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9010308c..e4200e36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - CI: Test coverage improved to 94%. - Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. The `continuations` macro also outputs a hygienically captured `aif` when transforming an `or` expression that occurs in tail position. - This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) + - The implicit do (extra bracket syntax) also expands as a hygienically captured `do`, but e.g. in `let[]` it will then expand immediately (due to `let`'s inside-out expansion order) before control returns to the macro stepper. If you want to see the implicit `do[]` invocation, use the `"detailed"` mode of the stepper, which shows individual macro invocations even when expanding inside-out: `step_expansion["detailed"][...]`, `with step_expansion["detailed"]:`. **Breaking changes**: From 0b6f0af44640f9df6c9185df29ddcc83988057cc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:46:29 +0300 Subject: [PATCH 149/832] update language version note --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 623f6abb..00819446 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ None required. - [mcpyrate](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -The officially supported language versions are **CPython 3.8** and **PyPy3 3.7**. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +The officially supported language versions are **CPython 3.8** and **PyPy3** (language version 3.7). [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. From 923862cd8ba274d56a5957a87af67a3d498bbef8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:46:52 +0300 Subject: [PATCH 150/832] bump mcpyrate in requirements, for colorizer updates --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index b1112c4f..c33fd1df 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -mcpyrate>=3.4.0 +mcpyrate>=3.4.1 sympy>=1.4 From f4bbacc6c329b2a7584f62bf41330e6c742798c9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:56:14 +0300 Subject: [PATCH 151/832] add `# pragma: no cover` to all compile-time syntax errors --- unpythonic/syntax/autocurry.py | 4 ++-- unpythonic/syntax/autoref.py | 6 +++--- unpythonic/syntax/dbg.py | 6 +++--- unpythonic/syntax/forall.py | 4 ++-- unpythonic/syntax/ifexprs.py | 10 ++++----- unpythonic/syntax/lambdatools.py | 18 ++++++++-------- unpythonic/syntax/lazify.py | 8 ++++---- unpythonic/syntax/letdo.py | 22 ++++++++++---------- unpythonic/syntax/letsyntax.py | 12 +++++------ unpythonic/syntax/nb.py | 4 ++-- unpythonic/syntax/prefix.py | 10 ++++----- unpythonic/syntax/tailtools.py | 12 +++++------ unpythonic/syntax/testingtools.py | 34 +++++++++++++++---------------- 13 files changed, 75 insertions(+), 75 deletions(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 580bf655..602021e4 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -64,9 +64,9 @@ def add3(a, b, c): assert add3(1)(2)(3) == 6 """ if syntax != "block": - raise SyntaxError("autocurry is a block macro only") + raise SyntaxError("autocurry is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("autocurry does not take an as-part") + raise SyntaxError("autocurry does not take an as-part") # pragma: no cover tree = expander.visit(tree) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 30f2beb8..6021c659 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -138,13 +138,13 @@ def autoref(tree, *, args, syntax, expander, **kw): attributes can be added and removed dynamically. """ if syntax != "block": - raise SyntaxError("autoref is a block macro only") + raise SyntaxError("autoref is a block macro only") # pragma: no cover if not args: - raise SyntaxError("autoref requires an argument, the object to be auto-referenced") + raise SyntaxError("autoref requires an argument, the object to be auto-referenced") # pragma: no cover target = kw.get("optional_vars", None) if target and type(target) is not Name: # tuples not accepted - raise SyntaxError("with autoref[...] as ... takes at most one name in the as-part") + raise SyntaxError("with autoref[...] as ... takes at most one name in the as-part") # pragma: no cover with dyn.let(_macro_expander=expander): return _autoref(block_body=tree, args=args, asname=target) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index c035bc5c..13f0891f 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -99,9 +99,9 @@ def dbg(tree, *, args, syntax, expander, **kw): extra parentheses). See ``mcpyrate.unparse``. """ if syntax not in ("expr", "block"): - raise SyntaxError("dbg is an expr and block macro only") + raise SyntaxError("dbg is an expr and block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("dbg (block mode) does not take an as-part") + raise SyntaxError("dbg (block mode) does not take an as-part") # pragma: no cover tree = expander.visit(tree) @@ -206,7 +206,7 @@ def _dbg_block(body, args): # TODO: add support for Attribute to support using a method as a custom print function # (the problem is we must syntactically find matches in the AST, and AST nodes don't support comparison) if type(args[0]) is not Name: # pragma: no cover, let's not test the macro expansion errors. - raise SyntaxError("Custom debug print function must be specified by a bare name") + raise SyntaxError("Custom debug print function must be specified by a bare name") # pragma: no cover pfunc = args[0] pname = pfunc.id # name of the print function as it appears in the user code else: diff --git a/unpythonic/syntax/forall.py b/unpythonic/syntax/forall.py index b4423e75..95e4337e 100644 --- a/unpythonic/syntax/forall.py +++ b/unpythonic/syntax/forall.py @@ -33,7 +33,7 @@ def forall(tree, *, syntax, expander, **kw): (8, 15, 17), (9, 12, 15), (12, 16, 20)) """ if syntax != "expr": - raise SyntaxError("forall is an expr macro only") + raise SyntaxError("forall is an expr macro only") # pragma: no cover tree = expander.visit(tree) @@ -41,7 +41,7 @@ def forall(tree, *, syntax, expander, **kw): def _forall(exprs): if type(exprs) is not Tuple: # pragma: no cover, let's not test macro expansion errors. - raise SyntaxError("forall body: expected a sequence of comma-separated expressions") + raise SyntaxError("forall body: expected a sequence of comma-separated expressions") # pragma: no cover itemno = 0 def build(lines, tree): if not lines: diff --git a/unpythonic/syntax/ifexprs.py b/unpythonic/syntax/ifexprs.py index 15f6f637..a915f7e3 100644 --- a/unpythonic/syntax/ifexprs.py +++ b/unpythonic/syntax/ifexprs.py @@ -42,14 +42,14 @@ def aif(tree, *, syntax, expander, **kw): brackets: ``[[1, 2, 3]]``. """ if syntax != "expr": - raise SyntaxError("aif is an expr macro only") + raise SyntaxError("aif is an expr macro only") # pragma: no cover # Detect the name(s) of `it` at the use site (this accounts for as-imports) # TODO: We don't know which binding this particular use site uses. # TODO: For now, we hack this by making `it` always rename itself to literal `it`. macro_bindings = extract_bindings(expander.bindings, it) if not macro_bindings: - raise SyntaxError("The use site of `aif` must macro-import `it`, too.") + raise SyntaxError("The use site of `aif` must macro-import `it`, too.") # pragma: no cover # Expand outside-in, but the implicit do[] needs the expander. with dyn.let(_macro_expander=expander): @@ -91,9 +91,9 @@ def it(tree, *, syntax, **kw): without renaming. """ if syntax != "name": - raise SyntaxError("`it` is a name macro only") + raise SyntaxError("`it` is a name macro only") # pragma: no cover if _aif_level.value < 1: - raise SyntaxError("`it` may only appear in the 'then' and 'otherwise' parts of an `aif[...]`") + raise SyntaxError("`it` may only appear in the 'then' and 'otherwise' parts of an `aif[...]`") # pragma: no cover return q[it] # always rename to literal `it` # -------------------------------------------------------------------------------- @@ -122,7 +122,7 @@ def cond(tree, *, syntax, expander, **kw): brackets: ``[[1, 2, 3]]``. """ if syntax != "expr": - raise SyntaxError("cond is an expr macro only") + raise SyntaxError("cond is an expr macro only") # pragma: no cover # Expand outside-in, but the implicit do[] needs the expander. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index e0e6b206..25c45913 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -64,9 +64,9 @@ def multilambda(tree, *, syntax, expander, **kw): For local variables, see ``do``. """ if syntax != "block": - raise SyntaxError("multilambda is a block macro only") + raise SyntaxError("multilambda is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("multilambda does not take an as-part") + raise SyntaxError("multilambda does not take an as-part") # pragma: no cover # Expand outside in. # multilambda should expand first before any let[], do[] et al. that happen @@ -117,9 +117,9 @@ def namedlambda(tree, *, syntax, expander, **kw): which will update ``__name__``, ``__qualname__`` and ``__code__.co_name``. """ if syntax != "block": - raise SyntaxError("namedlambda is a block macro only") + raise SyntaxError("namedlambda is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("namedlambda does not take an as-part") + raise SyntaxError("namedlambda does not take an as-part") # pragma: no cover # Two-pass macro. We pass in the expander to allow the macro to decide when to recurse. with dyn.let(_macro_expander=expander): @@ -150,7 +150,7 @@ def f(tree, *, syntax, expander, **kw): The macro does not descend into any nested ``f[]``. """ if syntax != "expr": - raise SyntaxError("f is an expr macro only") + raise SyntaxError("f is an expr macro only") # pragma: no cover # What's my name in the current expander? (There may be several names.) # https://github.com/Technologicat/mcpyrate/blob/master/doc/quasiquotes.md#hygienic-macro-recursion @@ -190,9 +190,9 @@ def quicklambda(tree, *, syntax, expander, **kw): The point is, this combo is now possible.) """ if syntax != "block": - raise SyntaxError("quicklambda is a block macro only") + raise SyntaxError("quicklambda is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("quicklambda does not take an as-part") + raise SyntaxError("quicklambda does not take an as-part") # pragma: no cover # This macro expands outside in. # @@ -225,9 +225,9 @@ def foo(n): lambda i: n << n + i """ if syntax != "block": - raise SyntaxError("envify is a block macro only") + raise SyntaxError("envify is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("envify does not take an as-part") + raise SyntaxError("envify does not take an as-part") # pragma: no cover # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index a0ec1c32..ea4ae8b6 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -32,7 +32,7 @@ def lazy(tree, *, syntax, **kw): In Racket, this operation is known as `delay`. """ if syntax != "expr": - raise SyntaxError("lazy is an expr macro only") + raise SyntaxError("lazy is an expr macro only") # pragma: no cover # Expand outside in. Ordering shouldn't matter here. return _lazy(tree) @@ -75,7 +75,7 @@ def lazyrec(tree, *, syntax, **kw): (lazy[1+2+3], lazy[4+5+6])) """ if syntax != "expr": - raise SyntaxError("lazyrec is an expr macro only") + raise SyntaxError("lazyrec is an expr macro only") # pragma: no cover # Expand outside in. Ordering shouldn't matter here. return _lazyrec(tree) @@ -410,9 +410,9 @@ def doit(): currently the only binding constructs to which auto-lazification is applied. """ if syntax != "block": - raise SyntaxError("lazify is a block macro only") + raise SyntaxError("lazify is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("lazify does not take an as-part") + raise SyntaxError("lazify does not take an as-part") # pragma: no cover # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index fd79e391..9be601ea 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -142,7 +142,7 @@ def let(tree, *, args, syntax, expander, **kw): is applied first to ``[body0, ...]``, and the result becomes ``body``. """ if syntax != "expr": - raise SyntaxError("let is an expr macro only") + raise SyntaxError("let is an expr macro only") # pragma: no cover # The `let[]` family of macros expands inside out. with dyn.let(_macro_expander=expander): @@ -158,7 +158,7 @@ def letseq(tree, *, args, syntax, expander, **kw): Expands to nested ``let`` expressions. """ if syntax != "expr": - raise SyntaxError("letseq is an expr macro only") + raise SyntaxError("letseq is an expr macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _letseq) @@ -178,7 +178,7 @@ def letrec(tree, *, args, syntax, expander, **kw): This is useful for locally defining mutually recursive functions. """ if syntax != "expr": - raise SyntaxError("letrec is an expr macro only") + raise SyntaxError("letrec is an expr macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _letrec) @@ -209,7 +209,7 @@ def count(): lexical scope of the ``def``. """ if syntax != "decorator": - raise SyntaxError("dlet is a decorator macro only") + raise SyntaxError("dlet is a decorator macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _dlet) @@ -230,7 +230,7 @@ def g(a): assert g(10) == 14 """ if syntax != "decorator": - raise SyntaxError("dletseq is a decorator macro only") + raise SyntaxError("dletseq is a decorator macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _dletseq) @@ -251,7 +251,7 @@ def f(x): Same cautions apply as to ``dlet``. """ if syntax != "decorator": - raise SyntaxError("dletrec is a decorator macro only") + raise SyntaxError("dletrec is a decorator macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _dletrec) @@ -268,7 +268,7 @@ def result(): assert result == 42 """ if syntax != "decorator": - raise SyntaxError("blet is a decorator macro only") + raise SyntaxError("blet is a decorator macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _blet) @@ -287,7 +287,7 @@ def result(): assert result == 4 """ if syntax != "decorator": - raise SyntaxError("bletseq is a decorator macro only") + raise SyntaxError("bletseq is a decorator macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _bletseq) @@ -317,7 +317,7 @@ def result(): assert result is True """ if syntax != "decorator": - raise SyntaxError("bletrec is a decorator macro only") + raise SyntaxError("bletrec is a decorator macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _destructure_and_apply_let(tree, args, expander, _bletrec) @@ -782,14 +782,14 @@ def do(tree, *, syntax, expander, **kw): declared in a separate block, which plays the role of ``local``. """ if syntax != "expr": - raise SyntaxError("do is an expr macro only") + raise SyntaxError("do is an expr macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _do(tree) def do0(tree, *, syntax, expander, **kw): """[syntax, expr] Like do, but return the value of the first expression.""" if syntax != "expr": - raise SyntaxError("do0 is an expr macro only") + raise SyntaxError("do0 is an expr macro only") # pragma: no cover with dyn.let(_macro_expander=expander): return _do0(tree) diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 32aced86..da5d42d9 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -130,9 +130,9 @@ def let_syntax(tree, *, args, syntax, expander, **kw): in `mcpyrate`. """ if syntax not in ("expr", "block"): - raise SyntaxError("let_syntax is an expr and block macro only") + raise SyntaxError("let_syntax is an expr and block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("let_syntax (block mode) does not take an as-part") + raise SyntaxError("let_syntax (block mode) does not take an as-part") # pragma: no cover if syntax == "expr": _let_syntax_expr_inside_out = partial(_let_syntax_expr, expand_inside=True) @@ -164,9 +164,9 @@ def abbrev(tree, *, args, syntax, expander, **kw): ``abbrev`` expands. """ if syntax not in ("expr", "block"): - raise SyntaxError("abbrev is an expr and block macro only") + raise SyntaxError("abbrev is an expr and block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("abbrev (block mode) does not take an as-part") + raise SyntaxError("abbrev (block mode) does not take an as-part") # pragma: no cover # DON'T expand inner macro invocations first - outside-in ordering is the default, so we simply do nothing. @@ -182,14 +182,14 @@ def abbrev(tree, *, args, syntax, expander, **kw): def expr(tree, *, syntax, **kw): """[syntax, block] ``with expr:`` inside a ``with let_syntax:``.""" if syntax != "block": - raise SyntaxError("`expr` is a block macro only") + raise SyntaxError("`expr` is a block macro only") # pragma: no cover raise SyntaxError("`expr` is only valid at the top level of a block-mode `let_syntax` or `abbrev`") # pragma: no cover, not intended to hit the expander @parametricmacro def block(tree, *, syntax, **kw): """[syntax, block] ``with block:`` inside a ``with let_syntax:``.""" if syntax != "block": - raise SyntaxError("`block` is a block macro only") + raise SyntaxError("`block` is a block macro only") # pragma: no cover raise SyntaxError("`block` is only valid at the top level of a block-mode `let_syntax` or `abbrev`") # pragma: no cover, not intended to hit the expander # -------------------------------------------------------------------------------- diff --git a/unpythonic/syntax/nb.py b/unpythonic/syntax/nb.py index a18f70e3..3a0e0863 100644 --- a/unpythonic/syntax/nb.py +++ b/unpythonic/syntax/nb.py @@ -37,9 +37,9 @@ def nb(tree, *, args, syntax, **kw): 3 * _ """ if syntax != "block": - raise SyntaxError("nb is a block macro only") + raise SyntaxError("nb is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("nb does not take an as-part") + raise SyntaxError("nb does not take an as-part") # pragma: no cover # Expand outside in. This macro is so simple and orthogonal the # ordering doesn't matter. This is cleaner. diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index d67765e6..8edff682 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -96,9 +96,9 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 **CAUTION**: This macro is experimental, not intended for production use. """ if syntax != "block": - raise SyntaxError("prefix is a block macro only") + raise SyntaxError("prefix is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("prefix does not take an as-part") + raise SyntaxError("prefix does not take an as-part") # pragma: no cover # Expand outside in. Any nested macros should get clean standard Python, # not having to worry about tuples possibly denoting function calls. @@ -115,14 +115,14 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 def q(tree, *, syntax, **kw): # noqa: F811 """[syntax, name] Quote operator. Only meaningful in a tuple inside a prefix block.""" if syntax != "name": - raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") + raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") # pragma: no cover raise SyntaxError("q (unpythonic.syntax.prefix.q) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander @namemacro def u(tree, *, syntax, **kw): # noqa: F811 """[syntax, name] Unquote operator. Only meaningful in a tuple inside a prefix block.""" if syntax != "name": - raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") + raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") # pragma: no cover raise SyntaxError("q (unpythonic.syntax.prefix.q) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander # TODO: This isn't a perfect solution, because there is no "call" macro kind. @@ -131,7 +131,7 @@ def u(tree, *, syntax, **kw): # noqa: F811 def kw(tree, *, syntax, **kw): # noqa: F811 """[syntax] Pass-named-args operator. Only meaningful in a tuple inside a prefix block.""" if syntax != "name": - raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is a name macro only") + raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is a name macro only") # pragma: no cover raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander # -------------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 8f132141..d1c8b6f6 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -111,9 +111,9 @@ def g(x): a statement (because in a sense, a statement always returns ``None``). """ if syntax != "block": - raise SyntaxError("autoreturn is a block macro only") + raise SyntaxError("autoreturn is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("autoreturn does not take an as-part") + raise SyntaxError("autoreturn does not take an as-part") # pragma: no cover # Expand outside in. Any nested macros should get clean standard Python, # not having to worry about implicit "return" statements. @@ -217,9 +217,9 @@ def result(ec): continuation. """ if syntax != "block": - raise SyntaxError("tco is a block macro only") + raise SyntaxError("tco is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("tco does not take an as-part") + raise SyntaxError("tco does not take an as-part") # pragma: no cover # Two-pass macro. with dyn.let(_macro_expander=expander): @@ -620,9 +620,9 @@ def result(ec): See the ``tco`` macro for details on the ``call_ec`` combo. """ if syntax != "block": - raise SyntaxError("continuations is a block macro only") + raise SyntaxError("continuations is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("continuations does not take an as-part") + raise SyntaxError("continuations does not take an as-part") # pragma: no cover # Two-pass macro. with dyn.let(_macro_expander=expander): diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 24b9407c..7e0942ba 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -239,15 +239,15 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 destroyed by the time the exception is caught by the test construct. """ if syntax not in ("expr", "block"): - raise SyntaxError("test is an expr and block macro only") + raise SyntaxError("test is an expr and block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("test (block mode) does not take an as-part") + raise SyntaxError("test (block mode) does not take an as-part") # pragma: no cover # Two-pass macros. with dyn.let(_macro_expander=expander): if syntax == "expr": if args: - raise SyntaxError("test[] in expression mode does not take macro arguments") + raise SyntaxError("test[] in expression mode does not take macro arguments") # pragma: no cover return _test_expr(tree) else: # syntax == "block": return _test_block(block_body=tree, args=args) @@ -291,15 +291,15 @@ def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 `the[]` mark is not supported. The block variant does not support `return`. """ if syntax not in ("expr", "block"): - raise SyntaxError("test_signals is an expr and block macro only") + raise SyntaxError("test_signals is an expr and block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("test_signals (block mode) does not take an as-part") + raise SyntaxError("test_signals (block mode) does not take an as-part") # pragma: no cover # Two-pass macros. with dyn.let(_macro_expander=expander): if syntax == "expr": if args: - raise SyntaxError("test_signals[] in expression mode does not take macro arguments") + raise SyntaxError("test_signals[] in expression mode does not take macro arguments") # pragma: no cover return _test_expr_signals(tree) else: # syntax == "block": return _test_block_signals(block_body=tree, args=args) @@ -342,14 +342,14 @@ def test_raises(tree, *, args, syntax, expander, **kw): # noqa: F811 `the[]` mark is not supported. The block variant does not support `return`. """ if syntax not in ("expr", "block"): - raise SyntaxError("test_raises is an expr and block macro only") + raise SyntaxError("test_raises is an expr and block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("test_raises (block mode) does not take an as-part") + raise SyntaxError("test_raises (block mode) does not take an as-part") # pragma: no cover with dyn.let(_macro_expander=expander): if syntax == "expr": if args: - raise SyntaxError("test_raises[] in expression mode does not take macro arguments") + raise SyntaxError("test_raises[] in expression mode does not take macro arguments") # pragma: no cover return _test_expr_raises(tree) else: # syntax == "block": return _test_block_raises(block_body=tree, args=args) @@ -375,7 +375,7 @@ def fail(tree, *, syntax, expander, **kw): # noqa: F811 See also `error[]`, `warn[]`. """ if syntax != "expr": - raise SyntaxError("fail is an expr macro only") + raise SyntaxError("fail is an expr macro only") # pragma: no cover # Expand outside in. The ordering shouldn't matter here. # The underlying `test` machinery needs to access the expander. @@ -395,7 +395,7 @@ def error(tree, *, syntax, expander, **kw): # noqa: F811 See also `warn[]`, `fail[]`. """ if syntax != "expr": - raise SyntaxError("error is an expr macro only") + raise SyntaxError("error is an expr macro only") # pragma: no cover # Expand outside in. The ordering shouldn't matter here. # The underlying `test` machinery needs to access the expander. @@ -419,7 +419,7 @@ def warn(tree, *, syntax, expander, **kw): # noqa: F811 See also `error[]`, `fail[]`. """ if syntax != "expr": - raise SyntaxError("warn is an expr macro only") + raise SyntaxError("warn is an expr macro only") # pragma: no cover # Expand outside in. The ordering shouldn't matter here. # The underlying `test` machinery needs to access the expander. @@ -462,9 +462,9 @@ def expand_testing_macros_first(tree, *, syntax, expander, **kw): `test[expr]` as input, but that's macros for you. Macros don't compose, after all. """ if syntax != "block": - raise SyntaxError("expand_testing_macros_first is a block macro only") + raise SyntaxError("expand_testing_macros_first is a block macro only") # pragma: no cover if syntax == "block" and kw['optional_vars'] is not None: - raise SyntaxError("expand_testing_macros_first does not take an as-part") + raise SyntaxError("expand_testing_macros_first does not take an as-part") # pragma: no cover testing_macros = [test, test_signals, test_raises, error, fail, warn] macro_bindings = extract_bindings(expander.bindings, *testing_macros) @@ -913,7 +913,7 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): exctype, tree = tree.elts message = q[None] else: - raise SyntaxError(f"Expected one of {syntaxname}[exctype, expr], {syntaxname}[exctype, expr, message]") + raise SyntaxError(f"Expected one of {syntaxname}[exctype, expr], {syntaxname}[exctype, expr, message]") # pragma: no cover # Before we edit the tree, get the source code in its pre-transformation # state, so we can include that into the test failure message. @@ -954,7 +954,7 @@ def _test_block(block_body, args): elif len(args) == 0: message = q[None] else: - raise SyntaxError('Expected `with test:` or `with test[message]:`') + raise SyntaxError('Expected `with test:` or `with test[message]:`') # pragma: no cover # Before we edit the tree, get the source code in its pre-transformation # state, so we can include that into the test failure message. @@ -1031,7 +1031,7 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): exctype = args[0] message = q[None] else: - raise SyntaxError(f'Expected `with {syntaxname}(exctype):` or `with {syntaxname}(exctype, message):`') + raise SyntaxError(f'Expected `with {syntaxname}(exctype):` or `with {syntaxname}(exctype, message):`') # pragma: no cover # Before we edit the tree, get the source code in its pre-transformation # state, so we can include that into the test failure message. From bf771992e903781ce91977634e435f34b1b6d92c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:56:39 +0300 Subject: [PATCH 152/832] fix syntax hint in error message (brackets for macro args) --- unpythonic/syntax/testingtools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 7e0942ba..afc4bd2e 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -1031,7 +1031,7 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): exctype = args[0] message = q[None] else: - raise SyntaxError(f'Expected `with {syntaxname}(exctype):` or `with {syntaxname}(exctype, message):`') # pragma: no cover + raise SyntaxError(f'Expected `with {syntaxname}(exctype):` or `with {syntaxname}[exctype, message]:`') # pragma: no cover # Before we edit the tree, get the source code in its pre-transformation # state, so we can include that into the test failure message. From d0c6883c0554d91138c29c3a966dc86a90274100 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:57:47 +0300 Subject: [PATCH 153/832] We don't emit many macro invocations; those we do should be correct. --- unpythonic/syntax/__init__.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index dca4a782..b5cb894c 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -88,12 +88,6 @@ # TODO: something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead -# TODO: `mcpyrate` does not auto-expand macros in quasiquoted code. -# - Consider when we should expand macros in quoted code and when not -# - Consider what changes this implies for other macros that read the partially expanded output -# (some things may change from expanded to unexpanded, facilitating easier analysis but requiring -# code changes) - # TODO: Consider using run-time compiler access in macro tests, like `mcpyrate` itself does. This compartmentalizes testing so that the whole test module won't crash on a macro-expansion error. # TODO: Change decorator macro invocations to use [] instead of () to pass macro arguments. Requires Python 3.9. From 2803bdfd50c164535891d0a6652ce523e7f47a2f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 16:59:16 +0300 Subject: [PATCH 154/832] Much of the cleanup has already been done. --- unpythonic/syntax/__init__.py | 9 --------- 1 file changed, 9 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index b5cb894c..21dfd30f 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -68,15 +68,6 @@ # # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# -# There are further cleanups of the macro layer possible with `mcpyrate`. For example: -# -# - Many macros could perhaps run in the outside-in pass. Some need a redesign for their AST analysis, -# but much of that has been sufficiently abstracted (e.g. `unpythonic.syntax.letdoutil`) so that this -# is mainly a case of carefully changing the analysis mode at all appropriate use sites. -# -# However, 0.15.0 is the initial version that runs on `mcpyrate`, and the focus is to just get this running. -# Cleanups can be done in a future release. # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? From 8e23ef4cb87106580c72e1a058aa1fca2bb6b6a8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 17:07:04 +0300 Subject: [PATCH 155/832] mcpyrate: drop now-unnecessary `# pragma: no cover` in quasiquoted code --- unpythonic/syntax/tests/test_letdoutil.py | 24 +++--- unpythonic/syntax/tests/test_scopeanalyzer.py | 84 +++++++++---------- unpythonic/syntax/tests/test_util.py | 62 +++++++------- 3 files changed, 85 insertions(+), 85 deletions(-) diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index 8167bf33..81d41d1a 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -59,7 +59,7 @@ def validate(lst): test[islet(the[expandrq[let[(x, 21) in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 test[islet(the[expandrq[let[2 * x, where(x, 21)]]]) == ("expanded_expr", "let")] # noqa: F821 - with expandrq as testdata: # pragma: no cover + with expandrq as testdata: @dlet((x, 21)) # noqa: F821 def f1(): return 2 * x # noqa: F821 @@ -86,7 +86,7 @@ def f1(): testdata = q[let[a in b]] # noqa: F821 test[not islet(the[testdata], expanded=False)] - with q as testdata: # pragma: no cover + with q as testdata: @dlet((x, 21)) # noqa: F821 def f2(): return 2 * x # noqa: F821 @@ -100,19 +100,19 @@ def f2(): # representation, leading to arguably funny but nonsensical things like # `ctx=currycall(ast.Load)`. with expandrq as testdata: - with autocurry: # pragma: no cover + with autocurry: let((x, 21))[2 * x] # noqa: F821 # note this goes into an ast.Expr thelet = testdata[0].value test[islet(the[thelet]) == ("curried_expr", "let")] with expandrq as testdata: - with autocurry: # pragma: no cover + with autocurry: let[(x, 21) in 2 * x] # noqa: F821 thelet = testdata[0].value test[islet(the[thelet]) == ("curried_expr", "let")] with expandrq as testdata: - with autocurry: # pragma: no cover + with autocurry: let[2 * x, where(x, 21)] # noqa: F821 thelet = testdata[0].value test[islet(the[thelet]) == ("curried_expr", "let")] @@ -124,7 +124,7 @@ def f2(): test[isdo(the[expandrq[do[x << 21, # noqa: F821 2 * x]]]) == "expanded"] # noqa: F821 - with expandrq as testdata: # pragma: no cover + with expandrq as testdata: with autocurry: do[x << 21, # noqa: F821 2 * x] # noqa: F821 @@ -219,7 +219,7 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # decorator - with q as testdata: # pragma: no cover + with q as testdata: @dlet((x, 21), (y, 2)) # noqa: F821 def f3(): return 2 * x # noqa: F821 @@ -300,7 +300,7 @@ def testexpandedletdestructuring(testdata): testexpandedletdestructuring(testdata) # decorator - with expandrq as testdata: # pragma: no cover + with expandrq as testdata: @dlet((x, 21), (y, 2)) # noqa: F821 def f4(): return 2 * x # noqa: F821 @@ -396,7 +396,7 @@ def testbindings(*expected): testexpandedletrecdestructuring(testdata) # decorator, letrec - with expandrq as testdata: # pragma: no cover + with expandrq as testdata: @dletrec((x, 21), (y, 2)) # noqa: F821 def f5(): return 2 * x # noqa: F821 @@ -413,13 +413,13 @@ def f5(): with testset("let destructuring (expanded) integration with autocurry"): with expandrq as testdata: - with autocurry: # pragma: no cover + with autocurry: let[((x, 21), (y, 2)) in y * x] # noqa: F821 # note this goes into an ast.Expr thelet = testdata[0].value testexpandedletdestructuring(thelet) with expandrq as testdata: - with autocurry: # pragma: no cover + with autocurry: letrec[((x, 21), (y, 2)) in y * x] # noqa: F821 thelet = testdata[0].value testexpandedletrecdestructuring(thelet) @@ -494,7 +494,7 @@ def f5(): "not an expanded do form"] with testset("do destructuring (expanded) integration with autocurry"): - with expandrq as testdata: # pragma: no cover + with expandrq as testdata: with autocurry: do[local[x << 21], # noqa: F821 2 * x] # noqa: F821 diff --git a/unpythonic/syntax/tests/test_scopeanalyzer.py b/unpythonic/syntax/tests/test_scopeanalyzer.py index ff5e45e4..35ea18e9 100644 --- a/unpythonic/syntax/tests/test_scopeanalyzer.py +++ b/unpythonic/syntax/tests/test_scopeanalyzer.py @@ -17,11 +17,11 @@ def runtests(): # test data with q as getnames_load: - x # noqa: F821, it's only quoted. # pragma: no cover + x # noqa: F821, it's only quoted. with q as getnames_del: - del x # noqa: F821 # pragma: no cover + del x # noqa: F821 with q as getnames_store_simple: - x = 42 # noqa: F841, it's only quoted. # pragma: no cover + x = 42 # noqa: F841, it's only quoted. with testset("isnewscope"): test[not isnewscope(q[x])] # noqa: F821, it's only quoted. @@ -29,15 +29,15 @@ def runtests(): test[not isnewscope(q[d['x']])] # noqa: F821 test[isnewscope(q[lambda x: 2 * x])] with q as fdef: - def f(): # pragma: no cover + def f(): pass test[isnewscope(fdef[0])] with q as afdef: # Python 3.5+ - async def g(): # pragma: no cover + async def g(): pass test[isnewscope(afdef[0])] with q as cdef: - class Cat: # pragma: no cover + class Cat: has_four_legs = True def sleep(): pass @@ -64,55 +64,55 @@ def sleep(): # assignment it is. test[get_names_in_store_context(getnames_store_simple) == ["x"]] with q as getnames_tuple: - x, y = 1, 2 # noqa: F841 # pragma: no cover + x, y = 1, 2 # noqa: F841 test[get_names_in_store_context(getnames_tuple) == ["x", "y"]] with q as getnames_starredtuple: - x, y, *rest = range(5) # noqa: F841 # pragma: no cover + x, y, *rest = range(5) # noqa: F841 test[get_names_in_store_context(getnames_starredtuple) == ["x", "y", "rest"]] # Function name, async function name, class name with q as getnames_func: - def f1(): # pragma: no cover + def f1(): pass test[get_names_in_store_context(getnames_func) == ["f1"]] with q as getnames_afunc: # Python 3.5+ - async def f2(): # pragma: no cover + async def f2(): pass test[get_names_in_store_context(getnames_afunc) == ["f2"]] with q as getnames_class: - class Classy: # pragma: no cover + class Classy: pass test[get_names_in_store_context(getnames_class) == ["Classy"]] # For loop target with q as getnames_for_simple: - for x in range(5): # pragma: no cover + for x in range(5): pass test[get_names_in_store_context(getnames_for_simple) == ["x"]] with q as getnames_for_tuple: - for x, y in zip(range(5), range(5)): # pragma: no cover + for x, y in zip(range(5), range(5)): pass test[get_names_in_store_context(getnames_for_tuple) == ["x", "y"]] with q as getnames_for_mixed: - for j, (x, y) in enumerate(zip(range(5), range(5))): # pragma: no cover + for j, (x, y) in enumerate(zip(range(5), range(5))): pass test[get_names_in_store_context(getnames_for_mixed) == ["j", "x", "y"]] # Async for loop target (Python 3.5+) with q as getnames_afor_simple: - async def g1(): # pragma: no cover + async def g1(): async for x in range(5): pass test[get_names_in_store_context(getnames_afor_simple) == ["g1"]] # we stop at scope boundaries test[get_names_in_store_context(getnames_afor_simple[0].body) == ["x"]] with q as getnames_afor_tuple: - async def g2(): # pragma: no cover + async def g2(): async for x, y in zip(range(5), range(5)): pass test[get_names_in_store_context(getnames_afor_tuple) == ["g2"]] test[get_names_in_store_context(getnames_afor_tuple[0].body) == ["x", "y"]] with q as getnames_afor_mixed: - async def g3(): # pragma: no cover + async def g3(): async for j, (x, y) in enumerate(zip(range(5), range(5))): pass test[get_names_in_store_context(getnames_afor_mixed) == ["g3"]] @@ -120,30 +120,30 @@ async def g3(): # pragma: no cover # Import statement with q as getnames_import: - import mymod # noqa: F401 # pragma: no cover - import yourmod as renamedmod # noqa: F401 # pragma: no cover - from othermod import original as renamed, other # noqa: F401 # pragma: no cover + import mymod # noqa: F401 + import yourmod as renamedmod # noqa: F401 + from othermod import original as renamed, other # noqa: F401 test[get_names_in_store_context(getnames_import) == ["mymod", "renamedmod", "renamed", "other"]] # Except clause target in try statement with q as getnames_try: - try: # pragma: no cover + try: pass - except Exception as err: # noqa: F841 # pragma: no cover + except Exception as err: # noqa: F841 pass - except KeyboardInterrupt as kbi: # noqa: F841 # pragma: no cover + except KeyboardInterrupt as kbi: # noqa: F841 pass test[get_names_in_store_context(getnames_try) == ["err", "kbi"]] # With statement target with q as getnames_with: - with Manager() as boss: # noqa: F821, F841 # pragma: no cover + with Manager() as boss: # noqa: F821, F841 pass test[get_names_in_store_context(getnames_with) == ["boss"]] # Async with statement target (Python 3.5+) with q as getnames_awith: - async def g4(): # pragma: no cover + async def g4(): async with Manager() as boss: # noqa: F821, F841 pass test[get_names_in_store_context(getnames_awith) == ["g4"]] @@ -159,24 +159,24 @@ async def g4(): # pragma: no cover # We ignore `del o.x` and `del d['x']`, because these # don't delete the lexical variables `o` and `d`. with q as getnames_del_attrib: - del o.x # noqa: F821, F841 # pragma: no cover + del o.x # noqa: F821, F841 test[get_names_in_del_context(getnames_del_attrib) == []] with q as getnames_del_subscript: - del d["x"] # noqa: F821, F841 # pragma: no cover + del d["x"] # noqa: F821, F841 test[get_names_in_del_context(getnames_del_subscript) == []] with q as getnames_del_scope_boundary: - del x # noqa: F821 # pragma: no cover - def f3(): # pragma: no cover + del x # noqa: F821 + def f3(): del y # noqa: F821 test[get_names_in_del_context(getnames_del_scope_boundary) == ["x"]] test[get_names_in_del_context(getnames_del_scope_boundary[1].body) == ["y"]] with testset("get_lexical_variables"): with q as getlexvars_fdef: - y = 21 # pragma: no cover - def myfunc(x, *args, kwonlyarg, **kwargs): # pragma: no cover + y = 21 + def myfunc(x, *args, kwonlyarg, **kwargs): nonlocal y # not really needed here, except for exercising the analyzer. global g def inner(blah): @@ -196,21 +196,21 @@ def inner(blah): ["y", "g"])] with q as getlexvars_classdef: - class WorldClassy(Classy): # pragma: no cover + class WorldClassy(Classy): pass test[get_lexical_variables(getlexvars_classdef[0]) == (["WorldClassy", "Classy"], [])] with q as getlexvars_listcomp_simple: - [x for x in range(5)] # note this goes into an ast.Expr # pragma: no cover + [x for x in range(5)] # note this goes into an ast.Expr test[get_lexical_variables(getlexvars_listcomp_simple[0].value) == (["x"], [])] with q as getlexvars_listcomp_tuple_in_expr: - [(x, y) for x in range(5) for y in range(x)] # pragma: no cover + [(x, y) for x in range(5) for y in range(x)] test[get_lexical_variables(getlexvars_listcomp_tuple_in_expr[0].value) == (["x", "y"], [])] with q as getlexvars_listcomp_tuple_in_target: - [(x, y) for x, y in zip(range(5), range(5))] # pragma: no cover + [(x, y) for x, y in zip(range(5), range(5))] test[get_lexical_variables(getlexvars_listcomp_tuple_in_target[0].value) == (["x", "y"], [])] @@ -227,44 +227,44 @@ def check(tree, actual_names): return check with q as scoped_onefunc: - def f(x): # noqa: F811 # pragma: no cover + def f(x): # noqa: F811 n["_apply_test_here_"] scoped_transform(scoped_onefunc, callback=make_checker(["f", "x"])) with q as scoped_nestedfunc1: - def f(x): # noqa: F811 # pragma: no cover + def f(x): # noqa: F811 n["_apply_test_here_"] def g(y): pass scoped_transform(scoped_nestedfunc1, callback=make_checker(["f", "x"])) with q as scoped_nestedfunc2: - def f(x): # noqa: F811 # pragma: no cover + def f(x): # noqa: F811 def g(y): n["_apply_test_here_"] scoped_transform(scoped_nestedfunc2, callback=make_checker(["f", "x", "g", "y"])) with q as scoped_classdef: - class WorldClassy(Classy): # noqa: F811 # pragma: no cover + class WorldClassy(Classy): # noqa: F811 n["_apply_test_here_"] scoped_transform(scoped_classdef, callback=make_checker(["WorldClassy", "Classy"])) with q as scoped_localvar1: - def f(): # noqa: F811 # pragma: no cover + def f(): # noqa: F811 x = 42 # noqa: F841 n["_apply_test_here_"] scoped_transform(scoped_localvar1, callback=make_checker(["f", "x"])) # TODO: In 0.15.x, fully lexical scope analysis; update this test at that time. with q as scoped_localvar2: - def f(): # noqa: F811 # pragma: no cover + def f(): # noqa: F811 n["_apply_test_here_"] x = 42 # noqa: F841 scoped_transform(scoped_localvar2, callback=make_checker(["f"])) # x not yet created # TODO: In 0.15.x, fully lexical scope analysis; update this test at that time. with q as scoped_localvar3: - def f(): # noqa: F811 # pragma: no cover + def f(): # noqa: F811 x = 42 # noqa: F841 del x n["_apply_test_here_"] diff --git a/unpythonic/syntax/tests/test_util.py b/unpythonic/syntax/tests/test_util.py index a80c46fe..18ad316c 100644 --- a/unpythonic/syntax/tests/test_util.py +++ b/unpythonic/syntax/tests/test_util.py @@ -36,17 +36,17 @@ def runtests(): test["my_fancy_ec" in the[detect_callec(q[call_ec(lambda my_fancy_ec: None)])]] with q as call_ec_testdata: - @call_ec # pragma: no cover + @call_ec def f(my_fancy_ec): - pass # pragma: no cover + pass test["my_fancy_ec" in the[detect_callec(call_ec_testdata)]] with testset("detect_lambda"): # We expand the `do[]` to generate an implicit lambda that should be ignored by the detector # (it specifically checks for expanded `do[]` forms). with expandrq as detect_lambda_testdata: - a = lambda: None # noqa: F841 # pragma: no cover - b = do[local[x << 21], # noqa: F821, F841 # pragma: no cover + a = lambda: None # noqa: F841 + b = do[local[x << 21], # noqa: F821, F841 lambda y: x * y] # noqa: F821 test[len(detect_lambda(detect_lambda_testdata)) == 2] @@ -55,12 +55,12 @@ def f(my_fancy_ec): test[is_decorator(q[decorate_with("flowers")], "decorate_with")] # noqa: F821 with q as has_tco_testdata1: - @trampolined # noqa: F821, just quoted. # pragma: no cover + @trampolined # noqa: F821, just quoted. def ihavetco(): - pass # pragma: no cover + pass with q as has_tco_testdata2: - def idonthavetco(): # pragma: no cover - pass # pragma: no cover + def idonthavetco(): + pass test[has_tco(has_tco_testdata1[0])] test[not has_tco(has_tco_testdata2[0])] test[not has_tco(q[lambda: None])] @@ -70,12 +70,12 @@ def idonthavetco(): # pragma: no cover test[has_tco(q[trampolined(decorate(lambda: None))])] # noqa: F821 with q as has_curry_testdata1: - @curry # noqa: F821, just quoted. # pragma: no cover + @curry # noqa: F821, just quoted. def ihavecurry(): - pass # pragma: no cover + pass with q as has_curry_testdata2: - def idonthavecurry(): # pragma: no cover - pass # pragma: no cover + def idonthavecurry(): + pass test[has_curry(has_curry_testdata1[0])] test[not has_curry(has_curry_testdata2[0])] test[not has_curry(q[lambda: None])] @@ -87,12 +87,12 @@ def idonthavecurry(): # pragma: no cover test[has_deco(["decorate"], q["surprise!"]) is None] # wrong AST type, test not applicable with q as has_deco_testdata1: - @artdeco # noqa: F821, just quoted. # pragma: no cover + @artdeco # noqa: F821, just quoted. def ihaveartdeco(): - pass # pragma: no cover + pass with q as has_deco_testdata2: - def idonthaveartdeco(): # pragma: no cover - pass # pragma: no cover + def idonthaveartdeco(): + pass test[has_deco(["artdeco"], has_deco_testdata1[0])] test[not has_deco(["artdeco"], has_deco_testdata2[0])] test[not has_deco(["artdeco"], q[lambda: None])] @@ -114,9 +114,9 @@ def idonthaveartdeco(): # pragma: no cover with q as sdi_testdata1: # This set of decorators makes no sense, but we want to exercise # the different branches of the analysis code. - @curry # noqa: F821 # pragma: no cover - @memoize # noqa: F821 # pragma: no cover - @call # noqa: F821 # pragma: no cover + @curry # noqa: F821 + @memoize # noqa: F821 + @call # noqa: F821 def purespicy(a, b, c): pass # pragma: no cover test[suggest_decorator_index("artdeco", sdi_testdata1[0].decorator_list) is None] # unknown decorator @@ -125,10 +125,10 @@ def purespicy(a, b, c): test[suggest_decorator_index("passthrough_lazy_args", sdi_testdata1[0].decorator_list) == 3] # after all of those already specified with q as sdi_testdata2: - @artdeco # noqa: F821 # pragma: no cover - @neoclassical # noqa: F821 # pragma: no cover + @artdeco # noqa: F821 + @neoclassical # noqa: F821 def architectural(): - pass # pragma: no cover + pass test[suggest_decorator_index("trampolined", sdi_testdata2[0].decorator_list) is None] # known decorator, but only unknown decorators in the decorator_list with testset("decorated lambda machinery"): @@ -172,15 +172,15 @@ def test_sort_lambda_decorators(testdata): with testset("statement utilities"): with q as transform_statements_testdata: - def myfunction(x): # pragma: no cover + def myfunction(x): "function body" - try: # pragma: no cover + try: "try" - if x: # pragma: no cover + if x: "if body" else: "if else" - except ValueError: # pragma: no cover + except ValueError: "except" finally: "finally" @@ -204,7 +204,7 @@ def ishello(tree): test[len(result) == 1 and ishello(result[0])] with q as eliminate_ifones_testdata2: - if 0: # pragma: no cover + if 0: "hello" result = eliminate_ifones(eliminate_ifones_testdata2) test[len(result) == 0] @@ -217,7 +217,7 @@ def ishello(tree): test[len(result) == 1 and ishello(result[0])] with q as eliminate_ifones_testdata4: - if False: # pragma: no cover + if False: "hello" result = eliminate_ifones(eliminate_ifones_testdata4) test[len(result) == 0] @@ -258,7 +258,7 @@ def ishello(tree): with testset("wrapwith"): with q as wrapwith_testdata: - 42 # pragma: no cover + 42 wrapped = wrapwith(q[n["ExampleContextManager"]], wrapwith_testdata) test[type(wrapped) is list] thewith = wrapped[0] @@ -274,10 +274,10 @@ def ishello(tree): with testset("isexpandedmacromarker"): with q as ismarker_testdata1: - with ExampleMarker: # noqa: F821 # pragma: no cover + with ExampleMarker: # noqa: F821 ... with q as ismarker_testdata2: - with NotAMarker1, NotAMarker2: # noqa: F821 # pragma: no cover + with NotAMarker1, NotAMarker2: # noqa: F821 ... test[isexpandedmacromarker("ExampleMarker", ismarker_testdata1[0])] test[not isexpandedmacromarker("AnotherMarker", ismarker_testdata1[0])] # right AST node type, different marker From b79a9a6cc13d1e7f76106241a95de660c4de2522 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 17:08:29 +0300 Subject: [PATCH 156/832] add `# pragma: no cover` to appropriate branches in astcompat --- unpythonic/syntax/astcompat.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/astcompat.py b/unpythonic/syntax/astcompat.py index c1eb4653..4dd444e7 100644 --- a/unpythonic/syntax/astcompat.py +++ b/unpythonic/syntax/astcompat.py @@ -58,12 +58,12 @@ def getconstant(tree): if type(tree) is ast.Constant: # Python 3.8+ return tree.value # up to Python 3.7 - elif type(tree) is ast.NameConstant: # up to Python 3.7 + elif type(tree) is ast.NameConstant: # up to Python 3.7 # pragma: no cover return tree.value - elif type(tree) is ast.Num: + elif type(tree) is ast.Num: # pragma: no cover return tree.n - elif type(tree) in (ast.Str, ast.Bytes): + elif type(tree) in (ast.Str, ast.Bytes): # pragma: no cover return tree.s - elif type(tree) is ast.Ellipsis: # `ast.Ellipsis` is the AST node type, `builtins.Ellipsis` is `...`. + elif type(tree) is ast.Ellipsis: # `ast.Ellipsis` is the AST node type, `builtins.Ellipsis` is `...`. # pragma: no cover return ... - raise TypeError(f"Not an AST node representing a constant: {type(tree)} with value {repr(tree)}") + raise TypeError(f"Not an AST node representing a constant: {type(tree)} with value {repr(tree)}") # pragma: no cover From cfa56e9d3ccbd45423be211444bdb44db521546e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 4 May 2021 17:09:12 +0300 Subject: [PATCH 157/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 21dfd30f..52459065 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -71,8 +71,6 @@ # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? -# TODO: Drop `# pragma: no cover` from macro tests as appropriate, since `mcpyrate` reports coverage correctly. - # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. # TODO: macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" From e22639a18604727915f2e1e694d9728ac4d57a2a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:17:50 +0300 Subject: [PATCH 158/832] improve docstring --- unpythonic/syntax/letdoutil.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 9aa5be50..b442ec4c 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -262,6 +262,11 @@ class UnexpandedEnvAssignView: This handles `mcpyrate.core.Done` `ASTMarker`s in the name position transparently, to accommodate for expanded `mcpyrate.namemacro`s. + In other words, if the AST for the LHS is `Name(id=...)`, reading/writing the `name` + property will access `lhs.id`. If the AST for the LHS is `Done(body=Name(id=...))`, + reading/writing the `name` property will access `lhs.body.id`. This means you don't + need to care about whether there is a `Done` or not. + **Attributes**: ``name``: the name of the variable, as a str. From b97a0886274678155f130d6c4d3dfbe36fac17b9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:18:00 +0300 Subject: [PATCH 159/832] improve comment --- unpythonic/syntax/testingtools.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index afc4bd2e..5740c12c 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -842,7 +842,10 @@ def _test_expr(tree): # Also, we need the lambda for passing in the value capture environment # for the `the[]` mark, anyway. # - # We name it `testexpr` to make the stack trace more understandable. + # We can't inject `lazy[]` here (to be more explicit this is a delay operation), + # because we need to pass the environment. + # + # We name the lambda `testexpr` to make the stack trace more understandable. # If you change the name, change it also in `unpythonic_assert`. thelambda = q[lambda _: a[tree]] thelambda.args.args[0] = arg(arg=envname) # inject the gensymmed parameter name From 1648d31566f79f0fb469f52522fc22c821264baa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:18:18 +0300 Subject: [PATCH 160/832] 0.15: remove deprecated "getvalue", "runpipe" Please use "exitpipe" instead; it performs both roles. --- unpythonic/seq.py | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 3400fc4f..1d325b7a 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -3,7 +3,7 @@ __all__ = ["begin", "begin0", "lazy_begin", "lazy_begin0", "pipe1", "piped1", "lazy_piped1", - "pipe", "piped", "getvalue", "lazy_piped", "runpipe", "exitpipe", + "pipe", "piped", "lazy_piped", "exitpipe", "pipec", # w/ curry "do", "do0", "assign"] @@ -12,7 +12,7 @@ from .fun import curry, iscurried from .dynassign import dyn from .arity import arity_includes, UnknownArity -from .singleton import Singleton +from .symbol import sym # sequence side effects in a lambda def begin(*vals): @@ -143,16 +143,7 @@ def pipe1(value0, *bodys): x = update(x) return x -class Getvalue(Singleton): # singleton sentinel with a nice repr - """Sentinel; pipe into this to exit a shell-like pipe and return the current value.""" - def __repr__(self): # pragma: no cover - return "" -getvalue = Getvalue() -runpipe = getvalue # same thing as getvalue, but semantically better name for lazy pipes - -# New unified name for v0.15.0; deprecating the separate "getvalue" and "runpipe" as of v0.14.2. -# TODO: Now that we have symbols, in v0.15.0, change this to `sym("exitpipe")` and delete the `Getvalue` class. -exitpipe = getvalue +exitpipe = sym("exitpipe") class piped1: """Shell-like piping syntax. From f80406394cd862cb4c4bab3e4faa650f00512bba Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:19:14 +0300 Subject: [PATCH 161/832] 0.15: remove deprecated "setescape", "escape" Please invoke them by the lispy standard names "catch" and "throw" instead. --- unpythonic/ec.py | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/unpythonic/ec.py b/unpythonic/ec.py index 466d65de..c612aa31 100644 --- a/unpythonic/ec.py +++ b/unpythonic/ec.py @@ -24,31 +24,13 @@ http://www.gigamonkeys.com/book/the-special-operators.html """ -__all__ = ["throw", "catch", "call_ec", - "setescape", "escape"] # old names, pre-0.14.2, will go away in 0.15.0 +__all__ = ["throw", "catch", "call_ec"] -from warnings import warn from functools import wraps from .regutil import register_decorator # from .symbol import gensym -def escape(value, tag=None, allow_catchall=True): # pragma: no cover - """Alias for `throw`, for backward compatibility. - - Will be removed in 0.15.0. - """ - warn("`escape` has been renamed `throw` as in Common Lisp; this alias will be removed in 0.15.0.", FutureWarning) - return throw(value, tag, allow_catchall) - -def setescape(tags=None, catch_untagged=True): # pragma: no cover - """Alias for `catch`, for backward compatibility. - - Will be removed in 0.15.0. - """ - warn("`setescape` has been renamed `catch` as in Common Lisp; this alias will be removed in 0.15.0.", FutureWarning) - return catch(tags, catch_untagged) - def throw(value, tag=None, allow_catchall=True): """Escape to a dynamically surrounding ``@catch``. From 8339ae3f492f4f36eb80948a6075bb45e6ad7128 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:20:09 +0300 Subject: [PATCH 162/832] 0.15: drop support for old `raisef` call signature Please use as `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. Resolves #30. --- unpythonic/misc.py | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/unpythonic/misc.py b/unpythonic/misc.py index a10d1689..abe07183 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -214,7 +214,7 @@ def applyfrozenargsto(f): return maybe_force_args(force(f), *args, **kwargs) return applyfrozenargsto -def raisef(exc, *args, cause=None, **kwargs): +def raisef(exc, *, cause=None): """``raise`` as a function, to make it possible for lambdas to raise exceptions. Example:: @@ -235,23 +235,7 @@ def raisef(exc, *args, cause=None, **kwargs): If `exc` was triggered as a direct consequence of another exception, and you would like to `raise ... from ...`, pass that other exception instance as `cause`. The default `None` performs a plain `raise ...`. - - *Changed in v0.14.2.* The parameters have changed to match `raise` itself as closely - as possible. Old-style parameters are still supported, but are now deprecated. Support - for them will be dropped in v0.15.0. The old-style parameters are: - - exc: type - The object type to raise as an exception. - - *args: anything - Passed on to the constructor of exc. - - **kwargs: anything - Passed on to the constructor of exc. """ - if args or kwargs: # old-style parameters - raise exc(*args, **kwargs) - if cause: raise exc from cause else: From 1ebdfd2eab95c1e8568901d8fa54ff632ef99a8f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:21:42 +0300 Subject: [PATCH 163/832] update changelog --- CHANGELOG.md | 42 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 34 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4200e36..a6eab608 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,32 +2,51 @@ This edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. -**Minimum Python language version is now 3.6**. For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). +**Minimum Python language version is now 3.6**. + +The optional **macro expander is now [`mcpyrate`](https://github.com/Technologicat/mcpyrate)**. + +If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. + +The same applies if you need the macro parts of `unpythonic` in your own project that uses MacroPy. Version 0.14.3 of `unpythonic` works up to Python 3.7. + +For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). + **New**: - **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) together with a kitchen-sink language extension macro package such as `unpythonic`. + - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) + - Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), `with expr`/`with block` (for `let_syntax`/`abbrev`), and `q`/`u`/`kw` (for `prefix`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. + - `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. + - Python 3.8 and 3.9 support added. + **Non-breaking changes**: - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. -- CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). -- CI: Test coverage improved to 94%. + - Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. The `continuations` macro also outputs a hygienically captured `aif` when transforming an `or` expression that occurs in tail position. - This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) - The implicit do (extra bracket syntax) also expands as a hygienically captured `do`, but e.g. in `let[]` it will then expand immediately (due to `let`'s inside-out expansion order) before control returns to the macro stepper. If you want to see the implicit `do[]` invocation, use the `"detailed"` mode of the stepper, which shows individual macro invocations even when expanding inside-out: `step_expansion["detailed"][...]`, `with step_expansion["detailed"]:`. +- CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). + +- CI: Test coverage improved to 94%. + + **Breaking changes**: -- Migrate to the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander; **MacroPy support dropped**. This change facilitates future development of the macro parts of `unpythonic`. +- Migrate to the `mcpyrate` macro expander; **MacroPy support dropped**. - **Macro arguments are now passed using brackets** `macroname[args]` instead of parentheses. - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. - - As a result of the new macro expander, macro test coverage should now be reported correctly. + - `mcpyrate` should report test coverage for macro-using code correctly; no need for `# pragma: no cover` in block macro invocations or in quasiquoted code. + - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **Any imports of these constructs in user code should be modified to point to the new locations.** - Unlike `macropy`'s `Lazy`, our `Lazy` does not define `__call__`; instead, it defines the method `force`, which has the same effect (it computes if necessary, and then returns the value of the promise). @@ -35,20 +54,27 @@ This edition concentrates on upgrading our dependencies, namely the macro expand - `f[]` now respects nesting: an invocation of `f[]` will not descend into another nested `f[]`. - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import. Now you **must** import also the macro `f` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `f` is currently bound to. + - **Rename the `curry` macro** to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. + - The `do[]` and `do0[]` macros now expand outside-in. The main differences from a user perspective are: - Any source code captures (such as those performed by `test[]`) show the expanded output of `do` and `do0`, because that's what they receive. - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. + - Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. -- Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. + +- Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. + **Fixed**: -- Make `callsite_filename` ignore our call helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`). -- In `aif`, `it` is now only valid in the `then` and `otherwise` parts, as it should. +- Make `unpythonic.misc.callsite_filename` ignore our call helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`). + +- In `aif`, `it` is now only valid in the `then` and `otherwise` parts, as it should always have been. --- From 4b655c61dda4141f74391b83dd14eba902aea1c5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:21:58 +0300 Subject: [PATCH 164/832] update lazify docs: document lazy[], lazyrec[] properly --- doc/macros.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index f5b6e59f..f567aff8 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -34,6 +34,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose [**Language features**](#language-features) - [``curry``: automatic currying for Python](#curry-automatic-currying-for-python) - [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) + - [``lazy[]`` and ``lazyrec[]`` macros](#lazy-and-lazyrec-macros) - [Forcing promises manually](#forcing-promises-manually) - [Binding constructs and auto-lazification](#binding-constructs-and-auto-lazification) - [Note about TCO](#note-about-tco) @@ -781,11 +782,11 @@ with lazify: assert f(21, 1/0) == 42 ``` -In a ``with lazify`` block, function arguments are evaluated only when actually used, at most once each, and in the order in which they are actually used. Promises are automatically forced on access. Automatic lazification applies to arguments in function calls and to let-bindings, since they play a similar role. **No other binding forms are auto-lazified.** +In a ``with lazify`` block, function arguments are evaluated only when actually used, at most once each, and in the order in which they are actually used (regardless of the ordering of the formal parameters that receive them). Delayed values (*promises*) are automatically evaluated (*forced*) on access. Automatic lazification applies to arguments in function calls and to let-bindings, since they play a similar role. **No other binding forms are auto-lazified.** Automatic lazification uses the ``lazyrec[]`` macro (see below), which recurses into certain types of container literals, so that the lazification will not interfere with unpacking. -Note ``my_if`` in the example is a run-of-the-mill runtime function, not a macro. Only the ``with lazify`` is imbued with any magic. Essentially, the above code expands into: +Note ``my_if`` in the example is a regular function, not a macro. Only the ``with lazify`` is imbued with any magic. Essentially, the above code expands into: ```python from unpythonic.syntax import macros, lazy @@ -806,7 +807,7 @@ def f(a, b): assert f(lazy[21], lazy[1/0]) == 42 ``` -plus some clerical details to allow mixing lazy and strict code. This second example relies on the magic of closures to capture f's ``a`` and ``b`` into the promises. +plus some clerical details to allow mixing lazy and strict code. This second example relies on the magic of closures to capture f's ``a`` and ``b`` into the ``lazy[]`` promises. Like ``with continuations``, no state or context is associated with a ``with lazify`` block, so lazy functions defined in one block may call those defined in another. @@ -816,14 +817,26 @@ Comboing with other block macros in ``unpythonic.syntax`` is supported, includin For more details, see the docstring of ``unpythonic.syntax.lazify``. -See also ``unpythonic.syntax.lazy``, which explicitly lazifies a single expression, and ``unpythonic.syntax.lazyrec``, which can be used to lazify expressions inside container literals, recursively. This allows code like ``tpl = lazyrec[(1*2*3, 4*5*6)]``. Each item becomes wrapped with ``lazy[]``, but the container itself is left alone, to avoid interfering with unpacking. Because ``lazyrec[]`` is a macro and must work by names only, it supports a fixed set of container types: ``list``, ``tuple``, ``set``, ``dict``, ``frozenset``, ``unpythonic.collections.frozendict``, ``unpythonic.collections.box``, and ``unpythonic.llist.cons`` (specifically, the constructors ``cons``, ``ll`` and ``llist``). - -(It must work by names only, because in an eager language any lazification must be performed as a syntax transformation before the code actually runs. Lazification in an eager language is a hack, by necessity. [Fexprs](https://fexpr.blogspot.com/2011/04/fexpr.html) (along with [a new calculus to go with them](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html)) are the clean, elegant solution, but this requires redesigning the whole language from ground up. Of course, if you're fine with a language not particularly designed for extensibility, and lazy evaluation is your top requirement, just use Haskell.) - Inspired by Haskell, Racket's ``(delay)`` and ``(force)``, and [lazy/racket](https://docs.racket-lang.org/lazy/index.html). **CAUTION**: The functions in ``unpythonic.fun`` are lazify-aware (so that e.g. ``curry`` and ``compose`` work with lazy functions), as are ``call`` and ``callwith`` in ``unpythonic.misc``, but a large part of ``unpythonic`` is not. Keep in mind that any call to a strict (regular Python) function will evaluate all of its arguments. +#### ``lazy[]`` and ``lazyrec[]`` macros + +We provide the macros ``unpythonic.syntax.lazy``, which explicitly lazifies a single expression, and ``unpythonic.syntax.lazyrec``, which can be used to lazify expressions inside container literals, recursively. + +Essentially, ``lazy[...]`` achieves the same result as ``memoize(lambda: ...)``, with the practical difference that a ``lazy[]`` promise ``p`` is evaluated by calling ``unpythonic.lazyutil.force(p)`` or ``p.force()``. In ``unpythonic``, the promise datatype (``unpythonic.lazyutil.Lazy``) does not have a ``__call__`` method, because the word ``force`` better conveys the intent. + +It is preferable to use the ``force`` function instead of the ``.force`` method, because the function will also pass through any non-promise value, whereas (obviously) a non-promise value will not have a ``.force`` method. Using the function, you can ``force`` a value just to be sure, without caring whether that value was a promise. The ``force`` function is available in the top-level namespace of ``unpythonic``. + +The ``lazify`` subsystem expects the ``lazy[...]`` notation in its analyzer, and will not recognize ``memoize(lambda: ...)`` as a delayed value. + +The ``lazyrec[]`` macro allows code like ``tpl = lazyrec[(1*2*3, 4*5*6)]``. Each item becomes wrapped with ``lazy[]``, but the container itself is left alone, to avoid interfering with unpacking. Because ``lazyrec[]`` is a macro and must work by names only, it supports a fixed set of container types: ``list``, ``tuple``, ``set``, ``dict``, ``frozenset``, ``unpythonic.collections.frozendict``, ``unpythonic.collections.box``, and ``unpythonic.llist.cons`` (specifically, the constructors ``cons``, ``ll`` and ``llist``). + +The `unpythonic` containers **must be from-imported** for ``lazyrec[]`` to recognize them. Either use ``from unpythonic import xxx`` (**recommended**), where ``xxx`` is a container type, or import the ``containers`` subpackage by ``from unpythonic import containers``, and then use ``containers.xxx``. (The analyzer only looks inside at most one level of attributes. This may change in the future.) + +(The analysis in ``lazyrec[]`` must work by names only, because in an eager language any lazification must be performed as a syntax transformation before the code actually runs, so the analysis must be performed statically - and locally, because ``lazyrec[]`` is an expr macro. [Fexprs](https://fexpr.blogspot.com/2011/04/fexpr.html) (along with [a new calculus to go with them](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html)) are the clean, elegant solution, but this requires redesigning the whole language from ground up. Of course, if you're fine with a language not particularly designed for extensibility, and lazy evaluation is your top requirement, you could just use Haskell.) + #### Forcing promises manually This is mainly useful if you ``lazy[]`` or ``lazyrec[]`` something explicitly, and want to compute its value outside a ``with lazify`` block. From d191fbea639c84b53c78c233580cfab85f1108eb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:23:19 +0300 Subject: [PATCH 165/832] naming: @generic_addmethod more descriptive than @generic_for Avoid semantically loaded word, it has nothing to do with `for` loops. --- CHANGELOG.md | 2 +- doc/features.md | 2 +- unpythonic/dispatch.py | 10 +++++----- unpythonic/tests/test_dispatch.py | 22 +++++++++++----------- 4 files changed, 18 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a6eab608..d9ab5e52 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,7 @@ For future plans, see our [Python language version support status](https://githu - Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), `with expr`/`with block` (for `let_syntax`/`abbrev`), and `q`/`u`/`kw` (for `prefix`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. -- `unpythonic.dispatch.generic_for`: add methods to a generic function defined elsewhere. +- `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. - Python 3.8 and 3.9 support added. diff --git a/doc/features.md b/doc/features.md index efb74423..a5ca5afb 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2986,7 +2986,7 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of **Changed in v0.14.3**. *The `@generic` and `@typed` decorators can now decorate also instance methods, class methods and static methods (beside regular functions, as previously in 0.14.2).* -**Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Added the `@generic_for` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope.* +**Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Added the `@generic_addmethod` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope.* The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 1c8d4f46..c6b8c060 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -6,7 +6,7 @@ https://docs.python.org/3/library/functools.html#functools.singledispatch """ -__all__ = ["generic", "generic_for", "typed"] +__all__ = ["generic", "generic_addmethod", "typed"] from functools import partial, wraps from itertools import chain @@ -37,7 +37,7 @@ # """ @register_decorator(priority=98) -def generic_for(target): +def generic_addmethod(target): """Parametric decorator. Add a method to function `target`. Like `@generic`, but the target function on which the method will be @@ -55,10 +55,10 @@ def f(x: int): # main.py - from unpythonic import generic_for + from unpythonic import generic_addmethod import example - @generic_for(example.f) + @generic_addmethod(example.f) def f(x: float): ... """ @@ -212,7 +212,7 @@ def _getfullname(f): def _register_generic(fullname, f): """Register a method for a generic function. - This is a low-level function; you'll likely want `generic` or `generic_for`. + This is a low-level function; you'll likely want `generic` or `generic_addmethod`. fullname: str, fully qualified name of target function to register the method on, used as key in the dispatcher registry. diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index a285ba00..7bcdc40a 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -5,7 +5,7 @@ import typing from ..fun import curry -from ..dispatch import generic, generic_for, typed +from ..dispatch import generic, generic_addmethod, typed @generic def zorblify(x: int, y: int): @@ -91,11 +91,11 @@ def runtests(): test[gargle(42, 6.022e23, "hello") == "int, float, str"] test[gargle(1, 2, 3) == "int"] # as many as in the [int, float, str] case - with testset("@generic_for"): + with testset("@generic_addmethod"): @generic def f1(x: typing.Any): return False - @generic_for(f1) + @generic_addmethod(f1) def f2(x: int): return x test[f1("hello") is False] @@ -103,8 +103,8 @@ def f2(x: int): def f3(x: typing.Any): # not @generic! return False - with test_raises[TypeError, "should not be able to @generic_for a non-generic function"]: - @generic_for(f3) + with test_raises[TypeError, "should not be able to @generic_addmethod a non-generic function"]: + @generic_addmethod(f3) def f4(x: int): return x @@ -269,12 +269,12 @@ def flippable(x: typing.Any): # default # Since these are in the same lexical scope as the original definition of the # generic function `flippable`, we could do this using `@generic`, but # later extensions (which are the whole point of traits) will need to specify - # on which function the new methods are to be registered, using `@generic_for`. + # on which function the new methods are to be registered, using `@generic_addmethod`. # So let's do that to show how it's done. - @generic_for(flippable) + @generic_addmethod(flippable) def flippable(x: str): # noqa: F811 return IsFlippable() - @generic_for(flippable) + @generic_addmethod(flippable) def flippable(x: int): # noqa: F811 return IsNotFlippable() @@ -289,7 +289,7 @@ def flippable(x: int): # noqa: F811 def flip(x: typing.Any): return flip(flippable(x), x) - # Implementation of `flip`. Same comment about `@generic_for` as above. + # Implementation of `flip`. Same comment about `@generic_addmethod` as above. # # Here we provide one implementation for "flippable" objects and another one # for "nonflippable" objects. Note this dispatches regardless of the actual @@ -299,10 +299,10 @@ def flip(x: typing.Any): # We could also add methods for specific types if needed. Note this is not # Julia, so the first matching definition wins, instead of the most specific # one. - @generic_for(flip) + @generic_addmethod(flip) def flip(traitvalue: IsFlippable, x: typing.Any): # noqa: F811 return x[::-1] - @generic_for(flip) + @generic_addmethod(flip) def flip(traitvalue: IsNotFlippable, x: typing.Any): # noqa: F811 raise TypeError(f"{repr(x)} is IsNotFlippable") From ff6dadd9d6be8de9eaa04540034d588a3eba47dc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:34:18 +0300 Subject: [PATCH 166/832] move `ulp` and `almosteq` to new module `unpythonic.numutil` They are still available in the top-level namespace, as usual. --- unpythonic/__init__.py | 1 + unpythonic/mathseq.py | 41 +------------------ unpythonic/misc.py | 19 +-------- unpythonic/numutil.py | 67 ++++++++++++++++++++++++++++++++ unpythonic/tests/test_mathseq.py | 28 +------------ unpythonic/tests/test_misc.py | 11 +----- unpythonic/tests/test_numutil.py | 44 +++++++++++++++++++++ 7 files changed, 116 insertions(+), 95 deletions(-) create mode 100644 unpythonic/numutil.py create mode 100644 unpythonic/tests/test_numutil.py diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 037d1e53..87809af6 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -35,6 +35,7 @@ from .llist import * # noqa: F401, F403 from .mathseq import * # noqa: F401, F403 from .misc import * # noqa: F401, F403 +from .numutil import * # noqa: F401, F403 from .seq import * # noqa: F401, F403 from .singleton import * # noqa: F401, F403 from .slicing import * # noqa: F401, F403 diff --git a/unpythonic/mathseq.py b/unpythonic/mathseq.py index 3a78fc04..fee5452a 100644 --- a/unpythonic/mathseq.py +++ b/unpythonic/mathseq.py @@ -20,7 +20,6 @@ __all__ = ["s", "imathify", "gmathify", "m", "mg", # old names, pre-0.14.3, will go away in 0.15.0 - "almosteq", "sadd", "ssub", "sabs", "spos", "sneg", "sinvert", "smul", "spow", "struediv", "sfloordiv", "smod", "sdivmod", "sround", "strunc", "sfloor", "sceil", @@ -45,13 +44,13 @@ from .it import take, rev, window from .gmemo import imemoize, gmemoize +from .numutil import almosteq class _NoSuchType: pass # stuff to support float, mpf and SymPy expressions transparently # -from sys import float_info from math import log as math_log, copysign, trunc, floor, ceil try: from mpmath import mpf, almosteq as mpf_almosteq @@ -98,44 +97,6 @@ def sign(x): sign = _numsign _symExpr = _NoSuchType -# TODO: Overhaul `almosteq` in v0.15.0, should work like mpf for consistency. -# TODO: Also move it to `unpythonic.misc`, where `ulp` already is. Or make a `numutil`. -def almosteq(a, b, tol=1e-8): - """Almost-equality that supports several formats. - - The tolerance ``tol`` is used for the builtin ``float`` and ``mpmath.mpf``. - - For ``mpmath.mpf``, we just delegate to ``mpmath.almosteq``, with the given - ``tol``. For ``float``, we use the strategy suggested in: - - https://floating-point-gui.de/errors/comparison/ - - Anything else, for example SymPy expressions, strings, and containers - (regardless of content), is tested for exact equality. - - **CAUTION**: Although placed in ``unpythonic.mathseq``, this function - **does not** support iterables; rather, it is a low-level tool that is - exposed in the public API in the hope it may be useful elsewhere. - """ - if a == b: # infinities and such, plus any non-float type - return True - - if isinstance(a, mpf) and isinstance(b, mpf): - return mpf_almosteq(a, b, tol) - # compare as native float if only one is an mpf - elif isinstance(a, mpf) and isinstance(b, (float, int)): - a = float(a) - elif isinstance(a, (float, int)) and isinstance(b, mpf): - b = float(b) - - if not all(isinstance(x, (float, int)) for x in (a, b)): - return False # non-float type, already determined that a != b - min_normal = float_info.min - max_float = float_info.max - d = abs(a - b) - if a == 0 or b == 0 or d < min_normal: - return d < tol * min_normal - return d / min(abs(a) + abs(b), max_float) < tol def s(*spec): """Create a lazy mathematical sequence. diff --git a/unpythonic/misc.py b/unpythonic/misc.py index abe07183..be0e5dbc 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -5,16 +5,14 @@ "pack", "namelambda", "timer", "getattrrec", "setattrrec", "Popper", "CountingIterator", - "ulp", "slurp", "async_raise", "callsite_filename", "safeissubclass"] from copy import copy from functools import partial from itertools import count import inspect -from math import floor, log2 from queue import Empty -from sys import float_info, version_info +from sys import version_info import threading from time import monotonic from types import CodeType, FunctionType, LambdaType, TracebackType @@ -654,21 +652,6 @@ def __next__(self): self.count += 1 return x -# TODO: move to a new module unpythonic.numutil in v0.15.0. -def ulp(x): # Unit in the Last Place - """Given a float x, return the unit in the last place (ULP). - - This is the numerical value of the least-significant bit, as a float. - For x = 1.0, the ULP is the machine epsilon (by definition of machine epsilon). - - See: - https://en.wikipedia.org/wiki/Unit_in_the_last_place - """ - eps = float_info.epsilon - # m_min = abs. value represented by a mantissa of 1.0, with the same exponent as x has - m_min = 2**floor(log2(abs(x))) - return m_min * eps - def slurp(queue): """Slurp all items currently on a queue.Queue into a list. diff --git a/unpythonic/numutil.py b/unpythonic/numutil.py new file mode 100644 index 00000000..34450f45 --- /dev/null +++ b/unpythonic/numutil.py @@ -0,0 +1,67 @@ +# -*- coding: utf-8 -*- +"""Low-level utilities for numerics.""" + +__all__ = ["almosteq", "ulp"] + +from math import floor, log2 +import sys + +class _NoSuchType: + pass + +try: + from mpmath import mpf, almosteq as mpf_almosteq +except ImportError: # pragma: no cover, optional at runtime, but installed at development time. + # Can't use a gensym here since `mpf` must be a unique *type*. + mpf = _NoSuchType + mpf_almosteq = None + + +# TODO: Overhaul `almosteq` in v0.15.0, should work like mpf for consistency. +def almosteq(a, b, tol=1e-8): + """Almost-equality that supports several formats. + + The tolerance ``tol`` is used for the builtin ``float`` and ``mpmath.mpf``. + + For ``mpmath.mpf``, we just delegate to ``mpmath.almosteq``, with the given + ``tol``. For ``float``, we use the strategy suggested in: + + https://floating-point-gui.de/errors/comparison/ + + Anything else, for example SymPy expressions, strings, and containers + (regardless of content), is tested for exact equality. + """ + if a == b: # infinities and such, plus any non-float type + return True + + if isinstance(a, mpf) and isinstance(b, mpf): + return mpf_almosteq(a, b, tol) + # compare as native float if only one is an mpf + elif isinstance(a, mpf) and isinstance(b, (float, int)): + a = float(a) + elif isinstance(a, (float, int)) and isinstance(b, mpf): + b = float(b) + + if not all(isinstance(x, (float, int)) for x in (a, b)): + return False # non-float type, already determined that a != b + min_normal = sys.float_info.min + max_float = sys.float_info.max + d = abs(a - b) + if a == 0 or b == 0 or d < min_normal: + return d < tol * min_normal + return d / min(abs(a) + abs(b), max_float) < tol + + +def ulp(x): # Unit in the Last Place + """Given a float x, return the unit in the last place (ULP). + + This is the numerical value of the least-significant bit, as a float. + For x = 1.0, the ULP is the machine epsilon (by definition of machine epsilon). + + See: + https://en.wikipedia.org/wiki/Unit_in_the_last_place + """ + eps = sys.float_info.epsilon + # m_min = abs. value represented by a mantissa of 1.0, with the same exponent as x has + m_min = 2**floor(log2(abs(x))) + return m_min * eps diff --git a/unpythonic/tests/test_mathseq.py b/unpythonic/tests/test_mathseq.py index 8388c285..0c6a2de8 100644 --- a/unpythonic/tests/test_mathseq.py +++ b/unpythonic/tests/test_mathseq.py @@ -5,12 +5,11 @@ from operator import mul from math import exp, trunc, floor, ceil -from sys import float_info from ..mathseq import (s, imathify, gmathify, sadd, smul, spow, cauchyprod, primes, fibonacci, - sign, log, almosteq) + sign, log) from ..it import take, last from ..fold import scanl from ..gmemo import imemoize @@ -46,31 +45,6 @@ def runtests(): x = symbols("x", positive=True) test[log(symbolicExp(x)) == x] - with testset("almosteq"): - # For anything but floating-point inputs, it's exact equality. - test[almosteq("abc", "abc")] - test[not almosteq("ab", "abc")] - - test[almosteq(1.0, 1.0 + ulp(1.0))] - - # TODO: counterintuitively, need a large tolerance here, because when one operand is zero, - # TODO: the final tolerance is actually tol*min_normal. - min_normal = float_info.min - test[almosteq(min_normal / 2, 0, tol=1.0)] - - too_large = 2**int(1e6) - test_raises[OverflowError, float(too_large), "UPDATE THIS, need a float overflow here."] - test[almosteq(too_large, too_large + 1)] # works, because 1/too_large is very small. - - try: - from mpmath import mpf - except ImportError: # pragma: no cover - error["mpmath not installed in this Python, cannot test arbitrary precision input for mathseq."] - else: - test[almosteq(mpf(1.0), mpf(1.0 + ulp(1.0)))] - test[almosteq(1.0, mpf(1.0 + ulp(1.0)))] - test[almosteq(mpf(1.0), 1.0 + ulp(1.0))] - # explicitly listed elements, same as a genexpr using tuple input (but supports infix math) with testset("s, convenience"): test[tuple(s(1)) == (1,)] diff --git a/unpythonic/tests/test_misc.py b/unpythonic/tests/test_misc.py index edadb78f..cfee1bd4 100644 --- a/unpythonic/tests/test_misc.py +++ b/unpythonic/tests/test_misc.py @@ -6,7 +6,6 @@ from operator import add from functools import partial from collections import deque -from sys import float_info from queue import Queue from time import sleep import threading @@ -14,7 +13,7 @@ from ..misc import (call, callwith, raisef, tryf, equip_with_traceback, pack, namelambda, timer, - getattrrec, setattrrec, Popper, CountingIterator, ulp, slurp, + getattrrec, setattrrec, Popper, CountingIterator, slurp, async_raise, callsite_filename, safeissubclass) from ..fun import withself from ..env import env @@ -270,14 +269,6 @@ def __init__(self, x): test[it.count == k] test[it.count == 5] - # Unit in the Last Place, float utility - # https://en.wikipedia.org/wiki/Unit_in_the_last_place - with testset("ulp (unit in the last place; float utility)"): - test[ulp(1.0) == float_info.epsilon] - # test also at some base-2 exponent switch points - test[ulp(2.0) == 2 * float_info.epsilon] - test[ulp(0.5) == 0.5 * float_info.epsilon] - with testset("slurp (drain a queue into a list)"): q = Queue() for k in range(10): diff --git a/unpythonic/tests/test_numutil.py b/unpythonic/tests/test_numutil.py new file mode 100644 index 00000000..ae2f808d --- /dev/null +++ b/unpythonic/tests/test_numutil.py @@ -0,0 +1,44 @@ +# -*- coding: utf-8 -*- + +from ..syntax import macros, test, test_raises, error # noqa: F401 +from ..test.fixtures import session, testset + +import sys + +from ..numutil import almosteq, ulp + +def runtests(): + with testset("ulp (unit in the last place; float utility)"): + test[ulp(1.0) == sys.float_info.epsilon] + # test also at some base-2 exponent switch points + test[ulp(2.0) == 2 * sys.float_info.epsilon] + test[ulp(0.5) == 0.5 * sys.float_info.epsilon] + + with testset("almosteq"): + # For anything but floating-point inputs, it's exact equality. + test[almosteq("abc", "abc")] + test[not almosteq("ab", "abc")] + + test[almosteq(1.0, 1.0 + ulp(1.0))] + + # TODO: counterintuitively, need a large tolerance here, because when one operand is zero, + # TODO: the final tolerance is actually tol*min_normal. + min_normal = sys.float_info.min + test[almosteq(min_normal / 2, 0, tol=1.0)] + + too_large = 2**int(1e6) + test_raises[OverflowError, float(too_large), "UPDATE THIS, need a float overflow here."] + test[almosteq(too_large, too_large + 1)] # works, because 1/too_large is very small. + + try: + from mpmath import mpf + except ImportError: # pragma: no cover + error["mpmath not installed in this Python, cannot test arbitrary precision input for mathseq."] + else: + test[almosteq(mpf(1.0), mpf(1.0 + ulp(1.0)))] + test[almosteq(1.0, mpf(1.0 + ulp(1.0)))] + test[almosteq(mpf(1.0), 1.0 + ulp(1.0))] + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() From a7f9a2fe77e8d9b97fc21ed562d8780205b77b2d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:35:26 +0300 Subject: [PATCH 167/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d9ab5e52..4d58bd42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,6 +65,8 @@ For future plans, see our [Python language version support status](https://githu - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. +- The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From 2314b207bfe0cc32360783c485facd9b718fcfc7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:35:34 +0300 Subject: [PATCH 168/832] bump version --- unpythonic/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 87809af6..7e1e4db6 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.14.3' +__version__ = '0.15.0' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From 9782d770b337cd919a0b3c61f20e2375a16b7d11 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:39:49 +0300 Subject: [PATCH 169/832] 0.15: remove deprecated names `m`, `mg` Please use `imathify` and `gmathify`, respectively. The new names are more descriptive, as well as conform to naming conventions used elsewhere in `unpythonic`. --- doc/dialects/pytkell.md | 2 +- unpythonic/mathseq.py | 18 ------------------ 2 files changed, 1 insertion(+), 19 deletions(-) diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index 50d44bd8..cac0ef00 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -56,7 +56,7 @@ We also import some macros and functions to serve as dialect builtins: - Memoization ``memoize``, ``gmemoize``, ``imemoize``, ``fimemoize`` - Functional updates ``fup`` and ``fupdate`` - Immutable dict ``frozendict`` - - Mathematical sequences ``s``, ``m``, ``mg`` + - Mathematical sequences ``s``, ``imathify``, ``gmathify`` - Iterable utilities ``islice`` (`unpythonic`'s version), ``take``, ``drop``, ``split_at``, ``first``, ``second``, ``nth``, ``last`` - Function arglist reordering utilities ``flip``, ``rotate`` diff --git a/unpythonic/mathseq.py b/unpythonic/mathseq.py index fee5452a..6b012d7e 100644 --- a/unpythonic/mathseq.py +++ b/unpythonic/mathseq.py @@ -19,7 +19,6 @@ """ __all__ = ["s", "imathify", "gmathify", - "m", "mg", # old names, pre-0.14.3, will go away in 0.15.0 "sadd", "ssub", "sabs", "spos", "sneg", "sinvert", "smul", "spow", "struediv", "sfloordiv", "smod", "sdivmod", "sround", "strunc", "sfloor", "sceil", @@ -27,7 +26,6 @@ "cauchyprod", "diagonal_reduce", "fibonacci", "primes"] -from warnings import warn from itertools import repeat, takewhile, count from functools import wraps from operator import (add as primitive_add, mul as primitive_mul, @@ -595,14 +593,6 @@ def __ge__(self, other): def __gt__(self, other): return sgt(self, other) -class m(imathify): # pragma: no cover - """Alias for `imathify`, for backward compatibility. - - Will be removed in 0.15.0.""" - def __init__(self, iterable): - warn("`m` has been renamed `imathify`, which is more descriptive; this alias will be removed in 0.15.0.", FutureWarning) - super().__init__(iterable) - def gmathify(gfunc): """Decorator: make gfunc imathify() the returned generator instances. @@ -620,14 +610,6 @@ def mathify(*args, **kwargs): return imathify(gfunc(*args, **kwargs)) return mathify -def mg(gfunc): # pragma: no cover - """Alias for `gmathify`, for backward compatibility. - - Will be removed in 0.15.0. - """ - warn("`mg` has been renamed `gmathify`, which is more descriptive; this alias will be removed in 0.15.0.", FutureWarning) - return gmathify(gfunc) - # ----------------------------------------------------------------------------- # We expose the full set of "imathify" operators also as functions à la the ``operator`` module. # Prefix "s", short for "mathematical Sequence". From ba381c988e12f9214d5efd685a4123fbb503aadd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:44:51 +0300 Subject: [PATCH 170/832] update changelog --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d58bd42..d16fb30c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,6 +67,15 @@ For future plans, see our [Python language version support status](https://githu - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. +- As promised, names deprecated during 0.14.x have been removed. Old name on the left, new name on the right: + - `m` → `imathify` + - `mg` → `gmathify` + - `setescape` → `catch` + - `escape` → `throw` + - `getvalue`, `runpipe` → `exitpipe` (combined into one) + +- Drop support for deprecated argument format for `raisef`. Now the usage is `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From 6e8753902de6ce0a657bda25ae10176ae72ad6c6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:51:58 +0300 Subject: [PATCH 171/832] fix tests --- unpythonic/syntax/tests/test_lazify.py | 4 ++-- unpythonic/tests/test_conditions.py | 2 +- unpythonic/tests/test_it.py | 3 ++- unpythonic/tests/test_mathseq.py | 3 ++- unpythonic/tests/test_misc.py | 4 ---- 5 files changed, 7 insertions(+), 9 deletions(-) diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 308eb9dd..f4f1fc35 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -124,8 +124,8 @@ def my_if(p, a, b): # test the laziness # note the raisef() calls; in regular Python, they'd run anyway before my_if() gets control. - test[my_if(True, 23, raisef(RuntimeError, "I was evaluated!")) == 23] - test[my_if(False, raisef(RuntimeError, "I was evaluated!"), 42) == 42] + test[my_if(True, 23, raisef(RuntimeError("I was evaluated!"))) == 23] + test[my_if(False, raisef(RuntimeError("I was evaluated!")), 42) == 42] # In this example, the divisions by zero are never performed. test[my_if(True, 23, 1 / 0) == 23] diff --git a/unpythonic/tests/test_conditions.py b/unpythonic/tests/test_conditions.py index 0f7b00ed..e543dded 100644 --- a/unpythonic/tests/test_conditions.py +++ b/unpythonic/tests/test_conditions.py @@ -48,7 +48,7 @@ def lowlevel(): with restarts(use_value=(lambda x: x), double=(lambda x: 2 * x), drop=(lambda x: _drop), - bail=(lambda x: raisef(ValueError, x))) as result: + bail=(lambda x: raisef(ValueError(x)))) as result: # Let's pretend we only want to deal with even numbers. # Realistic errors would be something like nonexistent file, disk full, network down, ... if k % 2 == 1: diff --git a/unpythonic/tests/test_it.py b/unpythonic/tests/test_it.py index 788f81f1..14ec4cba 100644 --- a/unpythonic/tests/test_it.py +++ b/unpythonic/tests/test_it.py @@ -33,7 +33,8 @@ from ..fun import composel, identity, curry from ..gmemo import imemoize, gmemoize from ..mathseq import s -from ..misc import Popper, ulp +from ..misc import Popper +from ..numutil import ulp def runtests(): with testset("mapping and zipping"): diff --git a/unpythonic/tests/test_mathseq.py b/unpythonic/tests/test_mathseq.py index 0c6a2de8..e04f7305 100644 --- a/unpythonic/tests/test_mathseq.py +++ b/unpythonic/tests/test_mathseq.py @@ -13,7 +13,8 @@ from ..it import take, last from ..fold import scanl from ..gmemo import imemoize -from ..misc import timer, ulp +from ..misc import timer +from ..numutil import ulp def runtests(): with testset("sign (adapter, numeric and symbolic)"): diff --git a/unpythonic/tests/test_misc.py b/unpythonic/tests/test_misc.py index cfee1bd4..75057a35 100644 --- a/unpythonic/tests/test_misc.py +++ b/unpythonic/tests/test_misc.py @@ -117,10 +117,6 @@ def mul3(a, b, c): except ValueError as err: test[err.__cause__ is exc] # cause specified, like `raise ... from ...` - # raisef with old-style parameters (as of v0.14.2, deprecated, will be dropped in v0.15.0) - raise_instance = lambda: raisef(ValueError, "all ok") - test_raises[ValueError, raise_instance()] - # can also raise an exception class (no instance) test_raises[StopIteration, raisef(StopIteration)] From d8f72bbd7c9391873fdf934ee8d2d6eb7efed720 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:53:34 +0300 Subject: [PATCH 172/832] 0.15: curry-friendly parameter ordering in `unpythonic.it.window` Usage is now `window(n, iterable)`. --- CHANGELOG.md | 2 ++ doc/features.md | 4 ++-- unpythonic/it.py | 5 ++--- unpythonic/mathseq.py | 2 +- unpythonic/tests/test_it.py | 12 ++++++------ 5 files changed, 13 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d16fb30c..24600641 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,6 +76,8 @@ For future plans, see our [Python language version support status](https://githu - Drop support for deprecated argument format for `raisef`. Now the usage is `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. +- Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. diff --git a/doc/features.md b/doc/features.md index a5ca5afb..1bc7ebfa 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1481,7 +1481,7 @@ assert find(lambda x: x >= 3, gen) == 4 # if consumable, consumed as usual # window: length-n sliding window iterator for general iterables lst = (x for x in range(5)) out = [] -for a, b, c in window(lst, n=3): +for a, b, c in window(3, lst): out.append((a, b, c)) assert out == [(0, 1, 2), (1, 2, 3), (2, 3, 4)] @@ -3650,7 +3650,7 @@ from unpythonic import Popper, window inp = deque(range(3)) out = [] -for a, b in window(Popper(inp)): +for a, b in window(2, Popper(inp)): out.append((a, b)) if a < 10: inp.append(a + 10) diff --git a/unpythonic/it.py b/unpythonic/it.py index d5a434ff..6d707815 100644 --- a/unpythonic/it.py +++ b/unpythonic/it.py @@ -743,8 +743,7 @@ def find(predicate, iterable, default=None): """ return next(filter(predicate, iterable), default) -# TODO: in 0.15.0, maybe switch the argument order of window() for curry-friendliness? -def window(iterable, n=2): +def window(n, iterable): """Sliding length-n window iterator for a general iterable. Acts like ``zip(s, s[1:], ..., s[n-1:])`` for a sequence ``s``, but the input @@ -828,7 +827,7 @@ def within(tol, iterable): (infinite output, or terminating the output early if a part of it looks like a converging sequence; think a local maximum of `cos(x)`). """ - for a, b in window(iterable, n=2): + for a, b in window(2, iterable): yield a if abs(a - b) <= tol: yield b diff --git a/unpythonic/mathseq.py b/unpythonic/mathseq.py index 6b012d7e..8d2e041c 100644 --- a/unpythonic/mathseq.py +++ b/unpythonic/mathseq.py @@ -312,7 +312,7 @@ def analyze(*spec): # raw spec (part before '...' if any) --> description # Most unrecognized sequences trigger this case. raise SyntaxError(f"Specification did not match any supported formula: '{origspec}'") else: # more elements are optional but must be consistent - data = [analyze(*triplet) for triplet in window(iterable=spec, n=3)] + data = [analyze(*triplet) for triplet in window(3, spec)] seqtypes, x0s, ks = zip(*data) def isconst(xs): first, *rest = xs diff --git a/unpythonic/tests/test_it.py b/unpythonic/tests/test_it.py index 14ec4cba..2145aaa8 100644 --- a/unpythonic/tests/test_it.py +++ b/unpythonic/tests/test_it.py @@ -275,26 +275,26 @@ def primes(): with testset("window"): lst = list(range(5)) out = [] - for a, b, c in window(lst, n=3): + for a, b, c in window(3, lst): out.append((a, b, c)) test[lst == list(range(5))] test[out == [(0, 1, 2), (1, 2, 3), (2, 3, 4)]] lst = range(5) out = [] - for a, b, c in window(lst, n=3): + for a, b, c in window(3, lst): out.append((a, b, c)) test[lst == range(5)] test[out == [(0, 1, 2), (1, 2, 3), (2, 3, 4)]] lst = (x for x in range(5)) out = [] - for a, b, c in window(lst, n=3): + for a, b, c in window(3, lst): out.append((a, b, c)) test[out == [(0, 1, 2), (1, 2, 3), (2, 3, 4)]] - test_raises[ValueError, window(range(5), n=1)] - test[tuple(window(range(5), n=10)) == ()] + test_raises[ValueError, window(1, range(5))] # n must be at least 2 + test[tuple(window(10, range(5))) == ()] # iterable shorter than window length with testset("window integration with Popper"): # This works because window() iter()s the Popper, but the Popper never @@ -305,7 +305,7 @@ def primes(): # because the window needs them to initialize itself.) inp = deque(range(3)) out = [] - for a, b in window(Popper(inp)): + for a, b in window(2, Popper(inp)): out.append((a, b)) if a < 10: inp.append(a + 10) From 8f800fca57e82e176f13b3d1551f89cf23263d07 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:57:53 +0300 Subject: [PATCH 173/832] mention change to parameter order for `window` --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 1bc7ebfa..3c05361e 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1361,7 +1361,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `mapr`, `zipr`, `mapr_longest`, `zipr_longest`: map/zip, then reverse the result. For multiple inputs, syncs the **left** ends. - `map`: curry-friendly wrapper for the builtin, making it mandatory to specify at least one iterable. **Added in v0.14.2.** - *windowing, chunking, and similar*: - - `window`: sliding length-n window iterator for general iterables. Acts like the well-known [n-gram zip trick](http://www.locallyoptimal.com/blog/2013/01/20/elegant-n-gram-generation-in-python/), but the input can be any iterable. + - `window`: sliding length-n window iterator for general iterables. Acts like the well-known [n-gram zip trick](http://www.locallyoptimal.com/blog/2013/01/20/elegant-n-gram-generation-in-python/), but the input can be any iterable. **Changed in v0.15.0.** *Parameter ordering is now `window(n, iterable)`, to make it curry-friendly.* - `chunked`: split an iterable into constant-length chunks. **Added in v0.14.2.** - `pad`: extend an iterable to length at least `n` with a `fillvalue`. **Added in v0.14.2.** - `interleave`: interleave items from several iterables: `interleave(a, b, c)` → `a0, b0, c0, a1, b1, c1, ...` until the next item does not exist. **Added in v0.14.2.** From 55e96b7fcc00299c3a363130d50dbae76a18c1c0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 13:59:18 +0300 Subject: [PATCH 174/832] bump scope analyzer changes to 0.16 at earliest --- unpythonic/syntax/scopeanalyzer.py | 6 +++--- unpythonic/syntax/tests/test_scopeanalyzer.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index eafea116..50b16c2f 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -24,7 +24,7 @@ **CAUTION**: -What we do currently (before v0.15.0) doesn't fully make sense. +What we do currently (before v0.16.0) doesn't fully make sense. Scope - in the sense of controlling lexical name resolution - is a static (purely lexical) concept, but whether a particular name (once lexically @@ -37,7 +37,7 @@ exceptional trivial cases such as `if 1`, this depends on the condition part of the `if` at run time, and thus can't be statically determined. -In order to make more sense, in v0.15.0, we will migrate to a fully static analysis. +In order to make more sense, in v0.16.0, we will migrate to a fully static analysis. This will make the analyzer consistent with how Python itself handles scoping, at the cost of slightly (but backward-incompatibly) changing the semantics of some corner cases in the usage of `let` and `do`. @@ -47,7 +47,7 @@ It is disabled when `scoped_transform` calls `get_lexical_variables`, to preserve old behavior until the next opportunity for a public interface change. -In v0.15.0, we will make `scoped_transform` use the fully lexical mode. +In v0.16.0, we will make `scoped_transform` use the fully lexical mode. **NOTE**: diff --git a/unpythonic/syntax/tests/test_scopeanalyzer.py b/unpythonic/syntax/tests/test_scopeanalyzer.py index 35ea18e9..2c2b0927 100644 --- a/unpythonic/syntax/tests/test_scopeanalyzer.py +++ b/unpythonic/syntax/tests/test_scopeanalyzer.py @@ -255,14 +255,14 @@ def f(): # noqa: F811 n["_apply_test_here_"] scoped_transform(scoped_localvar1, callback=make_checker(["f", "x"])) - # TODO: In 0.15.x, fully lexical scope analysis; update this test at that time. + # TODO: In 0.16.x, fully lexical scope analysis; update this test at that time. with q as scoped_localvar2: def f(): # noqa: F811 n["_apply_test_here_"] x = 42 # noqa: F841 scoped_transform(scoped_localvar2, callback=make_checker(["f"])) # x not yet created - # TODO: In 0.15.x, fully lexical scope analysis; update this test at that time. + # TODO: In 0.16.x, fully lexical scope analysis; update this test at that time. with q as scoped_localvar3: def f(): # noqa: F811 x = 42 # noqa: F841 From 8e111b1aa3357fba5f0bf509967af902e8f31280 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 14:07:39 +0300 Subject: [PATCH 175/832] 0.15: parameter naming: `l` -> `length` --- CHANGELOG.md | 2 ++ unpythonic/collections.py | 60 +++++++++++++++++++-------------------- 2 files changed, 31 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 24600641..4534dbcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,8 @@ For future plans, see our [Python language version support status](https://githu - Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. +- Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. diff --git a/unpythonic/collections.py b/unpythonic/collections.py index b71b91ff..0a7d6dbe 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -786,65 +786,63 @@ def _getone(self, k): return self.v[i] return self.seq[k] # not in slice -# TODO: fix flake8 E741 ambiguous variable name "l". Here it's part of the public API, so we'll have to wait until 15.0 to change the name. -def in_slice(i, s, l=None): +def in_slice(i, s, length=None): """Return whether the int i is in the slice s. For convenience, ``s`` may be int instead of slice; then return whether ``i == s``. - The optional ``l`` is the length of the sequence being indexed, used for + The optional ``length`` is the length of the sequence being indexed, used for interpreting any negative indices, and default start and stop values (if ``s.start`` or ``s.stop`` is ``None``). - If ``l is None``, negative or missing ``s.start`` or ``s.stop`` may raise + If ``length is None``, negative or missing ``s.start`` or ``s.stop`` may raise ValueError. (A negative ``s.step`` by itself does not need ``l``.) """ if not isinstance(s, (slice, int)): raise TypeError(f"s must be slice or int, got {type(s)} with value {s}") if not isinstance(i, int): raise TypeError(f"i must be int, got {type(i)} with value {i}") - wrap = _make_negidx_converter(l) + wrap = _make_negidx_converter(length) i = wrap(i) if isinstance(s, int): s = wrap(s) return i == s - start, stop, step = _canonize_slice(s, l, wrap) + start, stop, step = _canonize_slice(s, length, wrap) cmp_start, cmp_end = (ge, lt) if step > 0 else (le, gt) at_or_after_start = cmp_start(i, start) before_stop = cmp_end(i, stop) on_grid = (i - start) % step == 0 return at_or_after_start and on_grid and before_stop -# TODO: fix flake8 E741 ambiguous variable name "l". Here it's part of the public API, so we'll have to wait until 15.0 to change the name. -def index_in_slice(i, s, l=None): +def index_in_slice(i, s, length=None): """Return the index of the int i in the slice s, or None if i is not in s. (I.e. how-manyth item of the slice the index i is.) - The optional sequence length ``l`` works the same as in ``in_slice``. + The optional sequence length ``length`` works the same as in ``in_slice``. """ - return _index_in_slice(i, s, l) + return _index_in_slice(i, s, length) # efficiency: allow skipping the validation check for call sites # that have already checked with in_slice(). -def _index_in_slice(i, s, n=None, _validate=True): # n: length of sequence being indexed - if (not _validate) or in_slice(i, s, n): - wrap = _make_negidx_converter(n) - start, _, step = _canonize_slice(s, n, wrap) +def _index_in_slice(i, s, length=None, _validate=True): + if (not _validate) or in_slice(i, s, length): + wrap = _make_negidx_converter(length) + start, _, step = _canonize_slice(s, length, wrap) return (wrap(i) - start) // step -def _make_negidx_converter(n): # n: length of sequence being indexed - if n is not None: - if not isinstance(n, int): - raise TypeError(f"n must be int, got {type(n)} with value {n}") - if n <= 0: - raise ValueError(f"n must be an int >= 1, got {n}") +def _make_negidx_converter(length): + if length is not None: + if not isinstance(length, int): + raise TypeError(f"length must be int, got {type(length)} with value {length}") + if length <= 0: + raise ValueError(f"length must be an int >= 1, got {length}") def apply_conversion(k): - return k % n + return k % length else: def apply_conversion(k): - raise ValueError("Need n to interpret negative indices") + raise ValueError("Need length to interpret negative indices") def convert(k): if k is not None: if not isinstance(k, int): @@ -856,12 +854,12 @@ def convert(k): # Almost standard semantics for negative indices. Usually -n < k < n, # but here we must allow for conversion of the end position, for # which the last valid value is one past the end. - if n is not None and not -n <= k <= n: - raise IndexError(f"Should have -n <= k <= n, but n = {n}, and k = {k}") + if length is not None and not -length <= k <= length: + raise IndexError(f"Should have -length <= k <= length, but length = {length}, and k = {k}") return apply_conversion(k) if k < 0 else k return convert -def _canonize_slice(s, n=None, wrap=None): # convert negatives, inject defaults. +def _canonize_slice(s, length=None, wrap=None): # convert negatives, inject defaults. if not isinstance(s, slice): # Not triggered in the current code, because this is an internal function # and `in_slice` already checks; but let's be careful in case this is later @@ -874,23 +872,23 @@ def _canonize_slice(s, n=None, wrap=None): # convert negatives, inject defaults if step == 0: raise ValueError("slice step cannot be zero") # message copied from range(5)[0:4:0] - wrap = wrap or _make_negidx_converter(n) + wrap = wrap or _make_negidx_converter(length) start = wrap(s.start) if start is None: if step > 0: start = 0 else: - if n is None: - raise ValueError("Need n to determine default start for step < 0") + if length is None: + raise ValueError("Need length to determine default start for step < 0") start = wrap(-1) stop = wrap(s.stop) if stop is None: if step > 0: - if n is None: - raise ValueError("Need n to determine default stop for step > 0") - stop = n + if length is None: + raise ValueError("Need length to determine default stop for step > 0") + stop = length else: stop = -1 # yes, really -1 to have index 0 inside the slice From fb7901fca4dbcad0ded7a4b12777dd4e81505ba1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 14:09:10 +0300 Subject: [PATCH 176/832] bump `almosteq` semantic change to 0.16 --- unpythonic/numutil.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/numutil.py b/unpythonic/numutil.py index 34450f45..f573f262 100644 --- a/unpythonic/numutil.py +++ b/unpythonic/numutil.py @@ -17,7 +17,7 @@ class _NoSuchType: mpf_almosteq = None -# TODO: Overhaul `almosteq` in v0.15.0, should work like mpf for consistency. +# TODO: Overhaul `almosteq` in v0.16.0, should work like mpf for consistency. def almosteq(a, b, tol=1e-8): """Almost-equality that supports several formats. From cddc34c40f01dcd8dd54e4b9c74f7e67806bf620 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 14:09:38 +0300 Subject: [PATCH 177/832] 0.15: all pending interface changes should now be done. --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 52459065..4d4f010b 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -83,8 +83,6 @@ # TODO: Check expansion order of several macros in the same `with` statement -# TODO: grep codebase for "0.15", may have some pending interface changes that don't have their own GitHub issue (e.g. parameter ordering of `unpythonic.it.window`) - # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... # TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. From 0ae54a194aeae0445114d4e9aca818948dbec3a0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 14:31:16 +0300 Subject: [PATCH 178/832] add some missing unit tests --- unpythonic/syntax/tests/test_nameutil.py | 30 +++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_nameutil.py b/unpythonic/syntax/tests/test_nameutil.py index a7c1fcf8..b1852a4b 100644 --- a/unpythonic/syntax/tests/test_nameutil.py +++ b/unpythonic/syntax/tests/test_nameutil.py @@ -6,7 +6,10 @@ from mcpyrate.quotes import macros, q, h # noqa: F401, F811 -from ...syntax.nameutil import isx, getname +from mcpyrate.expander import MacroExpander + +from ...syntax.nameutil import (isx, getname, + is_unexpanded_expr_macro, is_unexpanded_block_macro) from ast import Call @@ -32,6 +35,31 @@ def runtests(): test[getname(attribute) == "ok"] test[getname(attribute, accept_attr=False) is None] + with testset("is_unexpanded_expr_macro"): + def dummymacro(tree, **kw): + return tree + m = MacroExpander({"dummy": dummymacro}, filename="") + + # we need a macro that is bound in the expander we pass to the analyzer. + test[is_unexpanded_expr_macro(dummymacro, m, q[dummy[blah]])] # noqa: F821, only quoted + test[not is_unexpanded_expr_macro(dummymacro, m, q[notdummy[blah]])] # noqa: F821, only quoted + test[not is_unexpanded_expr_macro(dummymacro, m, q[42])] + + with testset("is_unexpanded_block_macro"): + with q as quoted: + with dummy: # noqa: F821, only quoted + ... + test[is_unexpanded_block_macro(dummymacro, m, quoted[0])] + + with q as quoted: + with notdummy: # noqa: F821, only quoted + ... + test[not is_unexpanded_block_macro(dummymacro, m, quoted[0])] + + with q as quoted: + a = 42 # noqa: F841 + test[not is_unexpanded_block_macro(dummymacro, m, quoted[0])] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 2c43a8a5c91c820b940413d5b739d9d4d03825e2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 14:34:15 +0300 Subject: [PATCH 179/832] update test --- unpythonic/syntax/tests/test_nameutil.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/tests/test_nameutil.py b/unpythonic/syntax/tests/test_nameutil.py index b1852a4b..6158e639 100644 --- a/unpythonic/syntax/tests/test_nameutil.py +++ b/unpythonic/syntax/tests/test_nameutil.py @@ -40,9 +40,11 @@ def dummymacro(tree, **kw): return tree m = MacroExpander({"dummy": dummymacro}, filename="") - # we need a macro that is bound in the expander we pass to the analyzer. - test[is_unexpanded_expr_macro(dummymacro, m, q[dummy[blah]])] # noqa: F821, only quoted - test[not is_unexpanded_expr_macro(dummymacro, m, q[notdummy[blah]])] # noqa: F821, only quoted + # The tree being tested needs to invoke a macro that is bound in the expander we pass to the analyzer. + # Note we detect whether the invocation is bound to the macro function we expect, + # and we don't care about the name (because in `mcpyrate`, macros can be as-imported). + test[is_unexpanded_expr_macro(dummymacro, m, q[dummy[...]])] # noqa: F821, only quoted + test[not is_unexpanded_expr_macro(dummymacro, m, q[notdummy[...]])] # noqa: F821, only quoted test[not is_unexpanded_expr_macro(dummymacro, m, q[42])] with testset("is_unexpanded_block_macro"): From 653f7f03e48e535a3cb721c5c1034c47111429a7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 15:33:13 +0300 Subject: [PATCH 180/832] add missing top-level imports --- unpythonic/__init__.py | 1 + 1 file changed, 1 insertion(+) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 7e1e4db6..7ac9df39 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -47,3 +47,4 @@ from .lazyutil import _init_module _init_module() del _init_module +from .lazyutil import Lazy, force1, force # noqa: F401 From 5c07c4887dc7c74d66d221ba5623bd268e0550f6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 15:33:50 +0300 Subject: [PATCH 181/832] fix exports --- unpythonic/lazyutil.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpythonic/lazyutil.py b/unpythonic/lazyutil.py index 22f3e866..75b654dd 100644 --- a/unpythonic/lazyutil.py +++ b/unpythonic/lazyutil.py @@ -5,7 +5,8 @@ upon which other regular code is allowed to depend. """ -__all__ = ["passthrough_lazy_args", "maybe_force_args", "force1", "force"] +__all__ = ["Lazy", "force1", "force", # intended also for end-users + "islazy", "maybe_force_args", "passthrough_lazy_args"] # mostly for use inside `unpythonic` from .regutil import register_decorator from .dynassign import make_dynvar From 7e90626700b6759dcede87ed1f43045a09eeb695 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 15:34:23 +0300 Subject: [PATCH 182/832] use star-import re-exports for public macro API This unifies the re-export style between `unpythonic` and `unpythonic.syntax`. Also, move the functions `force` and `force1` to the top-level `unpythonic` package. (They actually come from `unpythonic.lazyutil`.) --- unpythonic/syntax/__init__.py | 42 ++++++++------------------ unpythonic/syntax/testingtools.py | 1 + unpythonic/syntax/tests/test_lazify.py | 3 +- 3 files changed, 15 insertions(+), 31 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 4d4f010b..7f55b157 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -87,36 +87,20 @@ # TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. -# Re-exports - macro interfaces -from .autocurry import autocurry # noqa: F401 -from .autoref import autoref # noqa: F401 -from .dbg import dbg # noqa: F401 -from .forall import forall # noqa: F401 -from .ifexprs import aif, it, cond # noqa: F401 -from .lambdatools import multilambda, namedlambda, f, quicklambda, envify # noqa: F401 -from .lazify import lazy, lazyrec, lazify # noqa: F401 -from .letdo import (let, letseq, letrec, # noqa: F401 - dlet, dletseq, dletrec, - blet, bletseq, bletrec, - local, delete, do, do0) -from .letsyntax import let_syntax, abbrev # noqa: F401 -from .nb import nb # noqa: F401 -from .prefix import prefix # noqa: F401 -from .tailtools import (autoreturn, # noqa: F401 - tco, - continuations, call_cc) -from .testingtools import (the, test, # noqa: F401 - test_signals, test_raises, - fail, error, warn, - expand_testing_macros_first) - -# Re-exports - regular code -from .dbg import dbgprint_block, dbgprint_expr # noqa: F401, re-export for re-use in a decorated variant. -from .forall import insist, deny # noqa: F401 +from .autocurry import * # noqa: F401, F403 +from .autoref import * # noqa: F401, F403 +from .dbg import * # noqa: F401, F403 +from .forall import * # noqa: F401, F403 +from .ifexprs import * # noqa: F401, F403 +from .lambdatools import * # noqa: F401, F403 +from .lazify import * # noqa: F401, F403 +from .letdo import * # noqa: F401, F403 from .letdoutil import where # noqa: F401 -from .lazify import force, force1 # noqa: F401 -from .letsyntax import block, expr # noqa: F401 -from .prefix import q, u, kw # noqa: F401 # TODO: bad names, `mcpyrate` uses them too. +from .letsyntax import * # noqa: F401, F403 +from .nb import * # noqa: F401, F403 +from .prefix import * # noqa: F401, F403 +from .tailtools import * # noqa: F401, F403 +from .testingtools import * # noqa: F401, F403 # We use `dyn` to pass the `expander` parameter to the macro implementations. class _NoExpander: diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 5740c12c..8d634bc9 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -8,6 +8,7 @@ "test_signals", "test_raises", "fail", "error", "warn", "expand_testing_macros_first", + # these are mostly for use in other parts of `unpythonic` "isunexpandedtestmacro", "isexpandedtestmacro", "istestmacro"] from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index f4f1fc35..6cfa8ead 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -9,7 +9,6 @@ tco, autocurry, continuations, call_cc) -from ...syntax import force, force1 # Doesn't really override the earlier curry import. The first one is a macro, # and this one is a regular run-time function. @@ -22,7 +21,7 @@ from ...misc import raisef, call, callwith from ...tco import trampolined, jump -from ...lazyutil import islazy, Lazy # Lazy usually not needed in client code; for our tests only +from ...lazyutil import islazy, Lazy, force1, force # Lazy usually not needed in client code; for our tests only from sys import stderr import gc From 584611ce94d34af836ce78ed479c91fa4f3eb537 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 15:36:27 +0300 Subject: [PATCH 183/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7f55b157..2a879291 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,6 +69,10 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! +# TODO: Move `where` from letdoutil to letdo, make it a @namemacro, and declare it public. + +# TODO: Consider reversing the MRO of @generic (to latest first), since the point is to be extensible. + # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. From c0b6f2911f47e7c4e239e7db688556cb60ade0e5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 15:40:21 +0300 Subject: [PATCH 184/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4534dbcf..cc0d98ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,6 +80,8 @@ For future plans, see our [Python language version support status](https://githu - Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). +- Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From 25b8920eecb07732aa3413d26fd56bf72d149a30 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 15:45:26 +0300 Subject: [PATCH 185/832] update changelog --- CHANGELOG.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cc0d98ee..d242d7c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -**0.15.0** (in progress; updated 23 April 2021) - *The very latest future obsolete* edition: +**0.15.0** (in progress; updated 5 May 2021) - *The very latest future obsolete* edition: This edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. @@ -42,7 +42,7 @@ For future plans, see our [Python language version support status](https://githu **Breaking changes**: - Migrate to the `mcpyrate` macro expander; **MacroPy support dropped**. - - **Macro arguments are now passed using brackets** `macroname[args]` instead of parentheses. + - **Macro arguments are now passed using brackets**, `macroname[args]`, instead of parentheses. - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. - `mcpyrate` should report test coverage for macro-using code correctly; no need for `# pragma: no cover` in block macro invocations or in quasiquoted code. @@ -78,7 +78,7 @@ For future plans, see our [Python language version support status](https://githu - Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. -- Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). +- Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). This fixes a `flake8` E741 warning, and is more descriptive. - Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) From 08b9c1604bb72628fdd825e3c2bf717ff9cbe958 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:02:25 +0300 Subject: [PATCH 186/832] polish the changelog --- CHANGELOG.md | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d242d7c3..04e380ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,29 +1,30 @@ -**0.15.0** (in progress; updated 5 May 2021) - *The very latest future obsolete* edition: +**0.15.0** (in progress; updated 5 May 2021) - *"We say 'howdy' around these parts"* edition: This edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. -**Minimum Python language version is now 3.6**. +**IMPORTANT**: -The optional **macro expander is now [`mcpyrate`](https://github.com/Technologicat/mcpyrate)**. +- **Minimum Python language version is now 3.6**. +- The optional **macro expander is now [`mcpyrate`](https://github.com/Technologicat/mcpyrate)**. +- For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. -The same applies if you need the macro parts of `unpythonic` in your own project that uses MacroPy. Version 0.14.3 of `unpythonic` works up to Python 3.7. - -For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). +The same applies if you need the macro parts of `unpythonic` (i.e. import anything from `unpythonic.syntax`) in your own project that uses MacroPy. Version 0.14.3 of `unpythonic` works up to Python 3.7. **New**: -- **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) together with a kitchen-sink language extension macro package such as `unpythonic`. +- **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) (i.e. full-module code transformer) together with a kitchen-sink language extension macro package such as `unpythonic`. + - These dialects have been moved from the now-obsolete [`pydialect`](https://github.com/Technologicat/pydialect) project and ported to use [`mcpyrate`](https://github.com/Technologicat/mcpyrate). -- `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) +- **Python 3.8 and 3.9 support added.** -- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), `with expr`/`with block` (for `let_syntax`/`abbrev`), and `q`/`u`/`kw` (for `prefix`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. That is, the error is now raised *at compile time*. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. +- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), `with expr`/`with block` (for `with let_syntax`/`with abbrev`), and `q`/`u`/`kw` (for `with prefix`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. -- `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. +- `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) -- Python 3.8 and 3.9 support added. +- Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. **Non-breaking changes**: @@ -57,16 +58,6 @@ For future plans, see our [Python language version support status](https://githu - **Rename the `curry` macro** to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. -- The `do[]` and `do0[]` macros now expand outside-in. The main differences from a user perspective are: - - Any source code captures (such as those performed by `test[]`) show the expanded output of `do` and `do0`, because that's what they receive. - - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. - -- Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. - - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) - - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. - -- The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - - As promised, names deprecated during 0.14.x have been removed. Old name on the left, new name on the right: - `m` → `imathify` - `mg` → `gmathify` @@ -78,10 +69,20 @@ For future plans, see our [Python language version support status](https://githu - Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. -- Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). This fixes a `flake8` E741 warning, and is more descriptive. +- Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). This fixes a `flake8` E741 warning, as well as is more descriptive. - Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) +- The `do[]` and `do0[]` macros now expand outside-in. The main differences from a user perspective are: + - Any source code captures (such as those performed by `test[]`) show the expanded output of `do` and `do0`, because that's what they receive. + - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. + +- The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + +- Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. + - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) + - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From 4723be40dbf4a299fc648f32b45d535da83f9873 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:02:38 +0300 Subject: [PATCH 187/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 2a879291..7185dd62 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -73,6 +73,8 @@ # TODO: Consider reversing the MRO of @generic (to latest first), since the point is to be extensible. +# TODO: Add docs navigation to all documentation files, like `mcpyrate` has. + # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. From 325800f055085dea466e96dd89c9f826fb18e566 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:14:31 +0300 Subject: [PATCH 188/832] use the xmas tree block macro ordering --- unpythonic/dialects/lispython.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index 2743107b..eeb7ef7c 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -38,7 +38,7 @@ def transform_ast(self, tree): # tree is an ast.Module # Auxiliary syntax elements for the macros. from unpythonic.syntax import where, block, expr # noqa: F401, F811 from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn # noqa: F401, F811 - with namedlambda, autoreturn, quicklambda, multilambda, tco: + with autoreturn, quicklambda, multilambda, tco, namedlambda: __paste_here__ # noqa: F821, just a splicing marker. tree.body = splice_dialect(tree.body, template, "__paste_here__") return tree From 2ab16d67116dfc14b4857bf4899d766c2ab45e0f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:17:53 +0300 Subject: [PATCH 189/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7185dd62..e7d81845 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -71,6 +71,8 @@ # TODO: Move `where` from letdoutil to letdo, make it a @namemacro, and declare it public. +# TODO: Emit a `do[]` in `multilambda`. + # TODO: Consider reversing the MRO of @generic (to latest first), since the point is to be extensible. # TODO: Add docs navigation to all documentation files, like `mcpyrate` has. From 2aa96b86bb5f03bb538e9e4c0f897a80c4e2f47a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:42:49 +0300 Subject: [PATCH 190/832] multilambda: emit do[] --- unpythonic/syntax/lambdatools.py | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 25c45913..27b91f42 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -26,7 +26,7 @@ from ..env import env from .astcompat import getconstant, Str, NamedExpr -from .letdo import _do +from .letdo import _implicit_do, _do from .letdoutil import islet, isenvassign, UnexpandedLetView, UnexpandedEnvAssignView, ExpandedDoView from .nameutil import getname from .util import (is_decorated_lambda, isx, has_deco, @@ -241,19 +241,9 @@ class MultilambdaTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): return tree # don't recurse! - if not (type(tree) is Lambda and type(tree.body) is List): - return self.generic_visit(tree) - bodys = tree.body - # bracket magic: - # - don't recurse to the implicit lambdas generated by the "do" we are inserting here - # - for each item, "do" internally inserts a lambda to delay execution, - # as well as to bind the environment - # - we must do() instead of q[h[do][...]] for pickling reasons - # - but recurse manually into each *do item*; these are explicit - # user-provided code so we should transform them - bodys = self.visit(bodys) - tree.body = _do(bodys) # insert the do, with the implicit lambdas - return tree + if type(tree) is Lambda: + tree.body = _implicit_do(tree.body) + return self.generic_visit(tree) # multilambda should expand first before any let[], do[] et al. that happen # to be inside the block, to avoid misinterpreting implicit lambdas # generated by those constructs. From 8e1066503207c6fb3ea46b0df6f1b15362599d7a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:43:03 +0300 Subject: [PATCH 191/832] envify: use some more quasiquotes in implementation --- unpythonic/syntax/lambdatools.py | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 27b91f42..322961e4 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -7,9 +7,8 @@ "quicklambda", "envify"] -from ast import (Lambda, List, Name, Assign, Subscript, Call, FunctionDef, - AsyncFunctionDef, Attribute, keyword, Dict, Constant, arg, - copy_location) +from ast import (Lambda, Name, Assign, Subscript, Call, FunctionDef, + AsyncFunctionDef, Attribute, keyword, Dict, Constant, arg) from copy import deepcopy from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 @@ -454,9 +453,9 @@ def isourupdate(thecall): ename = gensym("e") theenv = q[h[_envify]()] theenv.keywords = kws - assignment = Assign(targets=[q[n[ename]]], - value=theenv) - assignment = copy_location(assignment, tree) + with q as quoted: + n[ename] = a[theenv] + assignment = quoted[0] tree.body.insert(0, assignment) elif type(tree) is Lambda and id(tree) in userlambdas: # We must in general inject a new do[] even if one is already there, @@ -465,15 +464,15 @@ def isourupdate(thecall): # the name should revert to mean the formal parameter. # # inject a do[] and reuse its env - tree.body = _do(List(elts=[q[n["_here_"]], - tree.body])) + tree.body = _do(q[n["_here_"], + a[tree.body]]) view = ExpandedDoView(tree.body) # view.body: [(lambda e14: ...), ...] ename = view.body[0].args.args[0].arg # do[] environment name - theupdate = Attribute(value=q[n[ename]], attr="update") + theupdate = q[n[f"{ename}.update"]] thecall = q[a[theupdate]()] thecall.keywords = kws tree.body = splice_expression(thecall, tree.body, "_here_") - newbindings.update({k: Attribute(value=q[n[ename]], attr=k) for k in argnames}) # "x" --> e.x + newbindings.update({k: q[n[f"{ename}.{k}"]] for k in argnames}) # "x" --> e.x self.generic_withstate(tree, enames=(enames + [ename]), bindings=newbindings) else: # leave alone the _envify() added by us @@ -484,7 +483,9 @@ def isourupdate(thecall): elif isenvassign(tree): view = UnexpandedEnvAssignView(tree) if view.name in bindings.keys(): - envset = Attribute(value=bindings[view.name].value, attr="set") + # Grab the envname from the actual binding of "varname", of the form `e.varname` + # (so it's the `id` of a `Name` that is the `value` of an `Attribute`). + envset = q[n[f"{bindings[view.name].value.id}.set"]] newvalue = self.visit(view.value) return q[a[envset](u[view.name], a[newvalue])] # transform references to currently active bindings From 9817e0607855bde00227ea68cb887366c5086263 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 5 May 2021 16:44:26 +0300 Subject: [PATCH 192/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index e7d81845..7185dd62 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -71,8 +71,6 @@ # TODO: Move `where` from letdoutil to letdo, make it a @namemacro, and declare it public. -# TODO: Emit a `do[]` in `multilambda`. - # TODO: Consider reversing the MRO of @generic (to latest first), since the point is to be extensible. # TODO: Add docs navigation to all documentation files, like `mcpyrate` has. From cec70095ad0324a8f2c74c5752b5c27bbecaaa4f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 16:26:09 +0300 Subject: [PATCH 193/832] Make `where` (for let-where) error out in incorrect position --- CHANGELOG.md | 10 +++++++++- unpythonic/dialects/lispython.py | 3 ++- unpythonic/dialects/pytkell.py | 3 ++- unpythonic/dialects/tests/test_listhell.py | 3 +-- unpythonic/syntax/__init__.py | 3 --- unpythonic/syntax/letdo.py | 21 +++++++++++++++++++-- unpythonic/syntax/letdoutil.py | 7 +------ 7 files changed, 34 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 04e380ce..f3ce75ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,15 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Python 3.8 and 3.9 support added.** -- Robustness: several auxiliary syntactic constructs such as `local[]`/`delete[]` (for `do[]`), `call_cc[]` (for `with continuations`), `it` (for `aif[]`), `with expr`/`with block` (for `with let_syntax`/`with abbrev`), and `q`/`u`/`kw` (for `with prefix`) now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. +- Robustness: several auxiliary syntactic constructs now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. + - The full list is: + - `call_cc[]`, for `with continuations` + - `it`, for `aif[]` + - `local[]`/`delete[]`, for `do[]` + - `q`/`u`/`kw`, for `with prefix` + - `where`, for `let[body, where(k0=v0, ...)]` (also for `letseq`, `letrec`, `let_syntax`, `abbrev`) + - `with expr`/`with block`, for `with let_syntax`/`with abbrev` + - Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index eeb7ef7c..f114d678 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -29,6 +29,7 @@ def transform_ast(self, tree): # tree is an ast.Module __lang__ = "Lispython" # noqa: F841, just provide it to user code. from unpythonic.syntax import (macros, tco, autoreturn, # noqa: F401, F811 multilambda, quicklambda, namedlambda, f, + where, let, letseq, letrec, dlet, dletseq, dletrec, blet, bletseq, bletrec, @@ -36,7 +37,7 @@ def transform_ast(self, tree): # tree is an ast.Module let_syntax, abbrev, cond) # Auxiliary syntax elements for the macros. - from unpythonic.syntax import where, block, expr # noqa: F401, F811 + from unpythonic.syntax import block, expr # noqa: F401, F811 from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn # noqa: F401, F811 with autoreturn, quicklambda, multilambda, tco, namedlambda: __paste_here__ # noqa: F821, just a splicing marker. diff --git a/unpythonic/dialects/pytkell.py b/unpythonic/dialects/pytkell.py index cfcf3cc8..d676388a 100644 --- a/unpythonic/dialects/pytkell.py +++ b/unpythonic/dialects/pytkell.py @@ -18,13 +18,14 @@ def transform_ast(self, tree): # tree is an ast.Module with q as template: __lang__ = "Pytkell" # noqa: F841, just provide it to user code. from unpythonic.syntax import (macros, lazy, lazyrec, lazify, autocurry, # noqa: F401, F811 + where, let, letseq, letrec, dlet, dletseq, dletrec, blet, bletseq, bletrec, local, delete, do, do0, cond, forall) # Auxiliary syntax elements for the macros. - from unpythonic.syntax import where, insist, deny # noqa: F401 + from unpythonic.syntax import insist, deny # noqa: F401 # Functions that have a haskelly feel to them. from unpythonic import (foldl, foldr, scanl, scanr, # noqa: F401 s, imathify, gmathify, frozendict, diff --git a/unpythonic/dialects/tests/test_listhell.py b/unpythonic/dialects/tests/test_listhell.py index ded37944..ce2a0cb9 100644 --- a/unpythonic/dialects/tests/test_listhell.py +++ b/unpythonic/dialects/tests/test_listhell.py @@ -7,8 +7,7 @@ from ...syntax import macros, test # noqa: F401 from ...test.fixtures import session, testset -from ...syntax import macros, let, local, delete, do # noqa: F401, F811 -from ...syntax import where # for let-where # noqa: F401 +from ...syntax import macros, let, where, local, delete, do # noqa: F401, F811 from unpythonic import foldr, cons, nil, ll def runtests(): diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7185dd62..37750ff8 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,8 +69,6 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: Move `where` from letdoutil to letdo, make it a @namemacro, and declare it public. - # TODO: Consider reversing the MRO of @generic (to latest first), since the point is to be extensible. # TODO: Add docs navigation to all documentation files, like `mcpyrate` has. @@ -101,7 +99,6 @@ from .lambdatools import * # noqa: F401, F403 from .lazify import * # noqa: F401, F403 from .letdo import * # noqa: F401, F403 -from .letdoutil import where # noqa: F401 from .letsyntax import * # noqa: F401, F403 from .nb import * # noqa: F401, F403 from .prefix import * # noqa: F401, F403 diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 9be601ea..c50acf83 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -1,7 +1,8 @@ # -*- coding: utf-8 -*- """Local bindings (let), imperative code in expression position (do).""" -__all__ = ["let", "letseq", "letrec", +__all__ = ["where", + "let", "letseq", "letrec", "dlet", "dletseq", "dletrec", "blet", "bletseq", "bletrec", "local", "delete", "do", "do0"] @@ -30,7 +31,7 @@ from mcpyrate.quotes import macros, q, u, n, a, t, h # noqa: F401 -from mcpyrate import gensym, parametricmacro +from mcpyrate import gensym, namemacro, parametricmacro from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.walkers import ASTTransformer, ASTVisitor @@ -85,6 +86,22 @@ def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, allo # -------------------------------------------------------------------------------- # Macro interface - expr macros +@namemacro +def where(tree, *, syntax, **kw): + """[syntax, special] `where` operator for let. + + Usage:: + + let[body, where((k0, v0), ...)] + + Only meaningful for declaring the bindings in a let-where, for all + expression-form let constructs: `let`, `letseq`, `letrec`, `let_syntax`, + `abbrev`. + """ + if syntax != "name": + raise SyntaxError("where (unpythonic.syntax.letdo.where) is a name macro only") # pragma: no cover + raise SyntaxError("where() is only meaningful in a let[body, where((k0, v0), ...)]") # pragma: no cover + @parametricmacro def let(tree, *, args, syntax, expander, **kw): """[syntax, expr] Introduce expression-local variables. diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index b442ec4c..a674dea9 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -1,8 +1,7 @@ # -*- coding: utf-8 -*- """Detect let and do forms, and destructure them writably.""" -__all__ = ["where", - "canonize_bindings", # used by the macro interface layer +__all__ = ["canonize_bindings", # used by the macro interface layer "isenvassign", "islet", "isdo", "UnexpandedEnvAssignView", "UnexpandedLetView", "UnexpandedDoView", "ExpandedLetView", "ExpandedDoView"] @@ -16,10 +15,6 @@ from .astcompat import getconstant, Str from .nameutil import isx, getname -def where(*bindings): - """[syntax] Only meaningful in a let[body, where((k0, v0), ...)].""" - raise RuntimeError("where() is only meaningful in a let[body, where((k0, v0), ...)]") # pragma: no cover - letf_name = "letter" # must match what ``unpythonic.syntax.letdo._let_expr_impl`` uses in its output. dof_name = "dof" # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. currycall_name = "currycall" # output of ``unpythonic.syntax.curry`` From e9b6e0c3d1afdecfcc1ea12c77edc1ca278d55e7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 16:27:45 +0300 Subject: [PATCH 194/832] fix imports: `block` and `expr` are macros --- unpythonic/dialects/lispython.py | 4 +--- unpythonic/syntax/tests/test_conts_gen.py | 3 +-- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index f114d678..6949db6b 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -34,10 +34,8 @@ def transform_ast(self, tree): # tree is an ast.Module dlet, dletseq, dletrec, blet, bletseq, bletrec, local, delete, do, do0, - let_syntax, abbrev, + let_syntax, abbrev, block, expr, cond) - # Auxiliary syntax elements for the macros. - from unpythonic.syntax import block, expr # noqa: F401, F811 from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn # noqa: F401, F811 with autoreturn, quicklambda, multilambda, tco, namedlambda: __paste_here__ # noqa: F821, just a splicing marker. diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index af582ac9..b6739a8a 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -21,8 +21,7 @@ from ...syntax import macros, test # noqa: F401 from ...test.fixtures import session, testset -from ...syntax import macros, continuations, call_cc, dlet, abbrev, let_syntax # noqa: F401, F811 -from ...syntax import block +from ...syntax import macros, continuations, call_cc, dlet, abbrev, let_syntax, block # noqa: F401, F811 from ...fploop import looped from ...fun import identity From ef6089473138739af04e783c4fbc00b9ae5b0f76 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 16:27:59 +0300 Subject: [PATCH 195/832] improve docstring of `kw()` --- unpythonic/syntax/prefix.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 8edff682..6f841d7c 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -129,7 +129,14 @@ def u(tree, *, syntax, **kw): # noqa: F811 # TODO: We currently trigger the error on any appearance of the name `kw` outside a valid context. @namemacro def kw(tree, *, syntax, **kw): # noqa: F811 - """[syntax] Pass-named-args operator. Only meaningful in a tuple inside a prefix block.""" + """[syntax, special] Pass-named-args operator for `with prefix`. + + Usage:: + + (f, a0, ..., kw(k0=v0, ...)) + + Only meaningful in a tuple inside a prefix block. + """ if syntax != "name": raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is a name macro only") # pragma: no cover raise SyntaxError("kw (unpythonic.syntax.prefix.kw) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander From 62990bab37d8e2eebbc5f008af2f31e4a51feda9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 16:41:58 +0300 Subject: [PATCH 196/832] The @generic MRO was already correct, document it. --- doc/features.md | 2 ++ unpythonic/syntax/__init__.py | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index 3c05361e..8f2f7b7c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2998,6 +2998,8 @@ For what kind of things can be done with this, see particularly the [*holy trait The ``generic`` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. +If several methods of the same generic function match the arguments given, the most recently registered method wins. (**CAUTION**: This is different from Julia, where the most specific method wins. Doing that requires a more careful type analysis than what we have here.) + The details are best explained by example: ```python diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 37750ff8..021d33bd 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,8 +69,6 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: Consider reversing the MRO of @generic (to latest first), since the point is to be extensible. - # TODO: Add docs navigation to all documentation files, like `mcpyrate` has. # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? From 313a129d4d488d8acf1bd0f32f10fbed50fa8f54 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 17:03:38 +0300 Subject: [PATCH 197/832] add docs navigation to all documentation files --- CONTRIBUTING.md | 21 ++++++++++ README.md | 16 ++++---- doc/design-notes.md | 72 +++++++++++++++++++++-------------- doc/dialects.md | 22 +++++++++++ doc/dialects/lispython.md | 47 ++++++++++++++++++----- doc/dialects/listhell.md | 41 ++++++++++++++++---- doc/dialects/pytkell.md | 38 +++++++++++++++--- doc/features.md | 11 ++++++ doc/macros.md | 11 ++++++ doc/readings.md | 21 +++++++++- doc/repl.md | 29 ++++++++++++++ unpythonic/syntax/__init__.py | 2 - 12 files changed, 270 insertions(+), 61 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 69a0a881..c48bc92b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,24 @@ +**Navigation** + +- [README](README.md) +- [Pure-Python feature set](doc/features.md) +- [Syntactic macro feature set](doc/macros.md) +- [Examples of creating dialects using `mcpyrate`](doc/dialects.md) +- [REPL server](doc/repl.md) +- [Design notes](doc/design-notes.md) +- [Additional reading](doc/readings.md) +- **Contribution guidelines** + + +**Table of Contents** + +- [Hacking unpythonic, a.k.a. contribution guidelines](#hacking-unpythonic-aka-contribution-guidelines) + - [Most importantly](#most-importantly) + - [Technical overview](#technical-overview) + - [Style guide](#style-guide) + + + # Hacking unpythonic, a.k.a. contribution guidelines **Rule #1**: Code and/or documentation contributions are welcome! diff --git a/README.md b/README.md index 00819446..ecbecabd 100644 --- a/README.md +++ b/README.md @@ -35,19 +35,21 @@ The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (langua ### Documentation -[Pure-Python feature set](doc/features.md) -[Syntactic macro feature set](doc/macros.md) -[Examples of creating dialects using `mcpyrate`](doc/dialects.md): Python the way you want it. -[REPL server](doc/repl.md): interactively hot-patch your running Python program. -[Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. -[Contribution guidelines](CONTRIBUTING.md): for understanding the codebase, or if you're interested in making a code or documentation PR. +- **README**: you are here. +- [Pure-Python feature set](doc/features.md) +- [Syntactic macro feature set](doc/macros.md) +- [Examples of creating dialects using `mcpyrate`](doc/dialects.md): Python the way you want it. +- [REPL server](doc/repl.md): interactively hot-patch your running Python program. +- [Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. +- [Additional reading](doc/readings.md): links to material relevant in the context of ``unpythonic``. +- [Contribution guidelines](CONTRIBUTING.md): for understanding the codebase, or if you're interested in making a code or documentation PR. The features of `unpythonic` are built out of, in increasing order of [magic](https://macropy3.readthedocs.io/en/latest/discussion.html#levels-of-magic): - Pure Python (e.g. batteries for `itertools`), - Macros driving a pure-Python core (`do`, `let`), - Pure macros (e.g. `continuations`, `lazify`, `dbg`). - - Whole-module transformations a.k.a. dialects. + - Whole-module transformations, a.k.a. dialects. This depends on the purpose of each feature, as well as ease-of-use considerations. See the design notes for more information. diff --git a/doc/design-notes.md b/doc/design-notes.md index 130c410a..49391c70 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -1,21 +1,35 @@ -# Design Notes +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- [Syntactic macro feature set](macros.md) +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- [REPL server](repl.md) +- **Design notes** +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + + +**Table of Contents** - [Design Philosophy](#design-philosophy) -- [Macros do not Compose](#macros-do-not-compose) -- [Language Discontinuities](#language-discontinuities) -- [What Belongs in Python?](#what-belongs-in-python) -- [Killer features of Common Lisp](#killer-features-of-common-lisp) -- [Common Lisp, Python, and productivity](#common-lisp-python-and-productivity) -- [Python is not a Lisp](#python-is-not-a-lisp) -- [On ``let`` and Python](#on-let-and-python) -- [Assignment Syntax](#assignment-syntax) -- [TCO Syntax and Speed](#tco-syntax-and-speed) -- [No Monads?](#no-monads) -- [No Types?](#no-types) -- [Detailed Notes on Macros](#detailed-notes-on-macros) -- [Miscellaneous notes](#miscellaneous-notes) - -## Design Philosophy + - [Macros do not Compose](#macros-do-not-compose) + - [Language Discontinuities](#language-discontinuities) + - [What Belongs in Python?](#what-belongs-in-python) + - [Killer features of Common Lisp](#killer-features-of-common-lisp) + - [Common Lisp, Python, and productivity](#common-lisp-python-and-productivity) + - [Python is not a Lisp](#python-is-not-a-lisp) + - [On ``let`` and Python](#on-let-and-python) + - [Assignment syntax](#assignment-syntax) + - [TCO syntax and speed](#tco-syntax-and-speed) + - [No Monads?](#no-monads) + - [No Types?](#no-types) + - [Detailed Notes on Macros](#detailed-notes-on-macros) + - [Miscellaneous notes](#miscellaneous-notes) + + + +# Design Philosophy The main design considerations of `unpythonic` are simplicity, robustness, and minimal dependencies. Some complexity is tolerated, if it is essential to make features interact better, or to provide a better user experience. @@ -37,7 +51,7 @@ Finally, when the whole purpose of the feature is to automatically transform a p When to implement your own feature as a syntactic macro, see the discussion in Chapter 8 of [Paul Graham: On Lisp](http://paulgraham.com/onlisp.html). MacroPy's documentation also provides [some advice on the topic](https://macropy3.readthedocs.io/en/latest/discussion.html). -### Macros do not Compose +## Macros do not Compose Making macros work together is nontrivial, essentially because *macros don't compose*. [As pointed out by John Shutt](https://fexpr.blogspot.com/2013/12/abstractive-power.html), in a multilayered language extension implemented with macros, the second layer of macros needs to understand all of the first layer. The issue is that the macro abstraction leaks the details of its expansion. Contrast with functions, which operate on values: the process that was used to arrive at a value doesn't matter. It's always possible for a function to take this value and transform it into another value, which can then be used as input for the next layer of functions. That's composability at its finest. @@ -49,7 +63,7 @@ Some aspects in the design of `unpythonic` could be simplified by expanding macr The lack of composability is a problem mainly when using macros to create a language extension, because the features of the extended language often interact. Macros can also be used in a much more everyday way, where composability is mostly a non-issue - to abstract and name common patterns that just happen to be of a nature that cannot be extracted as a regular function. See [Peter Seibel: Practical Common Lisp, chapter 3](http://www.gigamonkeys.com/book/practical-a-simple-database.html) for an example. -### Language Discontinuities +## Language Discontinuities The very act of extending a language creates points of discontinuity between the extended language and the original. This can become a particularly bad source of extra complexity, if the extension can be enabled locally for a piece of code - as is the case with block macros. Then the design of the extended language must consider how to treat interactions between pieces of code that use the extension and those that don't. Then exponentiate those design considerations by the number of extensions that can be enabled independently. This issue is simply absent when designing a new language from scratch. @@ -59,7 +73,7 @@ For another example, it's likely that e.g. `continuations` still doesn't integra For a third example, consider *decorated lambdas*. This is an `unpythonic` extension - essentially, a compiler feature implemented (by calling some common utility code) by each of the transformers of the pure-macro features - that understands a lambda enclosed in a nested sequence of single-argument function calls *as a decorated function definition*. This is painful, because the Python AST has no place to store the decorator list for a lambda; Python sees it just as a nested sequence of function calls, terminating in a lambda. This has to be papered over by the transformers. We also introduce a related complication, the decorator registry (see `regutil`), so that we can automatically sort decorator invocations - so that pure-macro features know at which index to inject a particular decorator (so it works properly) when they need to do that. Needing such a registry is already a complication, but the *decorated lambda* machinery feels the pain more acutely. -### What Belongs in Python? +## What Belongs in Python? If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), it is because they come with the territory. @@ -71,7 +85,7 @@ On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=1473 It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multiple-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I don't want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) -### Killer features of Common Lisp +## Killer features of Common Lisp In my opinion, Common Lisp has three legendary killer features: @@ -101,7 +115,7 @@ But for those of us that [don't like parentheses](https://srfi.schemers.org/srfi - PyPy (the JIT-enabled Python interpreter) itself is not the full story; the [RPython](https://rpython.readthedocs.io/en/latest/) toolchain from the PyPy project can *automatically produce a JIT for an interpreter for any new dynamic language implemented in the RPython language* (which is essentially a restricted dialect of Python 2.7). Now **that's** higher-order magic if anything is. - For the use case of numerics specifically, instead of Python, [Julia](https://docs.julialang.org/en/v1/manual/methods/) may be a better fit for writing high-level, yet performant code. It's a spiritual heir of Common Lisp, Fortran, *and Python*. Compilation to efficient machine code, with the help of gradual typing and automatic type inference, is a design goal. -### Common Lisp, Python, and productivity +## Common Lisp, Python, and productivity The various essays by Paul Graham, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for Lisp. So how does the programming world look in that light now, 20 years later? @@ -121,7 +135,7 @@ Haskell aims at code-data equivalence from a third angle (memoized pure function Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world isn't that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead (without restarting the whole app at each change). Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. -### Python is not a Lisp +## Python is not a Lisp The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macros.md)) is to make Python lambdas slightly more useful - which was really the starting point for the whole `unpythonic` experiment. @@ -139,7 +153,7 @@ The oft-quoted single-expression limitation of the Python ``lambda`` is ultimate Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. -### On ``let`` and Python +## On ``let`` and Python Why no `let*`, as a function? In Python, name lookup always occurs at runtime. Python gives us no compile-time guarantees that no binding refers to a later one - in [Racket](http://racket-lang.org/), this guarantee is the main difference between `let*` and `letrec`. @@ -155,7 +169,7 @@ The [macro versions](macros.md) of the `let` constructs **are** lexically scoped Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). -### Assignment syntax +## Assignment syntax Why the clunky `e.set("foo", newval)` or `e << ("foo", newval)`, which do not directly mention `e.foo`? This is mainly because in Python, the language itself is not customizable. If we could define a new operator `e.foo newval` to transform to `e.set("foo", newval)`, this would be easily solved. @@ -174,7 +188,7 @@ If we later choose go this route nevertheless, `<<` is a better choice for the s The current solution for the assignment syntax issue is to use macros, to have both clean syntax at the use site and a relatively hackfree implementation. -### TCO syntax and speed +## TCO syntax and speed Benefits and costs of ``return jump(...)``: @@ -198,7 +212,7 @@ For other libraries bringing TCO to Python, see: - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. -### No Monads? +## No Monads? (Beside List inside ``forall``.) @@ -206,7 +220,7 @@ Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done else If you want to roll your own monads for whatever reason, there's [this silly hack](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/monads.py) that wasn't packaged into this; or just read Stephan Boyer's quick introduction [[part 1]](https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern) [[part 2]](https://www.stephanboyer.com/post/10/monads-part-2-impure-computations) [[super quick intro]](https://www.stephanboyer.com/post/83/super-quick-intro-to-monads) and figure it out, it's easy. (Until you get to `State` and `Reader`, where [this](http://brandon.si/code/the-state-monad-a-tutorial-for-the-confused/) and maybe [this](https://gaiustech.wordpress.com/2010/09/06/on-monads/) can be helpful.) -### No Types? +## No Types? The `unpythonic` project will likely remain untyped indefinitely, since I don't want to enter that particular marshland with things like `curry` and `with continuations`. It may be possible to gradually type some carefully selected parts - but that's currently not on [the roadmap](https://github.com/Technologicat/unpythonic/milestones). I'm not against it, if someone wants to contribute. @@ -235,7 +249,7 @@ More on type systems: - In physics, units as used for dimension analysis are essentially a form of static typing. - This has been discussed on LtU, see e.g. [[1]](http://lambda-the-ultimate.org/node/33) [[2]](http://lambda-the-ultimate.org/classic/message11877.html). -### Detailed Notes on Macros +## Detailed Notes on Macros - ``continuations`` and ``tco`` are mutually exclusive, since ``continuations`` already implies TCO. - However, the ``tco`` macro skips any ``with continuations`` blocks inside it, **for the specific reason** of allowing modules written in the [Lispython dialect](https://github.com/Technologicat/pydialect) (which implies TCO for the whole module) to use ``with continuations``. @@ -285,7 +299,7 @@ More on type systems: - When in doubt, use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. -### Miscellaneous notes +## Miscellaneous notes - [Nick Coghlan (2011): Traps for the unwary in Python's import system](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html). diff --git a/doc/dialects.md b/doc/dialects.md index 61c77fe1..88f3294b 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -1,3 +1,25 @@ +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- [Syntactic macro feature set](macros.md) +- **Examples of creating dialects using `mcpyrate`** + - [Lispython](dialects/lispython.md) + - [Listhell](dialects/listhell.md) + - [Pytkell](dialects/pytkell.md) +- [REPL server](repl.md) +- [Design notes](design-notes.md) +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + + +**Table of Contents** + +- [Examples of creating dialects using `mcpyrate`](#examples-of-creating-dialects-using-mcpyrate) + + + + # Examples of creating dialects using `mcpyrate` What if Python had automatic tail-call optimization and an implicit return statement? Look no further: diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index d9d27622..0183f058 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -1,4 +1,33 @@ -## Lispython: The love child of Python and Scheme +**Navigation** + +- [README](../../README.md) +- [Pure-Python feature set](../features.md) +- [Syntactic macro feature set](../macros.md) +- [Examples of creating dialects using `mcpyrate`](../dialects.md) + - **Lispython** + - [Listhell](listhell.md) + - [Pytkell](pytkell.md) +- [REPL server](../repl.md) +- [Design notes](../design-notes.md) +- [Additional reading](../readings.md) +- [Contribution guidelines](../../CONTRIBUTING.md) + + +**Table of Contents** + +- [Lispython: The love child of Python and Scheme](#lispython-the-love-child-of-python-and-scheme) + - [Features](#features) + - [What Lispython is](#what-lispython-is) + - [Comboability](#comboability) + - [Lispython and continuations (call/cc)](#lispython-and-continuations-callcc) + - [Why extend Python?](#why-extend-python) + - [PG's accumulator-generator puzzle](#pgs-accumulator-generator-puzzle) + - [CAUTION](#caution) + - [Etymology?](#etymology) + + + +# Lispython: The love child of Python and Scheme Python with automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas. @@ -40,7 +69,7 @@ assert cdr(c) == 2 assert ll(1, 2, 3) == llist((1, 2, 3)) ``` -### Features +## Features In terms of ``unpythonic.syntax``, we implicitly enable ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, and ``quicklambda`` for the whole module: @@ -67,7 +96,7 @@ The builtin ``do[]`` constructs are ``do`` and ``do0``. If you need more stuff, `unpythonic` is effectively the standard library of Lispython, on top of what Python itself already provides. -### What Lispython is +## What Lispython is Lispython is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.lispython`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_lispython.py). @@ -80,7 +109,7 @@ We take the approach of a relatively thin layer of macros (and underlying functi Performance is only a secondary concern; performance-critical parts fare better at the other end of [the wide spectrum](https://en.wikipedia.org/wiki/Wide-spectrum_language), with [Cython](http://cython.org/). Lispython is for [the remaining 80%](https://en.wikipedia.org/wiki/Pareto_principle), where the bottleneck is human developer time. -### Comboability +## Comboability The aforementioned block macros are enabled implicitly for the whole module; this is the essence of the Lispython dialect. Other block macros can still be invoked manually in the user code. @@ -97,7 +126,7 @@ Basically, any block macro that can be invoked *lexically inside* a ``with tco`` If you need e.g. a lazy Lispython, the way to do that is to make a copy of the dialect module, change the dialect template to import the ``lazify`` macro, and then include a ``with lazify`` in the appropriate position, outside the ``with namedlambda`` block. Other customizations can be made similarly. -### Lispython and continuations (call/cc) +## Lispython and continuations (call/cc) Just use ``with continuations`` from ``unpythonic.syntax`` where needed. See its documentation for usage. @@ -114,7 +143,7 @@ Lispython works with ``with continuations``, because: Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in factorial tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython happily auto-injects a ``return`` to whatever is the last statement in any particular function. -### Why extend Python? +## Why extend Python? [Racket](https://racket-lang.org/) is an excellent Lisp, especially with [sweet](https://docs.racket-lang.org/sweet/), sweet expressions [[1]](https://sourceforge.net/projects/readable/) [[2]](https://srfi.schemers.org/srfi-110/srfi-110.html) [[3]](https://srfi.schemers.org/srfi-105/srfi-105.html), not to mention extremely pythonic. The word is *rackety*; the syntax of the language comes with an air of Zen minimalism (as perhaps expected of a descendant of Scheme), but the focus on *batteries included* and understandability are remarkably similar to the pythonic ideal. Racket even has an IDE (DrRacket) and an equivalent of PyPI, and the documentation is simply stellar. @@ -125,7 +154,7 @@ In certain other respects, Python the base language leaves something to be desir Practicality beats purity ([ZoP §9](https://www.python.org/dev/peps/pep-0020/)): hence, fix the minor annoyances that would otherwise quickly add up, and reap the benefits of both worlds. If Python is software glue, Lispython is an additive that makes it flow better. -### PG's accumulator-generator puzzle +## PG's accumulator-generator puzzle The puzzle was posted by Paul Graham in 2002, in the essay [Revenge of the Nerds](http://paulgraham.com/icad.html). It asks to implement, in the shortest code possible, an accumulator-generator. The desired behavior is: @@ -180,12 +209,12 @@ with envify: ``envify`` is not part of the Lispython dialect definition, because this particular, perhaps rarely used, feature is not really worth a global performance hit whenever a function is entered. -### CAUTION +## CAUTION No instrumentation exists (or is even planned) for the Lispython layer; you'll have to use regular Python tooling to profile, debug, and such. The Lispython layer should be thin enough for this not to be a major problem in practice. -### Etymology? +## Etymology? *Lispython* is obviously made of two parts: Python, and... diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index 030da744..173ddf63 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -1,4 +1,31 @@ -## Listhell: It's not Lisp, it's not Python, it's not Haskell +**Navigation** + +- [README](../../README.md) +- [Pure-Python feature set](../features.md) +- [Syntactic macro feature set](../macros.md) +- [Examples of creating dialects using `mcpyrate`](../dialects.md) + - [Lispython](lispython.md) + - **Listhell** + - [Pytkell](pytkell.md) +- [REPL server](../repl.md) +- [Design notes](../design-notes.md) +- [Additional reading](../readings.md) +- [Contribution guidelines](../../CONTRIBUTING.md) + + +**Table of Contents** + +- [Listhell: It's not Lisp, it's not Python, it's not Haskell](#listhell-its-not-lisp-its-not-python-its-not-haskell) + - [Features](#features) + - [What Listhell is](#what-listhell-is) + - [Comboability](#comboability) + - [Notes](#notes) + - [CAUTION](#caution) + - [Etymology?](#etymology) + + + +# Listhell: It's not Lisp, it's not Python, it's not Haskell Python with prefix syntax for function calls, and automatic currying. @@ -16,7 +43,7 @@ my_map = lambda f: (foldr, (compose, cons, f), nil) assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ``` -### Features +## Features In terms of ``unpythonic.syntax``, we implicitly enable ``prefix`` and ``curry`` for the whole module. @@ -32,7 +59,7 @@ For detailed documentation of the language features, see [``unpythonic.syntax``] If you need more stuff, `unpythonic` is effectively the standard library of Listhell, on top of what Python itself already provides. -### What Listhell is +## What Listhell is Listhell is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.listhell`](../../unpythonic/dialects/listhell.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_listhell.py). @@ -41,20 +68,20 @@ Listhell is essentially a demonstration of how Python could look, if it had Lisp It's also a minimal example of how to make an AST-transforming dialect. -### Comboability +## Comboability Only outside-in macros that should expand after ``curry`` (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``curry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Listhell dialect. -### Notes +## Notes If you like the idea and want autocurry for a Lisp, try [spicy](https://github.com/Technologicat/spicy) for [Racket](https://racket-lang.org/). -### CAUTION +## CAUTION Not intended for serious use. -### Etymology? +## Etymology? Prefix syntax of **Lis**p, speed of Py**th**on, and readability of Hask**ell**, all in one. diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index cac0ef00..aa161b67 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -1,4 +1,30 @@ -## Pytkell: Because it's good to have a kell +**Navigation** + +- [README](../../README.md) +- [Pure-Python feature set](../features.md) +- [Syntactic macro feature set](../macros.md) +- [Examples of creating dialects using `mcpyrate`](../dialects.md) + - [Lispython](lispython.md) + - [Listhell](listhell.md) + - **Pytkell** +- [REPL server](../repl.md) +- [Design notes](../design-notes.md) +- [Additional reading](../readings.md) +- [Contribution guidelines](../../CONTRIBUTING.md) + + +**Table of Contents** + +- [Pytkell: Because it's good to have a kell](#pytkell-because-its-good-to-have-a-kell) + - [Features](#features) + - [What Pytkell is](#what-pytkell-is) + - [Comboability](#comboability) + - [CAUTION](#caution) + - [Etymology?](#etymology) + + + +# Pytkell: Because it's good to have a kell Python with automatic currying and implicitly lazy functions. @@ -40,7 +66,7 @@ x = let[2 * a, where(a, 21)] assert x == 42 ``` -### Features +## Features In terms of ``unpythonic.syntax``, we implicitly enable ``curry`` and ``lazify`` for the whole module. @@ -69,7 +95,7 @@ The builtin ``do[]`` constructs are ``do`` and ``do0``. If you need more stuff, `unpythonic` is effectively the standard library of Pytkell, on top of what Python itself already provides. -### What Pytkell is +## What Pytkell is Pytkell is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.pytkell`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_pytkell.py). @@ -78,20 +104,20 @@ Pytkell essentially makes Python feel slightly more haskelly. It's also a minimal example of how to make an AST-transforming dialect. -### Comboability +## Comboability **Not** comboable with most of the block macros in ``unpythonic.syntax``, because ``curry`` and ``lazify`` appear in the dialect template, hence at the lexically outermost position. Only outside-in macros that should expand after ``lazify`` has recorded its userlambdas (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``curry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Pytkell dialect. -### CAUTION +## CAUTION No instrumentation exists (or is even planned) for the Pytkell layer; you'll have to use regular Python tooling to profile, debug, and such. This layer is not quite as thin as Lispython's, but the dialect is not intended for serious use, either. -### Etymology? +## Etymology? The other obvious contraction, *Pyskell*, sounds like a serious programming language - or possibly the name of a fantasy airship - whereas *Pytkell* is obviously something quickly thrown together for system testing. diff --git a/doc/features.md b/doc/features.md index 8f2f7b7c..c5328c7d 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1,3 +1,14 @@ +**Navigation** + +- [README](../README.md) +- **Pure-Python feature set** +- [Syntactic macro feature set](macros.md) +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- [REPL server](repl.md) +- [Design notes](design-notes.md) +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + # Unpythonic: Python meets Lisp and Haskell This is the pure-Python API of `unpythonic`. Most features listed here need no macros, and are intended to be used directly. diff --git a/doc/macros.md b/doc/macros.md index f567aff8..e4e8ffdc 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1,3 +1,14 @@ +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- **Syntactic macro feature set** +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- [REPL server](repl.md) +- [Design notes](design-notes.md) +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + # Language extensions using ``unpythonic.syntax`` Our extensions to the Python language are built on [``mcpyrate``](https://github.com/Technologicat/mcpyrate), from the PyPI package [``mcpyrate``](https://pypi.org/project/mcpyrate/). diff --git a/doc/readings.md b/doc/readings.md index c3a09676..e58ddf3f 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -1,3 +1,22 @@ +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- [Syntactic macro feature set](macros.md) +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- [REPL server](repl.md) +- [Design notes](design-notes.md) +- **Additional reading** +- [Contribution guidelines](../CONTRIBUTING.md) + + +**Table of Contents** + +- [Links to relevant reading](#links-to-relevant-reading) +- [Python-related FP resources](#python-related-fp-resources) + + + # Links to relevant reading This document collects links to blog posts, online articles and actual scientific papers on topics at least somewhat relevant in the context of `unpythonic`. @@ -149,7 +168,7 @@ The common denominator is programming. Some relate to language design, some to c - We have a demonstration in [unpythonic.tests.test_dispatch](../unpythonic/tests/test_dispatch.py). -## Python-related FP resources +# Python-related FP resources Python clearly wants to be an impure-FP language. A decorator with arguments *is a curried closure* - how much more FP can you get? diff --git a/doc/repl.md b/doc/repl.md index 93d2a7fd..3d3c10e8 100644 --- a/doc/repl.md +++ b/doc/repl.md @@ -1,3 +1,32 @@ +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- [Syntactic macro feature set](macros.md) +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- **REPL server** +- [Design notes](design-notes.md) +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + + +**Table of Contents** + +- [The Unpythonic REPL server](#the-unpythonic-repl-server) + - [Try the server](#try-the-server) + - [Connect with the client](#connect-with-the-client) + - [Netcat compatibility](#netcat-compatibility) + - [Embed the server in your Python app](#embed-the-server-in-your-python-app) + - [SECURITY WARNING!](#security-warning) + - [Design for hot-patching](#design-for-hot-patching) + - [ZODB in 5 minutes](#zodb-in-5-minutes) + - [Why a custom REPL server/client](#why-a-custom-repl-serverclient) + - [Future directions](#future-directions) + - [Authentication and encryption](#authentication-and-encryption) + - [Note on macro-enabled consoles](#note-on-macro-enabled-consoles) + + + # The Unpythonic REPL server Hot-patch a running Python process! With **syntactic macros** in the [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop)! diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 021d33bd..46f94551 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,8 +69,6 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: Add docs navigation to all documentation files, like `mcpyrate` has. - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. From 0314156d12ba98317a8d087df22d0ad7a74e0f4c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 17:11:34 +0300 Subject: [PATCH 198/832] update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f3ce75ef..87cd31ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -72,6 +72,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `setescape` → `catch` - `escape` → `throw` - `getvalue`, `runpipe` → `exitpipe` (combined into one) + - **CAUTION**: Now `exitpipe` is an `unpythonic.symbol.sym` (like a Lisp symbol). This is not compatible with existing, pickled `exitpipe` instances; it used to be an instance of the class `Getvalue`, which has been removed. (There's not much reason to pickle an `exitpipe` instance, but we're mentioning this for the sake of completeness.) - Drop support for deprecated argument format for `raisef`. Now the usage is `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. From 7b35163a2f0f3ae046506907c72d0abe296f0faf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 6 May 2021 17:14:21 +0300 Subject: [PATCH 199/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 46f94551..7f9e6fdf 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,23 +69,27 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! +# TODO: Macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" + +# TODO: Check expansion order of several macros in the same `with` statement + # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. -# TODO: macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" - -# TODO: something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead +# TODO: Get rid of `UnpythonicExpandedMacroMarker`, use `ASTMarker`. We don't really need to keep markers +# TODO: until run time. This requires a hook in the expander for custom postprocessors, so we can auto-remove +# TODO: our AST markers when macro expansion of a module is complete. # TODO: Consider using run-time compiler access in macro tests, like `mcpyrate` itself does. This compartmentalizes testing so that the whole test module won't crash on a macro-expansion error. -# TODO: Change decorator macro invocations to use [] instead of () to pass macro arguments. Requires Python 3.9. +# TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. -# TODO: Check expansion order of several macros in the same `with` statement +# TODO: Something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead. # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... -# TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. +# TODO: Change decorator macro invocations to use [] instead of () to pass macro arguments. Requires Python 3.9. from .autocurry import * # noqa: F401, F403 from .autoref import * # noqa: F401, F403 From e0e7a2787b8c91da540b87f3b8e4cde411cc7286 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 01:20:09 +0300 Subject: [PATCH 200/832] Macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" --- doc/design-notes.md | 6 +- doc/macros.md | 4 +- unpythonic/syntax/__init__.py | 2 - unpythonic/syntax/autoref.py | 9 ++- unpythonic/syntax/letdoutil.py | 8 +-- unpythonic/syntax/tailtools.py | 2 +- unpythonic/syntax/testingtools.py | 61 +++++++++++---------- unpythonic/syntax/tests/test_conts_gen.py | 2 +- unpythonic/syntax/tests/test_lambdatools.py | 2 +- unpythonic/syntax/tests/test_letsyntax.py | 4 +- unpythonic/syntax/tests/test_prefix.py | 2 +- unpythonic/syntax/tests/test_tco.py | 4 +- unpythonic/syntax/util.py | 14 +++-- 13 files changed, 65 insertions(+), 55 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 49391c70..e751e601 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -254,7 +254,7 @@ More on type systems: - ``continuations`` and ``tco`` are mutually exclusive, since ``continuations`` already implies TCO. - However, the ``tco`` macro skips any ``with continuations`` blocks inside it, **for the specific reason** of allowing modules written in the [Lispython dialect](https://github.com/Technologicat/pydialect) (which implies TCO for the whole module) to use ``with continuations``. - - ``prefix``, ``autoreturn``, ``quicklambda`` and ``multilambda`` are first-pass macros (expand from outside in), because they change the semantics: + - ``prefix``, ``autoreturn``, ``quicklambda`` and ``multilambda`` expand outside-in, because they change the semantics: - ``prefix`` transforms things-that-look-like-tuples into function calls, - ``autoreturn`` adds ``return`` statements where there weren't any, - ``quicklambda`` transforms things-that-look-like-list-lookups into ``lambda`` function definitions, @@ -272,7 +272,7 @@ More on type systems: - This allows ``with tco`` to work together with the functions in ``unpythonic.fploop``, which imply TCO. - Macros that transform lambdas (notably ``continuations`` and ``tco``): - - Perform a first pass to take note of all lambdas that appear in the code *before the expansion of any inner macros*. Then in the second pass, *after the expansion of all inner macros*, only the recorded lambdas are transformed. + - Perform an outside-in pass to take note of all lambdas that appear in the code *before the expansion of any inner macros*. Then in an inside-out pass, *after the expansion of all inner macros*, only the recorded lambdas are transformed. - This mechanism distinguishes between explicit lambdas in the client code, and internal implicit lambdas automatically inserted by a macro. The latter are a technical detail that should not undergo the same transformations as user-written explicit lambdas. - The identification is based on the ``id`` of the AST node instance. Hence, if you plan to write your own macros that work together with those in ``unpythonic.syntax``, avoid going overboard with FP. Modifying the tree in-place, preserving the original AST node instances as far as sensible, is just fine. - For the interested reader, grep the source code for ``userlambdas``. @@ -285,7 +285,7 @@ More on type systems: - ``unpythonic.fploop.breakably_looped`` internally inserts the ``call_ec`` at the right step, and gives you the ec as ``brk``. - For the interested reader, look at ``unpythonic.syntax.util``. - - ``namedlambda`` is a two-pass macro. In the first pass (outside-in), it names lambdas inside ``let[]`` expressions before they are expanded away. The second pass (inside-out) of ``namedlambda`` must run after ``autocurry`` to analyze and transform the auto-curried code produced by ``with autocurry``. In most cases, placing ``namedlambda`` in a separate outer ``with`` block runs both operations in the correct order. + - ``namedlambda`` is a two-pass macro. In the outside-in pass, it names lambdas inside ``let[]`` expressions before they are expanded away. The inside-out pass of ``namedlambda`` must run after ``autocurry`` to analyze and transform the auto-curried code produced by ``with autocurry``. In most cases, placing ``namedlambda`` in a separate outer ``with`` block runs both operations in the correct order. - ``autoref`` does not need in its output to be curried (hence after ``curry`` to gain some performance), but needs to run before ``lazify``, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what ``namedlambda`` does, so it does not matter in which exact order these two run. diff --git a/doc/macros.md b/doc/macros.md index e4e8ffdc..7e5a8c0f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -452,7 +452,7 @@ When used as an expr macro, all bindings are registered first, and then the body #### `abbrev` -The ``abbrev`` macro is otherwise exactly like ``let_syntax``, but it expands in the first pass (outside in). Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the ``abbrev`` itself expands before any macros invoked in its body. This allows things like: +The ``abbrev`` macro is otherwise exactly like ``let_syntax``, but it expands outside-in. Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the ``abbrev`` itself expands before any macros invoked in its body. This allows things like: ```python abbrev[(m, macrowithverylongname)][ @@ -1830,7 +1830,7 @@ The `with test_raises[exctype]` and `with test_signals[exctype]` blocks assert t The point of `unpythonic.test.fixtures` is to make testing macro-enabled Python as frictionless as reasonably possible. -Inside a `test[]` expression, or anywhere within the code in a `with test` block, the `the[]` macro can be used to declare any number of subexpressions as interesting, for capturing the source code and value into the test failure message, which is shown if the test fails. Source code is captured in the first pass (outside in), before any nested second-pass (inside out) macros expand. (Most of the macros defined by `unpythonic` expand in the second pass.) The value is captured at run time as a side effect just after the value has been evaluated. +Inside a `test[]` expression, or anywhere within the code in a `with test` block, the `the[]` macro can be used to declare any number of subexpressions as interesting, for capturing the source code and value into the test failure message, which is shown if the test fails. Source code is captured in the outside-in pass, before any nested inside-out macros expand. (Many macros defined by `unpythonic` expand inside-out.) The value is captured at run time as a side effect just after the value has been evaluated. By default (if no explicit `the[]` is present), `test[]` implicitly inserts a `the[]` for the leftmost term if the top-level expression is a comparison (common use case), and otherwise does not capture anything. diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 7f9e6fdf..f8123a70 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,8 +69,6 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: Macro docs: "first pass" -> "outside in"; "second pass" -> "inside out" - # TODO: Check expansion order of several macros in the same `with` statement # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 6021c659..a36b0535 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -160,6 +160,7 @@ def _autoref_resolve(args): return False, None def _autoref(block_body, args, asname): + # first pass, outside-in if len(args) != 1: raise SyntaxError("expected exactly one argument, the expr to implicitly reference") # pragma: no cover if not block_body: @@ -167,6 +168,8 @@ def _autoref(block_body, args, asname): block_body = dyn._macro_expander.visit(block_body) + # second pass, inside-out + # `autoref`'s analyzer needs the `ctx` attributes in `tree` to be filled in correctly. block_body = fix_ctx(block_body, copy_seen_nodes=False) # TODO: or maybe copy seen nodes? @@ -292,9 +295,9 @@ def transform(self, tree): # Skip (by name) some common references inserted by other macros. # - # We are a second-pass macro (inside out), so any first-pass macro invocations, - # as well as any second-pass macro invocations inside the `with autoref` block, - # have already expanded by the time we run our transformer. + # This part runs in the inside-out pass, so any outside-in macro invocations, + # as well as any inside-out macro invocations inside the `with autoref` + # block, have already expanded by the time we run our transformer. always_skip = ['letter', 'dof', # let/do subsystem 'namelambda', # lambdatools subsystem 'curry', 'curryf' 'currycall', # autocurry subsystem diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index a674dea9..85416170 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -529,8 +529,8 @@ class ExpandedLetView: We support both "with autocurry" and bare formats. This is for simple in-place modifications; changing the number of bindings - is currently not supported. Prefer doing any extensive modifications in the - first pass, before the ``let[]`` expands. + is currently not supported. Prefer doing any extensive modifications + **before** the ``let[]`` expands. The bindings are contained in an `ast.Tuple`. Each binding is also an `ast.Tuple`. @@ -711,8 +711,8 @@ class ExpandedDoView: We support both "with autocurry" and bare formats. This is for simple in-place modifications; changing the number of do-items - is currently not supported. Prefer doing any extensive modifications in the - first pass, before the ``do[]`` expands. + is currently not supported. Prefer doing any extensive modifications + **before** the ``do[]`` expands. ``body`` is a ``list``, where each item is of the form ``lambda e: ...``. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index d1c8b6f6..aff82cd0 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -688,7 +688,7 @@ def transform_tailstmt(tree): elif type(tree) is Expr: tree = Return(value=tree.value) return tree - # This is a first-pass macro. Any nested macros should get clean standard Python, + # This macro expands outside-in. Any nested macros should get clean standard Python, # not having to worry about implicit "return" statements. return AutoreturnTransformer().visit(block_body) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 8d634bc9..f731ca07 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -807,6 +807,8 @@ def _warn_expr(tree): # Expr variants. def _test_expr(tree): + # first pass, outside-in + # Note we want the line number *before macro expansion*, so we capture it now. ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] @@ -822,9 +824,15 @@ def _test_expr(tree): # Before we edit the tree, get the source code in its pre-transformation # state, so we can include that into the test failure message. # - # We capture the source in the first pass, so that no macros in tree are - # expanded yet. For the same reason, we process the `the[]` marks in the - # first pass. + # We capture the source in the outside-in pass, so that no macros inside `tree` + # are expanded yet. For the same reason, we process the `the[]` marks in the + # outside-in pass. + # + # (Note, however, that if the `test[]` is nested within the invocation of + # a code-walking block macro, that macro may have performed edits already. + # For this reason, we provide `with expand_testing_macros_first`, which + # in itself is a code-walking block macro, whose only purpose is to force + # `test[]` and its sisters to expand first.) sourcecode = unparse(tree) envname = gensym("e") # for injecting the captured value @@ -834,9 +842,10 @@ def _test_expr(tree): if not the_exprs and type(tree) is Compare: # inject the implicit the[] on the LHS tree.left = _inject_value_recorder(envname, tree.left) - # End of first pass. tree = dyn._macro_expander.visit(tree) + # second pass, inside-out + # We delay the execution of the test expr using a lambda, so # `unpythonic_assert` can get control first before the expr runs. # @@ -906,6 +915,8 @@ def _test_expr_raises(tree): return _test_expr_signals_or_raises(tree, "test_raises", q[h[unpythonic_assert_raises]]) def _test_expr_signals_or_raises(tree, syntaxname, asserter): + # first pass, outside-in + ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] @@ -919,16 +930,13 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): else: raise SyntaxError(f"Expected one of {syntaxname}[exctype, expr], {syntaxname}[exctype, expr, message]") # pragma: no cover - # Before we edit the tree, get the source code in its pre-transformation - # state, so we can include that into the test failure message. - # - # We capture the source in the first pass, so that no macros in tree are - # expanded yet. + # Same remark about outside-in source code capture as in `_test_expr`. sourcecode = unparse(tree) - # End of first pass. tree = dyn._macro_expander.visit(tree) + # second pass, inside-out + return q[(a[asserter])(a[exctype], u[sourcecode], lambda: a[tree], @@ -942,6 +950,7 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): # The strategy is we capture the block body into a new function definition, # and then `unpythonic_assert` on that function. def _test_block(block_body, args): + # first pass, outside-in if not block_body: return [] # pragma: no cover, cannot happen through the public API. first_stmt = block_body[0] @@ -960,12 +969,7 @@ def _test_block(block_body, args): else: raise SyntaxError('Expected `with test:` or `with test[message]:`') # pragma: no cover - # Before we edit the tree, get the source code in its pre-transformation - # state, so we can include that into the test failure message. - # - # We capture the source in the first pass, so that no macros in tree are - # expanded yet. For the same reason, we process the `the[]` marks in the - # first pass. + # Same remark about outside-in source code capture as in `_test_expr`. sourcecode = unparse(block_body) envname = gensym("e") # for injecting the captured value @@ -973,9 +977,12 @@ def _test_block(block_body, args): # Handle the `the[...]` marks, if any. block_body, the_exprs = _transform_important_subexpr(block_body, envname=envname) - # End of first pass. block_body = dyn._macro_expander.visit(block_body) + # second pass, inside-out + + # Prepare the function template to be injected, and splice the contents + # of the `with test` block as the function body. testblock_function_name = gensym("_test_block") thetest = q[(a[asserter])(u[sourcecode], n[testblock_function_name], @@ -989,13 +996,14 @@ def _insert_funcname_here_(_insert_envname_here_): thefunc = newbody[0] thefunc.name = testblock_function_name thefunc.args.args[0] = arg(arg=envname) # inject the gensymmed parameter name + thefunc.body = block_body # Handle the return statement. # # We just check if there is at least one; if so, we don't need to do # anything; the returned value is what the test should return to the # asserter. - for stmt in block_body: + for stmt in thefunc.body: if type(stmt) is Return: retval = stmt.value if not the_exprs and type(retval) is Compare: @@ -1003,12 +1011,11 @@ def _insert_funcname_here_(_insert_envname_here_): retval.left = _inject_value_recorder(envname, retval.left) else: # When there is no return statement at the top level of the `with test` block, - # we inject a `return True` to satisfy the test when the function returns normally. + # we inject a `return True` to satisfy the test when the injected function + # returns normally. with q as thereturn: return True - block_body.extend(thereturn) - - thefunc.body = block_body + thefunc.body.extend(thereturn) return newbody @@ -1019,6 +1026,7 @@ def _test_block_raises(block_body, args): return _test_block_signals_or_raises(block_body, args, "test_raises", q[h[unpythonic_assert_raises]]) def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): + # first pass, outside-in if not block_body: return [] # pragma: no cover, cannot happen through the public API. first_stmt = block_body[0] @@ -1037,16 +1045,13 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): else: raise SyntaxError(f'Expected `with {syntaxname}(exctype):` or `with {syntaxname}[exctype, message]:`') # pragma: no cover - # Before we edit the tree, get the source code in its pre-transformation - # state, so we can include that into the test failure message. - # - # We capture the source in the first pass, so that no macros in tree are - # expanded yet. + # Same remark about outside-in source code capture as in `_test_expr`. sourcecode = unparse(block_body) - # End of first pass. block_body = dyn._macro_expander.visit(block_body) + # second pass, inside-out + testblock_function_name = gensym("_test_block") thetest = q[(a[asserter])(a[exctype], u[sourcecode], diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index b6739a8a..53da0e87 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -93,7 +93,7 @@ def result(loop, i=0): # A basic generator template using abbrev[]. with testset("integration with abbrev"): with continuations: - # We must expand abbreviations in the first pass, before the @dlet that's + # We must expand abbreviations in the outside-in pass, before the @dlet that's # not part of the template (since we splice in stuff that is intended to # refer to the "k" in the @dlet env). So use abbrev[] instead of let_syntax[]. with abbrev: diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index b68ba445..44632104 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -147,7 +147,7 @@ def decorated(*args, **kwargs): test[f6(10) == (2, 3, 100)] test[f6.__name__ == "f6"] - # presence of autocurry should not confuse the first-pass output + # presence of autocurry should not break the output of the outside-in pass with namedlambda: with autocurry: foo = let[(f7, None) in f7 << (lambda x: x)] # noqa: F821 diff --git a/unpythonic/syntax/tests/test_letsyntax.py b/unpythonic/syntax/tests/test_letsyntax.py index 26b1c8ad..9b112d0b 100644 --- a/unpythonic/syntax/tests/test_letsyntax.py +++ b/unpythonic/syntax/tests/test_letsyntax.py @@ -171,8 +171,8 @@ def alias(): twice(appendxyz(7, 8, 9)) # a call is an expression, so as long as not yet expanded, this is ok test[lst == [7, 8, 9] * 2] - with testset("abbrev (first-pass let_syntax)"): - # abbrev: like let_syntax, but expands in the first pass, outside in + with testset("abbrev (outside-in let_syntax)"): + # abbrev: like let_syntax, but expands outside-in # - no lexically scoped nesting # - but can locally rename also macros (since abbrev itself expands before its body) y = abbrev((f, verylongfunctionname))[[ # noqa: F821 diff --git a/unpythonic/syntax/tests/test_prefix.py b/unpythonic/syntax/tests/test_prefix.py index 5c164ec7..87071a84 100644 --- a/unpythonic/syntax/tests/test_prefix.py +++ b/unpythonic/syntax/tests/test_prefix.py @@ -105,7 +105,7 @@ def double(x): # the prefix syntax of Lisp, the speed of Python, and the readability of Haskell! # If you want to play around with this idea, see `unpythonic.dialects.listhell`. with testset("Listhell"): - # `prefix` is a first-pass macro, so placed on the outside, it expands first. + # `prefix` expands outside-in, so placed on the outside, it expands first. with prefix: with autocurry: mymap = lambda f: (foldr, (compose, cons, f), nil) diff --git a/unpythonic/syntax/tests/test_tco.py b/unpythonic/syntax/tests/test_tco.py index 6e7a2518..344d2fb5 100644 --- a/unpythonic/syntax/tests/test_tco.py +++ b/unpythonic/syntax/tests/test_tco.py @@ -78,7 +78,7 @@ def h(x): test[h(10) == 36] with testset("integration with autoreturn"): - # note: apply autoreturn first (first pass, so must be on the outside to run first) + # note: apply autoreturn first (outside-in, so must be on the outside to run first) with autoreturn: with tco: def evenp(x): @@ -143,7 +143,7 @@ def result(loop, x, acc): with testset("integration with quicklambda"): # f[] must expand first so that tco sees it as a lambda. - # `quicklambda` is a first-pass macro, so placed on the outside, it expands first. + # `quicklambda` is an outside-in macro, so placed on the outside, it expands first. with quicklambda: with tco: def g(x): diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 2e7eda76..42daff9a 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -87,11 +87,15 @@ def examine(self, tree): return fallbacks + detect(tree) def detect_lambda(tree): - """Find lambdas in tree. Helper for block macros. + """Find lambdas in tree. Helper for two-pass block macros. - Run ``detect_lambda(tree)`` in the first pass, before allowing any - nested macros to expand. (Those may generate more lambdas that your block - macro is not interested in.) + A two-pass block macro first performs some processing outside-in, then calls + `expander.visit(tree)` to make any nested macro invocations expand, and then + performs some processing inside-out. + + Run ``detect_lambda(tree)`` in the outside-in pass, before calling + `expander.visit(tree)`, because nested macro invocations may generate + more lambdas that your block macro is not interested in. The return value is a ``list``of ``id(lam)``, where ``lam`` is a Lambda node that appears in ``tree``. This list is suitable as ``userlambdas`` for the @@ -103,7 +107,7 @@ def detect_lambda(tree): """ class LambdaDetector(ASTVisitor): def examine(self, tree): - if isdo(tree): + if isdo(tree, expanded=True): thebody = ExpandedDoView(tree).body for thelambda in thebody: # lambda e: ... self.visit(thelambda.body) From 2d5573ec9370bdd5704cc16246b4f049a8a9276e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 01:22:40 +0300 Subject: [PATCH 201/832] with block[...]: pass macro args using brackets --- unpythonic/syntax/tests/test_conts_gen.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index 53da0e87..ef7571b2 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -97,7 +97,7 @@ def result(loop, i=0): # not part of the template (since we splice in stuff that is intended to # refer to the "k" in the @dlet env). So use abbrev[] instead of let_syntax[]. with abbrev: - with block(value) as my_yield: # noqa: F821, here `abbrev` defines the name `value` when we call `my_yield`. + with block[value] as my_yield: # noqa: F821, here `abbrev` defines the name `value` when we call `my_yield`. call_cc[my_yieldf(value)] # for this to work, abbrev[] must eliminate its "if 1" blocks. # noqa: F821, my_yieldf will be defined below and this is a macro. with block as begin_generator_body: # logic to resume after the last executed my_yield, if any @@ -129,9 +129,9 @@ def g(): # and the user code (generator body) doesn't refer to k directly. # (So "k" can be resolved lexically *in the input source code that goes to dlet[]*.) with let_syntax: - with block(value) as my_yield: # noqa: F821 + with block[value] as my_yield: # noqa: F821 call_cc[my_yieldf(value)] # for this to work, let_syntax[] must eliminate its "if 1" blocks. # noqa: F821 - with block(myname, body) as make_generator: # noqa: F821, `let_syntax` defines `myname` and `body` when we call `make_generator`. + with block[myname, body] as make_generator: # noqa: F821, `let_syntax` defines `myname` and `body` when we call `make_generator`. @dlet((k, None)) # noqa: F821 def myname(): # replaced by the user-supplied name, since "myname" is a template parameter. # logic to resume after the last executed my_yield, if any From c447f3747184431d2bb65deae5278b6842a18abc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 01:40:19 +0300 Subject: [PATCH 202/832] fix docstring: multiple the[] marks ARE supported (0.14.3+) --- unpythonic/syntax/testingtools.py | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index f731ca07..ee554a7b 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -123,8 +123,8 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 What `test[expr]` captures for reporting as "result" in the failure message, if the test fails: - - If a `the[...]` mark is present, the subexpression marked as `the[...]`. - At most one `the[]` may appear in a single `test[...]`. + - If any `the[...]` marks are present, the subexpressions marked + as `the[...]`. - Else if `expr` is a comparison, the LHS (leftmost term in case of a chained comparison). So e.g. `test[x < 3]` needs no annotation to do the right thing. This is a common use case, hence automatic. @@ -181,11 +181,9 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 between `def` and `lambda`. In the block variant, the "result" capture rules apply to the return value - designated by `return`. To override, the `the[]` mark can be used for - capturing the value of any one expression inside the block. The mark - doesn't have to be in the `return`. - - At most one `the[]` may appear in the same `with test` block. + designated by `return`. To override, `the[]` marks can be used for capturing + the value of any expressions inside the block. The marks don't have to be + in the `return`; they can appear anywhere. **Failure and error signaling**: From 30028705da2e8f782a5d55a752e2e70c31f86e6c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 01:42:57 +0300 Subject: [PATCH 203/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87cd31ab..18f407a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,6 +103,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - In `aif`, `it` is now only valid in the `then` and `otherwise` parts, as it should always have been. +- Fix docstring of `test`: multiple `the[]` marks were already supported in 0.14.3, as the macro documentation already said, but the docstring claimed otherwise. + --- From d8174f54c1abe5be7952dd6331c6f7627c51e793 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 02:09:24 +0300 Subject: [PATCH 204/832] 0.15: make testing macros expand outside-in --- doc/macros.md | 64 ++++++++++++++++++++++++++++--- unpythonic/syntax/testingtools.py | 22 ----------- 2 files changed, 58 insertions(+), 28 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 7e5a8c0f..9c608849 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -73,6 +73,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose - [``unpythonic.test.fixtures``: a test framework for macro-enabled Python](#unpythonic-test-fixtures-a-test-framework-for-macro-enabled-python) - [Overview](#overview) - [Testing syntax quick reference](#testing-syntax-quick-reference) + - [Expansion order](#expansion-order) - [`with test`: test blocks](#with-test-test-blocks) - [`the`: capture the value of interesting subexpressions](#the-capture-the-value-of-interesting-subexpressions) - [Test sessions and testsets](#test-sessions-and-testsets) @@ -1707,16 +1708,38 @@ Because `unpythonic.test.fixtures` is, by design, a minimalistic *no-framework* #### Testing syntax quick reference -**Imports**: +**Imports** - complete list: ```python from unpythonic.syntax import (macros, test, test_raises, test_signals, - fail, error, warn, the) + fail, error, warn, the, expand_testing_macros_first) from unpythonic.test.fixtures import (session, testset, returns_normally, catch_signals, terminate) ``` -**Overall structure** - session and testsets: +**Overall structure** of typical unit test module: + +```python +from unpythonic.syntax import macros, test, test_raises, the +from unpythonic.test.fixtures import session, testset + +def runtests(): + with testset("something 1"): + ... + with testset("something 2"): + ... + ... + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() +``` + +The if-main idiom allows running this test module individually, but it is tagged with `# pragma: no cover`, so that the coverage reporter won't yell about it when the module is run by the test runner as part of the complete test suite (which, incidentally, is also a good opportunity to measure coverage). + +If you want to ensure that testing macros expand before anything else - including your own code-walking block macros (when you have tests inside the body) - import the macro `expand_testing_macros_first`, and put a `with expand_testing_macros_first` around the affected code. (See [Expansion order](#expansion-order), below.) + +**Sessions and testsets**: ```python with session(name): @@ -1814,11 +1837,38 @@ The constructs `with test_raises`, `with test_signals` do **not** support `the[] Tests can be nested; this is sometimes useful as an explicit signal barrier. +#### Expansion order + +**Changed in v0.15.0**. *The testing macros now expand outside-in; this allows `mcpyrate.debug.step_expansion` to treat them as a separate step. In v0.14.3, which introduced the test framework, they used to be two-pass macros.* + +Your test macro invocations may get partially expanded code, if those invocations reside in the body of an invocation of a block macro that also expands outside-in: + +```python +with yourblockmacro: # outside-in + test[...] +``` + +Here the `...` may be edited by `yourblockmacro` before `test[]` sees it. (It likely **will** be edited, since this pattern will commonly appear in the tests for `yourblockmacro`, where the whole point is to have the `...` depend on what `yourblockmacro` outputs.) + +If you need testing macros to expand before anything else even in this scenario (so you can more clearly see where in the unexpanded source code a particular expression came from), you can do this: + +```python +from unpythonic.syntax import macros, expand_testing_macros_first + +with expand_testing_macros_first: + with yourblockmacro: + test[...] +``` + +The `expand_testing_macros_first` macro is itself a code-walking block macro that does as it says on the tin. The testing macros are identified by scanning the bindings of the current macro expander; names don't matter, so it respects as-imports. + +This does imply that `your_block_macro` will then receive the expanded form of `test[...]` as input, but that's macros for you. You'll have to choose which is more important: seeing the unexpanded code in error messages, or receiving unexpanded `test[]` expressions in `yourblockmacro`. + #### `with test`: test blocks Test blocks are meant for testing code that requires Python statements; i.e. does not fit into Python's expression sublanguage. -In `unpythonic.test.fixtures`, a test block is implicitly lifted into a function. Hence, any local variables assigned to inside the block remain local to the implicit function. Use Python's `nonlocal` and `global` keywords, if needed. +In `unpythonic.test.fixtures`, **a test block is implicitly lifted into a function**. Hence, any local variables assigned to inside the block remain local to the implicit function. Use Python's `nonlocal` and `global` keywords, if needed. By default, a `with test` block asserts just that it completes normally. If you instead want to assert that an expression is truthy, use `return expr` to terminate the implicit function and return the value of the desired `expr`. The return value is passed to the test asserter for checking that it is truthy. @@ -1830,7 +1880,9 @@ The `with test_raises[exctype]` and `with test_signals[exctype]` blocks assert t The point of `unpythonic.test.fixtures` is to make testing macro-enabled Python as frictionless as reasonably possible. -Inside a `test[]` expression, or anywhere within the code in a `with test` block, the `the[]` macro can be used to declare any number of subexpressions as interesting, for capturing the source code and value into the test failure message, which is shown if the test fails. Source code is captured in the outside-in pass, before any nested inside-out macros expand. (Many macros defined by `unpythonic` expand inside-out.) The value is captured at run time as a side effect just after the value has been evaluated. +Inside a `test[]` expression, or anywhere within the code in a `with test` block, the `the[]` macro can be used to declare any number of subexpressions as interesting, for capturing the source code and value into the test failure message, which is shown if the test fails. Each `the[]` captures one subexpression (as many times as it is evaluated, in the order evaluated). + +Because test macros expand outside-in, the source code is captured before any nested inside-out macros expand. (Many macros defined by `unpythonic` expand inside-out.) The value is captured at run time as a side effect just after the value has been evaluated. By default (if no explicit `the[]` is present), `test[]` implicitly inserts a `the[]` for the leftmost term if the top-level expression is a comparison (common use case), and otherwise does not capture anything. @@ -1858,7 +1910,7 @@ The `with session()` in the example session above is optional. The human-readabl Tests can optionally be grouped into testsets. Each `testset` tallies passed, failed and errored tests within it, and displays the totals when it exits. Testsets can be named and nested. -It is useful to have at least one `testset` (e.g. the implicit top-level one established by `with session`), because the `testset` mechanism forms one half of the test framework. It is possible to use the test macros without a `testset`, but that is only intended for building alternative test frameworks. +It is useful to have at least one `testset` (the implicit top-level one established by `with session` is sufficient), because the `testset` mechanism forms one half of the test framework. It is possible to use the test macros without a `testset`, but that is only intended for building alternative test frameworks. Testsets also provide an option to locally install a `postproc` handler that gets a copy of each failure or error in that testset (and by default, any of its inner testsets), after the failure or error has been printed. In nested testsets, the dynamically innermost `postproc` wins. A failure is an instance of `unpythonic.test.fixtures.TestFailure`, an error is an instance of `unpythonic.test.fixtures.TestError`, and a warning is an instance of `unpythonic.test.fixtures.TestWarning`. All three inherit from `unpythonic.test.fixtures.TestingException`. Beside the human-readable message, these exception types contain attributes with programmatically inspectable information about what happened. diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index ee554a7b..69f6e31c 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -805,8 +805,6 @@ def _warn_expr(tree): # Expr variants. def _test_expr(tree): - # first pass, outside-in - # Note we want the line number *before macro expansion*, so we capture it now. ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] @@ -840,10 +838,6 @@ def _test_expr(tree): if not the_exprs and type(tree) is Compare: # inject the implicit the[] on the LHS tree.left = _inject_value_recorder(envname, tree.left) - tree = dyn._macro_expander.visit(tree) - - # second pass, inside-out - # We delay the execution of the test expr using a lambda, so # `unpythonic_assert` can get control first before the expr runs. # @@ -913,8 +907,6 @@ def _test_expr_raises(tree): return _test_expr_signals_or_raises(tree, "test_raises", q[h[unpythonic_assert_raises]]) def _test_expr_signals_or_raises(tree, syntaxname, asserter): - # first pass, outside-in - ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] @@ -931,10 +923,6 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): # Same remark about outside-in source code capture as in `_test_expr`. sourcecode = unparse(tree) - tree = dyn._macro_expander.visit(tree) - - # second pass, inside-out - return q[(a[asserter])(a[exctype], u[sourcecode], lambda: a[tree], @@ -948,7 +936,6 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): # The strategy is we capture the block body into a new function definition, # and then `unpythonic_assert` on that function. def _test_block(block_body, args): - # first pass, outside-in if not block_body: return [] # pragma: no cover, cannot happen through the public API. first_stmt = block_body[0] @@ -975,10 +962,6 @@ def _test_block(block_body, args): # Handle the `the[...]` marks, if any. block_body, the_exprs = _transform_important_subexpr(block_body, envname=envname) - block_body = dyn._macro_expander.visit(block_body) - - # second pass, inside-out - # Prepare the function template to be injected, and splice the contents # of the `with test` block as the function body. testblock_function_name = gensym("_test_block") @@ -1024,7 +1007,6 @@ def _test_block_raises(block_body, args): return _test_block_signals_or_raises(block_body, args, "test_raises", q[h[unpythonic_assert_raises]]) def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): - # first pass, outside-in if not block_body: return [] # pragma: no cover, cannot happen through the public API. first_stmt = block_body[0] @@ -1046,10 +1028,6 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): # Same remark about outside-in source code capture as in `_test_expr`. sourcecode = unparse(block_body) - block_body = dyn._macro_expander.visit(block_body) - - # second pass, inside-out - testblock_function_name = gensym("_test_block") thetest = q[(a[asserter])(a[exctype], u[sourcecode], From dcc71e57ae93a6dca56254398efb89b6e6f5e2ce Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 02:09:43 +0300 Subject: [PATCH 205/832] macro docs: 0.15.0, not just 0.15 --- doc/macros.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 9c608849..d6e6803c 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -19,7 +19,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose *This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* -**Changed in 0.15.** *To run macro-enabled programs, use the [`macropython`](https://github.com/Technologicat/mcpyrate/blob/master/doc/repl.md#macropython-the-universal-bootstrapper) bootstrapper from [`mcpyrate`](https://github.com/Technologicat/mcpyrate).* +**Changed in v0.15.0.** *To run macro-enabled programs, use the [`macropython`](https://github.com/Technologicat/mcpyrate/blob/master/doc/repl.md#macropython-the-universal-bootstrapper) bootstrapper from [`mcpyrate`](https://github.com/Technologicat/mcpyrate).* **This document is up-to-date for v0.14.3.** @@ -638,7 +638,7 @@ Support for other forms of assignment may or may not be added in a future versio ### ``f``: underscore notation (quick lambdas) for Python. -**Changed in 0.15.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves. The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else the underscore is available to be used as a regular variable. If you use `f[]`, change your import of this macro to `from unpythonic.syntax import macros, f`.* +**Changed in v0.15.0.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves. The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else the underscore is available to be used as a regular variable. If you use `f[]`, change your import of this macro to `from unpythonic.syntax import macros, f`.* The syntax ``f[...]`` creates a lambda, where each underscore in the ``...`` part introduces a new parameter. The macro does not descend into any nested ``f[]``. @@ -771,7 +771,7 @@ Manual uses of the `curry` decorator (on both `def` and `lambda`) are detected, ### ``lazify``: call-by-need for Python -**Changed in 0.15.** *Up to 0.14.x, the `lazy[]` macro, that is used together with `with lazify`, used to be provided by `macropy`, but now that we use `mcpyrate`, we provide it ourselves. If you use `lazy[]`, change your import of that macro to `from unpythonic.syntax import macros, lazy`*. +**Changed in v0.15.0.** *Up to 0.14.x, the `lazy[]` macro, that is used together with `with lazify`, used to be provided by `macropy`, but now that we use `mcpyrate`, we provide it ourselves. If you use `lazy[]`, change your import of that macro to `from unpythonic.syntax import macros, lazy`*. Also known as *lazy functions*. Like [lazy/racket](https://docs.racket-lang.org/lazy/index.html), but for Python. Note if you want *lazy sequences* instead, Python already provides those; just use the generator facility (and decorate your gfunc with ``unpythonic.gmemoize`` if needed). From 046a4f209f55587dfe65e3f743aeac6979517fab Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 02:15:44 +0300 Subject: [PATCH 206/832] docs: split off troubleshooting to its own document --- CONTRIBUTING.md | 1 + README.md | 21 +++------------------ doc/design-notes.md | 1 + doc/dialects.md | 1 + doc/dialects/lispython.md | 1 + doc/dialects/listhell.md | 1 + doc/dialects/pytkell.md | 1 + doc/features.md | 1 + doc/macros.md | 1 + doc/readings.md | 1 + doc/repl.md | 1 + doc/troubleshooting.md | 37 +++++++++++++++++++++++++++++++++++++ 12 files changed, 50 insertions(+), 18 deletions(-) create mode 100644 doc/troubleshooting.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c48bc92b..ce881f73 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,6 +5,7 @@ - [Syntactic macro feature set](doc/macros.md) - [Examples of creating dialects using `mcpyrate`](doc/dialects.md) - [REPL server](doc/repl.md) +- [Troubleshooting](doc/troubleshooting.md) - [Design notes](doc/design-notes.md) - [Additional reading](doc/readings.md) - **Contribution guidelines** diff --git a/README.md b/README.md index ecbecabd..c9211976 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (langua - [Syntactic macro feature set](doc/macros.md) - [Examples of creating dialects using `mcpyrate`](doc/dialects.md): Python the way you want it. - [REPL server](doc/repl.md): interactively hot-patch your running Python program. +- [Troubleshooting](doc/troubleshooting.md): possible solutions to possibly common issues. - [Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. - [Additional reading](doc/readings.md): links to material relevant in the context of ``unpythonic``. - [Contribution guidelines](CONTRIBUTING.md): for understanding the codebase, or if you're interested in making a code or documentation PR. @@ -686,29 +687,13 @@ or Not working as advertised? Missing a feature? Documentation needs improvement? +In case of a problem, see [Troubleshooting](doc/troubleshooting.md) first. Then: + **[Issue reports](https://github.com/Technologicat/unpythonic/issues) and [pull requests](https://github.com/Technologicat/unpythonic/pulls) are welcome.** [Contribution guidelines](CONTRIBUTING.md). While `unpythonic` is intended as a serious tool for improving productivity as well as for teaching, right now my work priorities mean that it's developed and maintained on whatever time I can spare for it. Thus getting a response may take a while, depending on which project I happen to be working on. -## Troubleshooting - -### Cannot import the name `macros`? - -Could be a stale bytecode cache that Python thinks is still valid. This can happen especially if you first accidentally run `python3 some_macro_program.py`, and only then realize the invocation should have been `macropython some_macro_program.py`. - -The invocation with bare Python may compile to bytecode successfully and write the bytecode cache, but there is indeed no run-time object named `macros`, so the program will crash at that point. When the program is run again via `macropython`, the loader sees the bytecode cache, and because its `mtime` (as compared to the `.py` file) suggests it's up to date, the `.py` file is not automatically recompiled. - -Try clearing the bytecode caches in the affected directory with `macropython -c .`; this will force a recompile of the `.py` files the next time they are loaded. Then run normally, with `macropython some_macro_program.py`. - - -### I'm hacking a macro inside a module in `unpythonic.syntax` and my changes don't take? - -As of `mcpyrate` 3.4.0, macro re-exports, as done by `unpythonic.syntax.__init__`, may confuse the macro-dependency analyzer that determines bytecode cache validity. It only looks at the macro-import dependency graph, not the full dependency graph. I might change this in the future, but doing so will make it a lot slower than it needs to be in most circumstances. - -Try clearing the bytecode cache in `unpythonic/syntax/`; this will force a recompile. - - ## License All original code is released under the 2-clause [BSD license](LICENSE.md). diff --git a/doc/design-notes.md b/doc/design-notes.md index e751e601..31733d21 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -5,6 +5,7 @@ - [Syntactic macro feature set](macros.md) - [Examples of creating dialects using `mcpyrate`](dialects.md) - [REPL server](repl.md) +- [Troubleshooting](troubleshooting.md) - **Design notes** - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/dialects.md b/doc/dialects.md index 88f3294b..bb6159ca 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -8,6 +8,7 @@ - [Listhell](dialects/listhell.md) - [Pytkell](dialects/pytkell.md) - [REPL server](repl.md) +- [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 0183f058..2b5f4c78 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -8,6 +8,7 @@ - [Listhell](listhell.md) - [Pytkell](pytkell.md) - [REPL server](../repl.md) +- [Troubleshooting](../troubleshooting.md) - [Design notes](../design-notes.md) - [Additional reading](../readings.md) - [Contribution guidelines](../../CONTRIBUTING.md) diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index 173ddf63..f171320b 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -8,6 +8,7 @@ - **Listhell** - [Pytkell](pytkell.md) - [REPL server](../repl.md) +- [Troubleshooting](../troubleshooting.md) - [Design notes](../design-notes.md) - [Additional reading](../readings.md) - [Contribution guidelines](../../CONTRIBUTING.md) diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index aa161b67..39e979e3 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -8,6 +8,7 @@ - [Listhell](listhell.md) - **Pytkell** - [REPL server](../repl.md) +- [Troubleshooting](../troubleshooting.md) - [Design notes](../design-notes.md) - [Additional reading](../readings.md) - [Contribution guidelines](../../CONTRIBUTING.md) diff --git a/doc/features.md b/doc/features.md index c5328c7d..f333a9e4 100644 --- a/doc/features.md +++ b/doc/features.md @@ -5,6 +5,7 @@ - [Syntactic macro feature set](macros.md) - [Examples of creating dialects using `mcpyrate`](dialects.md) - [REPL server](repl.md) +- [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/macros.md b/doc/macros.md index d6e6803c..0046d109 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -5,6 +5,7 @@ - **Syntactic macro feature set** - [Examples of creating dialects using `mcpyrate`](dialects.md) - [REPL server](repl.md) +- [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/readings.md b/doc/readings.md index e58ddf3f..b88e943c 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -5,6 +5,7 @@ - [Syntactic macro feature set](macros.md) - [Examples of creating dialects using `mcpyrate`](dialects.md) - [REPL server](repl.md) +- [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) - **Additional reading** - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/repl.md b/doc/repl.md index 3d3c10e8..6c101be6 100644 --- a/doc/repl.md +++ b/doc/repl.md @@ -5,6 +5,7 @@ - [Syntactic macro feature set](macros.md) - [Examples of creating dialects using `mcpyrate`](dialects.md) - **REPL server** +- [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md new file mode 100644 index 00000000..96a87855 --- /dev/null +++ b/doc/troubleshooting.md @@ -0,0 +1,37 @@ +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- [Syntactic macro feature set](macros.md) +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- [REPL server](repl.md) +- **Troubleshooting** +- [Design notes](design-notes.md) +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + + +**Table of Contents** + +- [Troubleshooting](#troubleshooting) + - [Cannot import the name `macros`?](#cannot-import-the-name-macros) + - [I'm hacking a macro inside a module in `unpythonic.syntax` and my changes don't take?](#im-hacking-a-macro-inside-a-module-in-unpythonicsyntax-and-my-changes-dont-take) + + + +## Troubleshooting + +### Cannot import the name `macros`? + +Could be a stale bytecode cache that Python thinks is still valid. This can happen especially if you first accidentally run `python3 some_macro_program.py`, and only then realize the invocation should have been `macropython some_macro_program.py`. + +The invocation with bare Python may compile to bytecode successfully and write the bytecode cache, but there is indeed no run-time object named `macros`, so the program will crash at that point. When the program is run again via `macropython`, the loader sees the bytecode cache, and because its `mtime` (as compared to the `.py` file) suggests it's up to date, the `.py` file is not automatically recompiled. + +Try clearing the bytecode caches in the affected directory with `macropython -c .`; this will force a recompile of the `.py` files the next time they are loaded. Then run normally, with `macropython some_macro_program.py`. + + +### I'm hacking a macro inside a module in `unpythonic.syntax` and my changes don't take? + +As of `mcpyrate` 3.4.0, macro re-exports, as done by `unpythonic.syntax.__init__`, may confuse the macro-dependency analyzer that determines bytecode cache validity. It only looks at the macro-import dependency graph, not the full dependency graph. I might change this in the future, but doing so will make it a lot slower than it needs to be in most circumstances. + +Try clearing the bytecode cache in `unpythonic/syntax/`; this will force a recompile. From 35b4b93b7ef9b467c6bb25b84ceaee7fee60579f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 03:07:58 +0300 Subject: [PATCH 207/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 18f407a4..8d42873a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. +- All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) + **Non-breaking changes**: From 3b6f42e5db907965c0ad2d587e21d0ebcf90f8a7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 03:08:06 +0300 Subject: [PATCH 208/832] extend troubleshooting doc --- doc/troubleshooting.md | 64 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index 96a87855..1693ea59 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -13,25 +13,71 @@ **Table of Contents** -- [Troubleshooting](#troubleshooting) +- [Common issues and questions](#common-issues-and-questions) + - [Do I need a macro expander to use `unpythonic`?](#do-i-need-a-macro-expander-to-use-unpythonic) + - [Why `mcpyrate` and not MacroPy?](#why-mcpyrate-and-not-macropy) - [Cannot import the name `macros`?](#cannot-import-the-name-macros) - - [I'm hacking a macro inside a module in `unpythonic.syntax` and my changes don't take?](#im-hacking-a-macro-inside-a-module-in-unpythonicsyntax-and-my-changes-dont-take) + - [But I did run my program with `macropython`?](#but-i-did-run-my-program-with-macropython) + - [I'm hacking a macro inside a module in `unpythonic.syntax`, and my changes don't take?](#im-hacking-a-macro-inside-a-module-in-unpythonicsyntax-and-my-changes-dont-take) -## Troubleshooting +## Common issues and questions + +### Do I need a macro expander to use `unpythonic`? + +If you intend to only use the [Pure-Python feature set](features.md), then no. This is why `unpythonic` does not automatically pull in a macro expander when you install it. + +On the other hand, `unpythonic` is a kitchen-sink language extension, and half of the functionality comes from macros. Even the test framework for `unpythonic`'s own automated tests uses macros! + +If you intend to **use** `unpythonic.syntax` or `unpythonic.dialects`, or if you intend to **develop** `unpythonic` (specifically: to be able to run its test suite), then you will need a macro expander. + +As of v0.15.0, specifically you'll need [`mcpyrate`](https://github.com/Technologicat/mcpyrate). + + +### Why `mcpyrate` and not MacroPy? + +[`mcpyrate`](https://github.com/Technologicat/mcpyrate) is an advanced, third-generation macro expander (and language lab) for Python, taking in the lessons learned from both [`macropy3`](https://github.com/azazel75/macropy) and [`mcpy`](https://github.com/delapuente/mcpy), and expanding (pun not intended) on that. + +Beside the advanced features, the reason we use `mcpyrate` is that the `unpythonic.syntax` rabbit hole has become deep enough to benefit from agile experimentation at the meta-metaprogramming level. Allowing the macro expander and the syntax layer of `unpythonic` to co-evolve results in better software. + ### Cannot import the name `macros`? -Could be a stale bytecode cache that Python thinks is still valid. This can happen especially if you first accidentally run `python3 some_macro_program.py`, and only then realize the invocation should have been `macropython some_macro_program.py`. +In `mcpyrate`-based programs, there is no run-time object named `macros`, so failing to import that usually means that, for some reason, the macro expander was not active. + +Macro-enabled, `mcpyrate`-based programs expect to be run with `macropython` (included in the [`mcpyrate` PyPI package](https://pypi.org/project/mcpyrate/)) instead of bare `python3`. + +Basically, you can `macropython script.py` or `macropython -m some.module`, like you would with `python3`. The advantage is you can run macro-enabled code without a per-project bootstrapper, since `macropython` handles bootstrapping the macro expander for you. + +See the [`macropython` documentation](https://github.com/Technologicat/mcpyrate/blob/master/doc/repl.md#macropython-the-universal-bootstrapper) for details. + + +### But I did run my program with `macropython`? + +The problem could be a stale bytecode cache that `mcpyrate` thinks is still valid. This can happen especially if you first accidentally run `python3 some_macro_program.py`, and only then realize the invocation should have been `macropython some_macro_program.py`. + +The invocation with bare Python may compile to bytecode successfully and write the bytecode cache, but there is indeed no run-time object named `macros`, so the program will crash at that point. When the program is run again via `macropython`, `mcpyrate`'s loader sees the existing bytecode cache, and because its `mtime` (as compared to the `.py` file) suggests it's up to date, the `.py` file is not automatically recompiled. + +Try clearing the bytecode caches in the affected directory with: +```bash +macropython -c . +``` +This will force a recompile of the `.py` files the next time they are loaded. Then run normally, with `macropython some_macro_program.py`. + + +### I'm hacking a macro inside a module in `unpythonic.syntax`, and my changes don't take? + +This is also likely due to a stale bytecode cache. As of `mcpyrate` 3.4.0, macro re-exports, used by `unpythonic.syntax.__init__`, may confuse the macro-dependency analyzer that determines bytecode cache validity. -The invocation with bare Python may compile to bytecode successfully and write the bytecode cache, but there is indeed no run-time object named `macros`, so the program will crash at that point. When the program is run again via `macropython`, the loader sees the bytecode cache, and because its `mtime` (as compared to the `.py` file) suggests it's up to date, the `.py` file is not automatically recompiled. +The thing to realize here is that as per macropythonic tradition, in `mcpyrate`, a function being a macro is a property of its **use site**, not of its definition site. So how do we re-export a macro? We simply re-export the macro function, like we would do for any other function. -Try clearing the bytecode caches in the affected directory with `macropython -c .`; this will force a recompile of the `.py` files the next time they are loaded. Then run normally, with `macropython some_macro_program.py`. +Importantly, the import to make that re-export happen does not look like a macro-import. This is the right way to do it, since we want to make the object (macro function) available for clients to import, **not** establish bindings in the macro expander *for compiling the module `unpythonic.syntax.__init__` itself*. (The latter is what a macro-import does - it establishes macro bindings *for the module it lexically appears in*.) +The problem is, the macro-dependency analyzer only looks at the macro-import dependency graph, not the full dependency graph, so when analyzing the user program (e.g. a unit test module in `unpythonic.syntax.tests`), it doesn't notice that the macro definition has changed. -### I'm hacking a macro inside a module in `unpythonic.syntax` and my changes don't take? +I might modify the `mcpyrate` analyzer in the future, but doing so will make the dependency scan a lot slower than it needs to be in most circumstances, because a large majority of imports in Python have nothing to do with macros. -As of `mcpyrate` 3.4.0, macro re-exports, as done by `unpythonic.syntax.__init__`, may confuse the macro-dependency analyzer that determines bytecode cache validity. It only looks at the macro-import dependency graph, not the full dependency graph. I might change this in the future, but doing so will make it a lot slower than it needs to be in most circumstances. +For now, we just note that this issue mainly concerns developers of large macro packages (such as `unpythonic.syntax`) that need to split - for factoring reasons - their macro definitions into separate modules, while presenting all macros to the user in one interface module. This issue does not affect the development of macro-using programs, or any programs where macros are imported from their original definition site (like they always were with MacroPy). -Try clearing the bytecode cache in `unpythonic/syntax/`; this will force a recompile. +Try clearing the bytecode cache in `unpythonic/`; this will force a recompile. From ab3ac70ab609fcc2e419bf0e68348f573eb003d3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 03:23:19 +0300 Subject: [PATCH 209/832] remove outdated comments --- unpythonic/syntax/testingtools.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 69f6e31c..4d125602 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -242,7 +242,6 @@ def test(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("test (block mode) does not take an as-part") # pragma: no cover - # Two-pass macros. with dyn.let(_macro_expander=expander): if syntax == "expr": if args: @@ -294,7 +293,6 @@ def test_signals(tree, *, args, syntax, expander, **kw): # noqa: F811 if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("test_signals (block mode) does not take an as-part") # pragma: no cover - # Two-pass macros. with dyn.let(_macro_expander=expander): if syntax == "expr": if args: @@ -396,8 +394,6 @@ def error(tree, *, syntax, expander, **kw): # noqa: F811 if syntax != "expr": raise SyntaxError("error is an expr macro only") # pragma: no cover - # Expand outside in. The ordering shouldn't matter here. - # The underlying `test` machinery needs to access the expander. with dyn.let(_macro_expander=expander): return _error_expr(tree) @@ -420,8 +416,6 @@ def warn(tree, *, syntax, expander, **kw): # noqa: F811 if syntax != "expr": raise SyntaxError("warn is an expr macro only") # pragma: no cover - # Expand outside in. The ordering shouldn't matter here. - # The underlying `test` machinery needs to access the expander. with dyn.let(_macro_expander=expander): return _warn_expr(tree) From ead149d44c3eb126ffe289d114a791a29825289b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 03:41:36 +0300 Subject: [PATCH 210/832] fix macropy-isms --- doc/design-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 31733d21..e984fbd1 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -260,7 +260,7 @@ More on type systems: - ``autoreturn`` adds ``return`` statements where there weren't any, - ``quicklambda`` transforms things-that-look-like-list-lookups into ``lambda`` function definitions, - ``multilambda`` transforms things-that-look-like-lists (in the body of a ``lambda``) into sequences of multiple expressions, using ``do[]``. - - Hence, a lexically outer block of one of these types *will expand first*, before any macros inside it are expanded, in contrast to the default *from inside out* expansion order. + - Hence, a lexically outer block of one of these types *will expand first*, before any macros inside it are expanded. - This yields clean, standard-ish Python for the rest of the macros, which then don't need to worry about their input meaning something completely different from what it looks like. - An already expanded ``do[]`` (including that inserted by `multilambda`) is accounted for by all ``unpythonic.syntax`` macros when handling expressions. @@ -295,7 +295,7 @@ More on type systems: - ``envify`` needs to see the output of ``lazify`` in order to shunt function args into an unpythonic ``env`` without triggering the implicit forcing. - Some of the block macros can be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``autocurry`` and ``namedlambda``) require their own ``with`` statement. - - This is a [known issue in MacroPy](https://github.com/azazel75/macropy/issues/21). I have made a [fix](https://github.com/azazel75/macropy/pull/22), but still need to make proper test cases to get it merged. + - This was the case with MacroPy [[issue report](https://github.com/azazel75/macropy/issues/21)] [[PR](https://github.com/azazel75/macropy/pull/22)]. Should work in `mcpyrate`, but needs testing. - If something goes wrong in the expansion of one block macro in a ``with`` statement that specifies several block macros, surprises may occur. - When in doubt, use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. From d1b13c246b78cb17e25740d8b7109555edd32e63 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 7 May 2021 19:06:56 +0300 Subject: [PATCH 211/832] update SLOC counts for test framework Changed since macro docstrings now live in implementation modules, no longer in `unpythonic.syntax.__init__`. --- CONTRIBUTING.md | 2 +- doc/macros.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ce881f73..3975b693 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -104,7 +104,7 @@ We use a custom testing framework, which lives in the modules `unpythonic.test.f In retrospect, given that the main aim was compact testing syntax for macro-enabled Python code (without installing another import hook, doing which would disable the macro expander), it might have made more sense to make the testing macros compile to [pytest](https://docs.pytest.org/en/latest/). But hey, it's short, may have applications in teaching... and now we can easily write custom test runners, since the testing framework is just a `mcpyrate` library. It's essentially a *no-framework* (cf. "NoSQL"), which provides the essentials and lets the user define the rest. -(The whole framework is about 1.3k SLOC, counting docstrings, comments and blanks; under 600 SLOC if counting only active code lines. Add another 800 SLOC (all) / 200 SLOC (active code lines) for the condition system.) +(The whole framework is about 1.8k SLOC, counting docstrings, comments and blanks; under 700 SLOC if counting only active code lines. Add another 800 SLOC (all) / 200 SLOC (active code lines) for the condition system.) Since `unpythonic` is a relatively loose collection of language extensions and utilities, that's about it for the 30 000 ft (9 144 m) view. diff --git a/doc/macros.md b/doc/macros.md index 0046d109..6002fa30 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1957,13 +1957,13 @@ Also, in my opinion, `unittest` is overly verbose to use; automated tests are al The central functional requirement for whatever would be used for testing `unpythonic` was to be able to easily deal with macro-enabled Python. No hoops to jump through, compared to testing regular Python, in order to be able to test all of `unpythonic` (including `unpythonic.syntax`) in a uniform way. -Simple and minimalistic would be a bonus. As of v0.14.3, the whole test framework is about 1.3k SLOC, counting docstrings, comments and blanks; under 600 SLOC if counting only active code lines. Add another 800 SLOC (all) / 200 SLOC (active code lines) for the machinery that implements conditions and restarts. +Simple and minimalistic would be a bonus. As of v0.14.3, the whole test framework is about 1.8k SLOC, counting docstrings, comments and blanks; under 700 SLOC if counting only active code lines. Add another 800 SLOC (all) / 200 SLOC (active code lines) for the machinery that implements conditions and restarts. The framework will likely still evolve a bit as I find more holes in the [UX](https://en.wikipedia.org/wiki/User_experience) - which so far has led to features such as `the[]` and AST value auto-unparsing - but most of the desired functionality is already there. For example, I consider pytest-style implicit fixtures and a central test discovery system as outside the scope of this system. It's clear that `unpythonic.test.fixtures` is not going to replace `pytest`, nor does it aim to do so - [any more than Chuck Moore's Forth-based VLSI tools](https://yosefk.com/blog/my-history-with-forth-stack-machines.html) were intended to replace the commercial [VLSI](https://en.wikipedia.org/wiki/Very_Large_Scale_Integration) offerings. -What we have is small, simple, custom-built for its purpose (works well with macro-enabled Python; integrates with conditions and restarts), arguably somewhat pedagogic (demonstrates how to build a test framework in under 1k SLOC), and importantly, works just fine. +What we have is small, simple, custom-built for its purpose (works well with macro-enabled Python; integrates with conditions and restarts), arguably somewhat pedagogic (demonstrates how to build a test framework in under 700 active SLOC), and importantly, works just fine. #### Etymology and roots From ac117c096ac291a872aef40e01d5bf54d0bda535 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 03:03:33 +0300 Subject: [PATCH 212/832] let macros: accept env-assignment syntax and brackets --- CHANGELOG.md | 41 ++++ unpythonic/syntax/letdo.py | 6 +- unpythonic/syntax/letdoutil.py | 273 ++++++++++++---------- unpythonic/syntax/letsyntax.py | 5 +- unpythonic/syntax/tests/test_letdoutil.py | 46 ++++ 5 files changed, 244 insertions(+), 127 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d42873a..d7d9a533 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,47 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `with expr`/`with block`, for `with let_syntax`/`with abbrev` - Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. +- For syntactic consistency, allow env-assignment notation and brackets to declare bindings in the `let` family of macros. The preferred syntaxes for the `let` macro are now: + + ```python + let[x << 42, y << 9001][...] # lispy expr + let[[x << 42, y << 9001] in ...] # haskelly let-in + let[..., where[x << 42, y << 9001]] # haskelly let-where + ``` + If there is just one binding, these become: + ```python + let[x << 42][...] + let[[x << 42] in ...] + let[..., where[x << 42]] + ``` + Similarly for `letseq`, `letrec`, and the decorator versions; and for the expr forms of `let_syntax`, `abbrev`. The reason for preferring this notation is that it is consistent with both `unpythonic`'s env-assignments (`let` bindings live in an `env`) and the use of brackets to denote macro invocations. + + However, all the following are also accepted, with the meaning exactly the same: + ```python + let[(x << 42, y << 9001) in ...] + let[..., where(x << 42, y << 9001)] + let[[x, 42], [y, 9001]][...] + let[(x, 42), (y, 9001)][...] + let[[[x, 42], [y, 9001]] in ...] + let[[(x, 42), (y, 9001)] in ...] + let[([x, 42], [y, 9001]) in ...] + let[((x, 42), (y, 9001)) in ...] + let[..., where[[x, 42], [y, 9001]]] + let[..., where[(x, 42), (y, 9001)]] + let[..., where([x, 42], [y, 9001])] + let[..., where((x, 42), (y, 9001))] + ``` + For a single binding, these are also accepted: + ```python + let[x, 42][...] + let[(x << 42) in ...] + let[[x, 42] in ...] + let[(x, 42) in ...] + let[..., where(x << 42)] + let[..., where[x, 42]] + let[..., where(x, 42)] + ``` + - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index c50acf83..56072323 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -73,11 +73,11 @@ # in the macro interface that gives a copy of the whole macro invocation # node (so we could see the exact original syntax). # -# allow_call_in_name_position: used by let_syntax to allow template definitions. -def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, allow_call_in_name_position=False): +# letsyntax_mode: used by let_syntax to allow template definitions. +def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, letsyntax_mode=False): with dyn.let(_macro_expander=macro_expander): # implicit do (extra bracket notation) needs this. if args: - bs = canonize_bindings(args, allow_call_in_name_position=allow_call_in_name_position) + bs = canonize_bindings(args, letsyntax_mode=letsyntax_mode) return let_transformer(bindings=bs, body=tree) # haskelly syntax, let[(...) in ...], let[..., where(...)] view = UnexpandedLetView(tree) # note "tree" here is only the part inside the brackets diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 85416170..0796561c 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -6,10 +6,11 @@ "UnexpandedEnvAssignView", "UnexpandedLetView", "UnexpandedDoView", "ExpandedLetView", "ExpandedDoView"] -from ast import (Call, Name, Subscript, Index, Compare, In, +from ast import (Call, Name, Subscript, Compare, In, Tuple, List, Constant, BinOp, LShift, Lambda) import sys +from mcpyrate import unparse from mcpyrate.core import Done from .astcompat import getconstant, Str @@ -19,37 +20,71 @@ dof_name = "dof" # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. currycall_name = "currycall" # output of ``unpythonic.syntax.curry`` -def canonize_bindings(elts, allow_call_in_name_position=False): # public as of v0.14.3+ +def _get_subscript_slice(tree): + assert type(tree) is Subscript + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + return tree.slice + return tree.slice.value +def _set_subscript_slice(tree, newslice): # newslice: AST + assert type(tree) is Subscript + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + tree.slice = newslice + tree.slice.value = newslice +def _normalize_macroargs_node(macroargs): + # We do this like `mcpyrate.expander.destructure_candidate` does, + # except that we also destructure a list. + if type(macroargs) in (List, Tuple): # [a0, a1, ...] + return macroargs.elts + return [macroargs] # anything that doesn't have at least one comma at the top level + +def canonize_bindings(elts, letsyntax_mode=False): # public as of v0.14.3+ """Wrap a single binding without container into a length-1 `list`. Pass through multiple bindings as-is. Yell if the input format is invalid. - elts: `list` of bindings, either:: - [(k0, v0), ...] # multiple bindings contained in a tuple - [(k, v),] # single binding contained in a tuple also ok - [k, v] # special single binding format, missing tuple container + elts: `list` of bindings, one of:: + [(k0, v0), ...] # multiple bindings contained in a tuple + [(k, v),] # single binding contained in a tuple also ok + [k, v] # special single binding format, missing tuple container + [[k0, v0], ...] # v0.15.0+: accept also brackets (for consistency) + [[k, v]] # v0.15.0+ + [k0 << v0, ...] # v0.15.0+: accept also env-assignment syntax + [k << v] # v0.15.0+ where the ks and vs are AST nodes. - allow_call_in_name_position: used by let_syntax to allow template definitions; - in the call, the "function" is the template name, and the positional "parameters" - are the template parameters (which may then appear in the template body). - (Despite the name, this recognizes `Subscript` too, to support brackets.) + letsyntax_mode: used by let_syntax to allow template definitions. + This allows, beside a bare name `k`, the formats `k(a0, ...)` and `k[a0, ...]` + to appear in the variable-name position. """ - def isname(x): + def isname(tree): + # Note we don't accept hygienic captures. # The `Done` may be produced by expanded `@namemacro`s. - return type(x) is Name or (isinstance(x, Done) and isname(x.body)) - def iskey(x): - return (isname(x) or - (allow_call_in_name_position and ((type(x) is Call and isname(x.func)) or - (type(x) is Subscript and isname(x.value))))) - if len(elts) == 2 and iskey(elts[0]): + return type(tree) is Name or (isinstance(tree, Done) and isname(tree.body)) + def isbindingtarget(tree): + return (isname(tree) or + (letsyntax_mode and ((type(tree) is Call and isname(tree.func)) or + (type(tree) is Subscript and isname(tree.value))))) + def iskvpairbinding(lst): + return len(lst) == 2 and isbindingtarget(lst[0]) + def isenvassignbinding(tree): + if not (type(tree) is BinOp and type(tree.op) is LShift): + return False + return isbindingtarget(tree.left) + + if len(elts) == 1 and isenvassignbinding(elts[0]): # [k << v] + return [Tuple(elts=[elts[0].left, elts[0].right])] + if len(elts) == 2 and iskvpairbinding(elts): # [k, v] return [Tuple(elts=elts)] # TODO: `mcpyrate`: just `q[t[elts]]`? - if all((type(b) is Tuple and len(b.elts) == 2 and iskey(b.elts[0])) for b in elts): + if all((type(b) is Tuple and iskvpairbinding(b.elts)) for b in elts): # [(k0, v0), ...] return elts - raise SyntaxError("expected bindings to be ((k0, v0), ...) or a single (k, v)") # pragma: no cover + if all((type(b) is List and iskvpairbinding(b.elts)) for b in elts): # [[k0, v0], ...] + return [Tuple(elts=b.elts) for b in elts] + if all((isenvassign(b) and isbindingtarget(b.left)) for b in elts): # [k0 << v0, ...] + return [Tuple(elts=[b.left, b.right]) for b in elts] + raise SyntaxError("expected bindings to be `(k0, v0), ...`, `[k0, v0], ...`, or `k0 << v0, ...`, or a single `k, v`, or `k << v`") # pragma: no cover def isenvassign(tree): """Detect whether tree is an unpythonic ``env`` assignment, ``name << value``. @@ -132,17 +167,11 @@ def islet(tree, expanded=True): # otherwise we should have an expr macro invocation if not type(tree) is Subscript: return False + # Note we don't care about the bindings format here. # let[(k0, v0), ...][body] # let((k0, v0), ...)[body] # ^^^^^^^^^^^^^^^^^^ macro = tree.value - # let[(k0, v0), ...][body] - # let((k0, v0), ...)[body] - # ^^^^ - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - expr = tree.slice - else: - expr = tree.slice.value exprnames = ("let", "letseq", "letrec", "let_syntax", "abbrev") if type(macro) is Subscript and type(macro.value) is Name: s = macro.value.id @@ -156,6 +185,10 @@ def islet(tree, expanded=True): elif type(macro) is Name: s = macro.id if any(s == x for x in exprnames): + # let[(k0, v0), ...][body] + # let((k0, v0), ...)[body] + # ^^^^ + expr = _get_subscript_slice(tree) h = _ishaskellylet(expr) if h: return (h, s) @@ -174,29 +207,34 @@ def _ishaskellylet(tree): To detect the full expression including the ``let[]``, use ``islet`` instead. """ # let[((k0, v0), ...) in body] + # let[[(k0, v0), ...] in body] def maybeiscontentofletin(tree): return (type(tree) is Compare and len(tree.ops) == 1 and type(tree.ops[0]) is In and - type(tree.left) is Tuple) + type(tree.left) in (List, Tuple)) # let[body, where((k0, v0), ...)] + # let[body, where[(k0, v0), ...]] def maybeiscontentofletwhere(tree): - return type(tree) is Tuple and len(tree.elts) == 2 and type(tree.elts[1]) is Call + return type(tree) is Tuple and len(tree.elts) == 2 and type(tree.elts[1]) in (Call, Subscript) if maybeiscontentofletin(tree): bindings = tree.left - if all((type(b) is Tuple and len(b.elts) == 2 and type(b.elts[0]) is Name) - for b in bindings.elts): - return "in_expr" - # Single binding special case: let's not require a trailing comma. - # In this case, the wrapper tuple containing the bindings is missing. - # (For consistency of surface syntax with the other variants that don't - # require it, because they look like function calls in the AST.) - if len(bindings.elts) == 2 and type(bindings.elts[0]) is Name: + try: + # This could be a `let_syntax` or `abbrev` using the haskelly let-in syntax. + # We don't want to care about that, so we always use `letsyntax_mode=True`. + _ = canonize_bindings(_normalize_macroargs_node(bindings), letsyntax_mode=True) return "in_expr" + except SyntaxError: + pass elif maybeiscontentofletwhere(tree): - thecall = tree.elts[1] - if type(thecall.func) is Name and thecall.func.id == "where": - return "where_expr" + # TODO: account for as-imports here? (use isx()) + thewhere = tree.elts[1] + if type(thewhere) is Call: + if type(thewhere.func) is Name and thewhere.func.id == "where": + return "where_expr" + elif type(thewhere) is Subscript: + if type(thewhere.value) is Name and thewhere.value.id == "where": + return "where_expr" return False # invalid syntax for haskelly let # TODO: This would benefit from macro destructuring in the expander. @@ -228,17 +266,14 @@ def isdo(tree, expanded=True): return False return kind + # TODO: account for as-imports here? (use isx()) if not (type(tree) is Subscript and type(tree.value) is Name and any(tree.value.id == x for x in ("do", "do0"))): return False # TODO: detect also do[] with a single expression inside? (now requires a comma) - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - if not type(tree.slice) is Tuple: - return False - else: - if not type(tree.slice) is Index and type(tree.slice.value) is Tuple: - return False + if not type(_get_subscript_slice(tree)) is Tuple: + return False return tree.value.id @@ -272,7 +307,7 @@ class UnexpandedEnvAssignView: """ def __init__(self, tree): if not isenvassign(tree): - raise TypeError(f"expected a tree representing an unexpanded env-assignment, got {tree}") + raise TypeError(f"expected a tree representing an unexpanded env-assignment, got {unparse(tree)}") self._tree = tree def _getname(self): @@ -293,7 +328,6 @@ def _setvalue(self, newvalue): self._tree.right = newvalue value = property(fget=_getvalue, fset=_setvalue, doc="The value of the assigned var, as an AST. Writable.") -# TODO: kwargs support for let(x=42)[...] if implemented later class UnexpandedLetView: """Destructure a let form, writably. @@ -323,8 +357,19 @@ class UnexpandedLetView: ((k0, v0), ...) in body (body, where((k0, v0), ...)) + Finally, in any of these, the bindings subform can be in any of the formats: + + ((k0, v0), ...) + ([k0, v0], ...) + [(k0, v0), ...] + [[k0, v0], ...] + (k0 << v0, ...) + [k0 << v0, ...] + k, v + k << v + This is a data abstraction that hides the detailed structure of the AST, - since there are three alternate syntaxes that can be used for a ``let`` + since there are many alternate syntaxes that can be used for a ``let`` expression. For the decorator forms, ``tree`` should be the decorator call. In this case @@ -337,6 +382,9 @@ class UnexpandedLetView: ``(k, v)``, where ``k`` is an ``ast.Name``. Writing to ``bindings`` updates the original. + The bindings are always presented in this format, regardless of the actual + syntax used in the `let` form. + ``body`` (when available) is an AST representing a single expression. If it is an ``ast.List``, it means an implicit ``do[]`` (handled by the ``let`` expander), allowing a multiple-expression body. @@ -363,7 +411,7 @@ def __init__(self, tree): # from the given tree, to send them to the let transformer). h = _ishaskellylet(tree) if not h: - raise TypeError(f"expected a tree representing an unexpanded let, got {tree}") + raise TypeError(f"expected a tree representing an unexpanded let, got {unparse(tree)}") data = (h, None) # cannot detect mode, because no access to the surrounding Subscript AST node self._has_subscript_container = False self._tree = tree @@ -373,71 +421,66 @@ def __init__(self, tree): # Resolve the "content" node in the haskelly format. def _theexpr_ref(self): - if self._has_subscript_container: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - return self._tree.slice - else: - return self._tree.slice.value - else: - return self._tree + if self._has_subscript_container: # `let[(...) in ...]`, `let[..., where(...)]` + return _get_subscript_slice(self._tree) + return self._tree # `(...) in ...`, `..., where(...)` def _getbindings(self): t = self._type - if t == "decorator": # bare Subscript, dlet[...], blet[...] - if type(self._tree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 - return canonize_bindings(self._tree.args) - # Subscript as decorator (Python 3.9+) - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - theargs = self._tree.slice - else: - theargs = self._tree.slice.value - return canonize_bindings(theargs.elts) - elif t == "lispy_expr": - # Subscript inside a Subscript, (let[...])[...] - if type(self._tree.value) is Subscript: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - theargs = self._tree.value.slice.elts - else: - theargs = self._tree.value.slice.value.elts - # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 - # Call inside a Subscript, (let(...))[...] - else: # type(self._tree.value) is Call: - theargs = self._tree.value.args - return canonize_bindings(theargs) - else: # haskelly let, let[(...) in ...], let[..., where(...)] - theexpr = self._theexpr_ref() + if t in ("decorator", "lispy_expr"): + if t == "decorator": + # dlet[...], blet[...] + # dlet(...), blet(...) + thetree = self._tree + else: # "lispy_expr" + # (let[...])[...] + # (let(...))[...] + # ^^^^^^^^^^ + thetree = self._tree.value + + if type(thetree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + return canonize_bindings(thetree.args) + # Subscript + theargs = _get_subscript_slice(thetree) + return canonize_bindings(_normalize_macroargs_node(theargs)) + else: # haskelly let, `let[(...) in ...]`, `let[..., where(...)]` + theexpr = self._theexpr_ref() # `(...) in ...`, `..., where(...)` if t == "in_expr": - return canonize_bindings(theexpr.left.elts) + return canonize_bindings(_normalize_macroargs_node(theexpr.left)) elif t == "where_expr": - return canonize_bindings(theexpr.elts[1].args) + thewhere = theexpr.elts[1] + if type(thewhere) is Call: + return canonize_bindings(thewhere.args) + else: # Subscript + return canonize_bindings(_normalize_macroargs_node(_get_subscript_slice(thewhere))) + assert False def _setbindings(self, newbindings): t = self._type - if t == "decorator": - if type(self._tree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 - self._tree.args = newbindings + if t in ("decorator", "lispy_expr"): + if t == "decorator": + # dlet[...], blet[...] + # dlet(...), blet(...) + thetree = self._tree + else: # "lispy_expr" + # (let[...])[...] + # (let(...))[...] + # ^^^^^^^^^^ + thetree = self._tree.value + + if type(thetree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + thetree.args = newbindings return - # Subscript as decorator (Python 3.9+) - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - self._tree.slice.elts = newbindings - else: - self._tree.slice.value.elts = newbindings - elif t == "lispy_expr": - # Subscript inside a Subscript, (let[...])[...] - if type(self._tree.value) is Subscript: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - self._tree.value.slice.elts = newbindings - else: - self._tree.value.slice.value.elts = newbindings - # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 - # Call inside a Subscript, (let(...))[...] - else: # type(self._tree.value) is Call: - self._tree.value.args = newbindings + _set_subscript_slice(thetree, Tuple(elts=newbindings)) else: theexpr = self._theexpr_ref() if t == "in_expr": - theexpr.left.elts = newbindings + theexpr.left = Tuple(elts=newbindings) elif t == "where_expr": - theexpr.elts[1].args = newbindings + thewhere = theexpr.elts[1] + if type(thewhere) is Call: + thewhere.args = newbindings + else: # Subscript + _set_subscript_slice(thewhere, Tuple(elts=newbindings)) bindings = property(fget=_getbindings, fset=_setbindings, doc="The bindings subform of the let. Writable.") def _getbody(self): @@ -445,10 +488,7 @@ def _getbody(self): if t == "decorator": raise TypeError("the body of a decorator let form is the body of decorated function, not a subform of the let.") elif t == "lispy_expr": - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - return self._tree.slice - else: - return self._tree.slice.value + return _get_subscript_slice(self._tree) else: theexpr = self._theexpr_ref() if t == "in_expr": @@ -460,10 +500,7 @@ def _setbody(self, newbody): if t == "decorator": raise TypeError("the body of a decorator let form is the body of decorated function, not a subform of the let.") elif t == "lispy_expr": - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - self._tree.slice = newbody - else: - self._tree.slice.value = newbody + _set_subscript_slice(self._tree, newbody) else: theexpr = self._theexpr_ref() if t == "in_expr": @@ -499,24 +536,18 @@ def __init__(self, tree): self._implicit = False if not isdo(tree, expanded=False): if type(tree) is not List: # for implicit do[] - raise TypeError(f"expected a tree representing an unexpanded do, got {tree}") + raise TypeError(f"expected a tree representing an unexpanded do, got {unparse(tree)}") self._implicit = True self._tree = tree def _getbody(self): if not self._implicit: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - return self._tree.slice.elts - else: - return self._tree.slice.value.elts + return _get_subscript_slice(self._tree).elts else: return self._tree.elts def _setbody(self, newbody): if not self._implicit: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - self._tree.slice.elts = newbody - else: - self._tree.slice.value.elts = newbody + _set_subscript_slice(self._tree, Tuple(elts=newbody)) else: self._tree.elts = newbody body = property(fget=_getbody, fset=_setbody, doc="The body of the do. Writable.") @@ -569,7 +600,7 @@ class ExpandedLetView: def __init__(self, tree): data = islet(tree, expanded=True) if not data: - raise TypeError(f"expected a tree representing an expanded let, got {tree}") + raise TypeError(f"expected a tree representing an expanded let, got {unparse(tree)}") self._tree = tree self._type, self.mode = data if self._type not in ("expanded_decorator", "expanded_expr", "curried_decorator", "curried_expr"): @@ -737,7 +768,7 @@ class ExpandedDoView: def __init__(self, tree): t = isdo(tree, expanded=True) if not t: - raise TypeError(f"expected a tree representing an expanded do, got {tree}") + raise TypeError(f"expected a tree representing an expanded do, got {unparse(tree)}") self.curried = t.startswith("curried") self._tree = tree self.envname = self._deduce_envname() # stash at init time to prevent corruption by user mutations. diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index da5d42d9..e4151835 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -136,8 +136,7 @@ def let_syntax(tree, *, args, syntax, expander, **kw): if syntax == "expr": _let_syntax_expr_inside_out = partial(_let_syntax_expr, expand_inside=True) - return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr_inside_out, - allow_call_in_name_position=True) + return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr_inside_out, letsyntax_mode=True) else: # syntax == "block": with dyn.let(_macro_expander=expander): return _let_syntax_block(block_body=tree, expand_inside=True) @@ -173,7 +172,7 @@ def abbrev(tree, *, args, syntax, expander, **kw): if syntax == "expr": _let_syntax_expr_outside_in = partial(_let_syntax_expr, expand_inside=False) return _destructure_and_apply_let(tree, args, expander, _let_syntax_expr_outside_in, - allow_call_in_name_position=True) + letsyntax_mode=True) else: with dyn.let(_macro_expander=expander): return _let_syntax_block(block_body=tree, expand_inside=False) diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index 81d41d1a..dffe2f29 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -41,6 +41,8 @@ def validate(lst): test[validate(the[canonize_bindings(q[k0, v0].elts)])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings(q[((k0, v0),)].elts)])] # noqa: F821 test[validate(the[canonize_bindings(q[(k0, v0), (k1, v1)].elts)])] # noqa: F821 + test[validate(the[canonize_bindings([q[k0 << v0]])])] # noqa: F821, it's quoted. + test[validate(the[canonize_bindings(q[k0 << v0, k1 << v1].elts)])] # noqa: F821, it's quoted. # -------------------------------------------------------------------------------- # AST structure matching @@ -199,22 +201,66 @@ def testletdestructuring(testdata): test[unparse(view.body) == "(z * t)"] # lispy expr + testdata = q[let[x << 21, y << 2][y * x]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[[x, 21], [y, 2]][y * x]] # noqa: F821 + testletdestructuring(testdata) testdata = q[let[(x, 21), (y, 2)][y * x]] # noqa: F821 testletdestructuring(testdata) # haskelly let-in + testdata = q[let[[x << 21, y << 2] in y * x]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[(x << 21, y << 2) in y * x]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[[[x, 21], [y, 2]] in y * x]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[[(x, 21), (y, 2)] in y * x]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[([x, 21], [y, 2]) in y * x]] # noqa: F821 + testletdestructuring(testdata) testdata = q[let[((x, 21), (y, 2)) in y * x]] # noqa: F821 testletdestructuring(testdata) # haskelly let-where + testdata = q[let[y * x, where[x << 21, y << 2]]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[y * x, where(x << 21, y << 2)]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[y * x, where[[x, 21], [y, 2]]]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[y * x, where[(x, 21), (y, 2)]]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[let[y * x, where([x, 21], [y, 2])]] # noqa: F821 + testletdestructuring(testdata) testdata = q[let[y * x, where((x, 21), (y, 2))]] # noqa: F821 testletdestructuring(testdata) # disembodied haskelly let-in (just the content, no macro invocation) + testdata = q[[x << 21, y << 2] in y * x] # noqa: F821 + testletdestructuring(testdata) + testdata = q[(x << 21, y << 2) in y * x] # noqa: F821 + testletdestructuring(testdata) + testdata = q[[[x, 21], [y, 2]] in y * x] # noqa: F821 + testletdestructuring(testdata) + testdata = q[[(x, 21), (y, 2)] in y * x] # noqa: F821 + testletdestructuring(testdata) + testdata = q[([x, 21], [y, 2]) in y * x] # noqa: F821 + testletdestructuring(testdata) testdata = q[((x, 21), (y, 2)) in y * x] # noqa: F821 testletdestructuring(testdata) # disembodied haskelly let-where (just the content, no macro invocation) + testdata = q[y * x, where[x << 21, y << 2]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[y * x, where(x << 21, y << 2)] # noqa: F821 + testletdestructuring(testdata) + testdata = q[y * x, where[[x, 21], [y, 2]]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[y * x, where[(x, 21), (y, 2)]] # noqa: F821 + testletdestructuring(testdata) + testdata = q[y * x, where([x, 21], [y, 2])] # noqa: F821 + testletdestructuring(testdata) testdata = q[y * x, where((x, 21), (y, 2))] # noqa: F821 testletdestructuring(testdata) From d0b40935151838105ed1761cb001373c9a091dd0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 03:05:38 +0300 Subject: [PATCH 213/832] update changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d7d9a533..203fce74 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ **0.15.0** (in progress; updated 5 May 2021) - *"We say 'howdy' around these parts"* edition: -This edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. +Beside introducing **dialects** (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. **IMPORTANT**: From 6bd6cd2aee6bfc71500f64323ba4d0ce22c984f8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 03:38:23 +0300 Subject: [PATCH 214/832] update `let` subsystem docstrings --- unpythonic/syntax/letdo.py | 77 +++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 56072323..bc954942 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -92,7 +92,7 @@ def where(tree, *, syntax, **kw): Usage:: - let[body, where((k0, v0), ...)] + let[body, where[k0 << v0, ...]] Only meaningful for declaring the bindings in a let-where, for all expression-form let constructs: `let`, `letseq`, `letrec`, `let_syntax`, @@ -110,18 +110,18 @@ def let(tree, *, args, syntax, expander, **kw): Usage:: - let[(k0, v0), ...][body] - let[(k0, v0), ...][[body0, ...]] + let[k0 << v0, ...][body] + let[k0 << v0, ...][[body0, ...]] where ``body`` is an expression. The names bound by ``let`` are local; they are available in ``body``, and do not exist outside ``body``. Alternative haskelly syntax is also available:: - let[((k0, v0), ...) in body] - let[((k0, v0), ...) in [body0, ...]] - let[body, where((k0, v0), ...)] - let[[body0, ...], where((k0, v0), ...)] + let[[k0 << v0, ...] in body] + let[[k0 << v0, ...] in [body0, ...]] + let[body, where[k0 << v0, ...]] + let[[body0, ...], where[k0 << v0, ...]] For a body with multiple expressions, use an extra set of brackets, as shown above. This inserts a ``do``. Only the outermost extra brackets @@ -133,8 +133,9 @@ def let(tree, *, args, syntax, expander, **kw): Each ``name`` in the same ``let`` must be unique. - Assignment to let-bound variables is supported with syntax such as ``x << 42``. - This is an expression, performing the assignment, and returning the new value. + Rebinding of let-bound variables inside `body` is supported with `unpythonic` + env-assignment syntax, ``x << 42``. This is an expression, performing the + assignment, and returning the new value. In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see ``do`` for details. @@ -209,7 +210,7 @@ def dlet(tree, *, args, syntax, expander, **kw): Example:: - @dlet[(x, 0)] + @dlet[x << 0] def count(): x << x + 1 return x @@ -239,9 +240,9 @@ def dletseq(tree, *, args, syntax, expander, **kw): Example:: - @dletseq[(x, 1), - (x, x+1), - (x, x+2)] + @dletseq[x << 1, + x << x + 1, + x << x + 2] def g(a): return a + x assert g(10) == 14 @@ -258,8 +259,8 @@ def dletrec(tree, *, args, syntax, expander, **kw): Example:: - @dletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))] + @dletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), + oddp << (lambda x: (x != 0) and evenp(x - 1))] def f(x): return evenp(x) assert f(42) is True @@ -279,9 +280,9 @@ def blet(tree, *, args, syntax, expander, **kw): Example:: - @blet[(x, 21)] + @blet[x << 21] def result(): - return 2*x + return 2 * x assert result == 42 """ if syntax != "decorator": @@ -296,9 +297,9 @@ def bletseq(tree, *, args, syntax, expander, **kw): Example:: - @bletseq[(x, 1), - (x, x+1), - (x, x+2)] + @bletseq[x << 1, + x << x + 1, + x << x + 2] def result(): return x assert result == 4 @@ -315,8 +316,8 @@ def bletrec(tree, *, args, syntax, expander, **kw): Example:: - @bletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))] + @bletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), + oddp << (lambda x: (x != 0) and evenp(x - 1))] def result(): return evenp(42) assert result is True @@ -550,20 +551,20 @@ def _let_decorator_impl(bindings, body, mode, kind): def _dletseq_impl(bindings, body, kind): # What we want: # - # @dletseq[(x, 1), - # (x, x+1), - # (x, x+2)] + # @dletseq[x << 1, + # x << x + 1, + # x << x + 2] # def g(*args, **kwargs): # return x # assert g() == 4 # # --> # - # @dlet[(x, 1)] + # @dlet[x << 1] # def g(*args, **kwargs, e1): # original args from tree go to the outermost def - # @dlet[(x, x+1)] # on RHS, important for e1.x to be in scope + # @dlet[x << x + 1] # on RHS, important for e1.x to be in scope # def g2(*, e2): - # @dlet[(x, x+2)] + # @dlet[x << x + 2] # def g3(*, e3): # expansion proceeds from inside out # return e3.x # original args travel here by the closure property # return g3() @@ -752,14 +753,14 @@ def do(tree, *, syntax, expander, **kw): uses, the ambiguity does not arise. The transformation inserts not only the word ``do``, but also the outermost brackets. For example:: - let[(x, 1), - (y, 2)][[ + let[x << 1, + y << 2][[ [x, y]]] transforms to:: - let[(x, 1), - (y, 2)][do[[ # "do[" is inserted between the two opening brackets + let[x << 1, + y << 2][do[[ # "do[" is inserted between the two opening brackets [x, y]]]] # and its closing "]" is inserted here which already gets rid of the ambiguity. @@ -770,21 +771,21 @@ def do(tree, *, syntax, expander, **kw): names, if the same names appear in the ``do``:: do[local[x << 17], - let[(x, 23)][ + let[x << 23][ print(x)], # 23, the "x" of the "let" print(x)] # 17, the "x" of the "do" The reason we require local names to be declared is to allow write access to lexically outer environments from inside a ``do``:: - let[(x, 17)][ + let[x << 17][ do[x << 23, # no "local[...]"; update the "x" of the "let" local[y << 42], # "y" is local to the "do" print(x, y)]] With the extra bracket syntax, the latter example can be written as:: - let[(x, 17)][[ + let[x << 17][[ x << 23, local[y << 42], print(x, y)]] @@ -880,7 +881,7 @@ def examine(self, tree): lines = [] for j, expr in enumerate(tree.elts, start=1): # Despite the recursion, this will not trigger false positives for nested do[] expressions, - # because do[] is a second-pass macro, so they expand from inside out. + # because the transformers only operate at the top level of this do[]. expr, newnames = transform_localdefs(expr) expr, deletednames = transform_deletes(expr) if newnames and deletednames: @@ -891,7 +892,7 @@ def examine(self, tree): # Before transforming any further, check that there are no local[] or delete[] further in, where # they don't belong. This allows the error message to show the *untransformed* source code for - # the erroneous invocation. + # the erroneous invocation. These checkers respect the boundaries of any nested do[]. check_stray_localdefs(expr) check_stray_deletes(expr) @@ -925,7 +926,7 @@ def _do0(tree): def _implicit_do(tree): """Allow a sequence of expressions in expression position. - Apply ``do[]`` if ``tree`` is a ``List``, otherwise return ``tree`` as-is. + Insert a ``do[]`` if ``tree`` is a ``List``, otherwise return ``tree`` as-is. Hence, in user code, to represent a sequence of expressions, use brackets:: From 183478d0aa34713d5bfb6abd96d6127278a02a06 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 03:40:31 +0300 Subject: [PATCH 215/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index f8123a70..4d88b591 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -83,6 +83,8 @@ # TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. +# TODO: 0.16: Add call-macros to `mcpyrate`. This allows the whole expression of `kw()`/`where()` to be detected as a macro invocation. (First, think whether this is a good idea.) + # TODO: Something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead. # TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... From 1a007f716cfcd6046d77a88b819d59479450b049 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 12:37:25 +0300 Subject: [PATCH 216/832] update let macro docs --- doc/macros.md | 228 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 136 insertions(+), 92 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 6002fa30..62d84d2f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -98,95 +98,140 @@ Macros that introduce new ways to bind identifiers. ### ``let``, ``letseq``, ``letrec`` as macros +**Changed in v0.15.0.** *Added support for env-assignment syntax in the bindings subform. For consistency with other env-assignments, this is now the preferred syntax to establish let bindings. Additionally, the old lispy syntax now accepts also brackets, for consistency with the use of brackets for macro invocations.* + Properly lexically scoped ``let`` constructs, no boilerplate: ```python from unpythonic.syntax import macros, let, letseq, letrec -let[(x, 17), # parallel binding, i.e. bindings don't see each other - (y, 23)][ +let[x << 17, # parallel binding, i.e. bindings don't see each other + y << 23][ print(x, y)] -letseq[(x, 1), # sequential binding, i.e. Scheme/Racket let* - (y, x+1)][ +letseq[x << 1, # sequential binding, i.e. Scheme/Racket let* + y << x+1][ print(x, y)] -letrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding, sequentially evaluated - (oddp, lambda x: (x != 0) and evenp(x - 1))][ +letrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding, sequentially evaluated + oddp << (lambda x: (x != 0) and evenp(x - 1))][ print(evenp(42))] ``` -As seen in the examples, the syntax is similar to [``unpythonic.lispylet``](../doc/features.md#lispylet-alternative-syntax). Assignment to variables in the environment is supported via the left-shift syntax ``x << 42``. +Even with just one binding, the syntax remains the same: + +```python +let[x << 21][2 * x] +``` + +There must be at least one binding; `let[][...]` is a syntax error, since Python's parser rejects an empty subscript slice. + +Bindings are established using the `unpythonic` *env-assignment* syntax, ``name << value``. The let bindings can be rebound in the body with the same env-assignment syntax, e.g. ``x << 42``. -The bindings are given as macro arguments as ``((name, value), ...)``, the body goes into the ``[...]``. +The same syntax for the bindings subform is used by: -#### Alternate syntaxes +- ``let``, ``letseq``, ``letrec`` (expressions) +- ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec`` (decorators) +- ``let_syntax``, ``abbrev`` (expression mode) + + +#### Haskelly let-in, let-where The following Haskell-inspired, perhaps more pythonic alternate syntaxes are also available: ```python -let[((x, 21), - (y, 17), - (z, 4)) in +let[[x << 21, + y << 17, + z << 4] in x + y + z] let[x + y + z, - where((x, 21), - (y, 17), - (z, 4))] + where[x << 21, + y << 17, + z << 4]] + +let[[x << 21] in 2 * x] +let[2 * x, where[x << 21]] ``` -These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``. +These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the ``...`` in `let[...]`. + +Note the bindings subform is always enclosed by brackets.

Semantically, these do the exact same thing as the original lispy syntax: >The bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. > ->These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the original lispy syntax, because there the body is in any case placed differently. +>These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the formats where the bindings subform is given in the macro arguments part, because there the body is in any case placed differently (it's the body of the function being decorated). > ->In the first variant above (the *let-in*), note the bindings block still needs the outer parentheses. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make it refer to all of the bindings, the bindings block must be parenthesized. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your parentheses. +>In the first variant above (the *let-in*), note that even there, the bindings block needs the brackets. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make the ``in`` refer to all of the bindings, the bindings block must be bracketed. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your brackets. > ->In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression.) +>In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression. It's not only syntactically valid Python, it's also syntactically valid English (at least for mathematicians).)
-#### Special syntax for one binding +#### Alternate syntaxes for the bindings subform + +**Changed in v0.15.0.** -If there is only one binding, to make the syntax more pythonic, the outer parentheses may be omitted in the bindings block of the **expr forms** of: +Beginning with v0.15.0, the env-assignment syntax presented above is the preferred syntax to establish let bindings, for consistency with other env-assignments. (Let variables live in an `env`, which is created by the `let`.) -- ``let``, ``letseq``, ``letrec`` -- ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec`` -- ``let_syntax``, ``abbrev`` +There is also an alternate, lispy notation for the bindings subform, where each name-value pair is given using brackets: ```python -let[x, 21][2*x] -let[(x, 21) in 2*x] -let[2*x, where(x, 21)] +let[[x, 42], [y, 9001]][...] +let[[[x, 42], [y, 9001]] in ...] +let[..., where[[x, 42], [y, 9001]]] + +# one-binding special case: outer brackets not needed +let[x, 42][...] +let[[x, 42] in ...] +let[..., where[x, 42]] ``` -This is valid also in the *let-in* variant, because there is still one set of parentheses enclosing the bindings block. +This is similar in spirit to the notation used in v0.14.3 and earlier. + +Actually, for backwards compatibility, we still support some use of parentheses instead of brackets in the bindings subform. The following formats, used in versions of `unpythonic` up to v0.14.3, are still accepted: + +```python +let((x, 42), (y, 9001))[...] +let[((x, 42), (y, 9001)) in ...] +let[..., where((x, 42), (y, 9001))] + +# one-binding special case: outer parentheses not needed +let(x, 42)[...] +let[(x, 42) in ...] +let[..., where(x, 42)] +``` + +Even though an expr macro invocation itself is always denoted using brackets, as of `unpythonic` v0.15.0 parentheses can still be used *to pass macro arguments*, hence ``let(...)[...]`` is still accepted. The code that interprets the AST for the let bindings accepts both lists and tuples for each key-value pair, and the top-level container for the bindings subform in a let-in or let-where can be either list or tuple, so whether brackets or parentheses are used does not matter there, either. + +Still, brackets are now the preferred delimiter, for consistency between the bindings and body subforms. + +We plan to drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum Python version supported. The reason we will wait that long is that up to Python 3.8, decorators cannot be subscripted. Up to Python 3.8, `@dlet[x, 42]` is rejected by Python's parser, whereas `@dlet(x, 42)` is accepted. + +The issue has been fixed in Python 3.9. If you already only use 3.9 and later, please prefer brackets to pass macro arguments. -This is essentially special-cased in the ``let`` expander. (If interested in the technical details, look at ``unpythonic.syntax.letdoutil.UnexpandedLetView``, which performs the destructuring. See also ``unpythonic.syntax.__init__.let``; the macro expander itself already destructures the original lispy syntax when the macro is invoked.) #### Multiple expressions in body The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body ([like in `multilambda`](#multilambda-supercharge-your-lambdas)): ```python -let[(x, 1), - (y, 2)][[ # note extra [ +let[x << 1, + y << 2][[ # note extra [ y << x + y, print(y)]] -let[((x, 1), # v0.12.0+ - (y, 2)) in +let[[x << 1, + y << 2] in [y << x + y, # body starts here print(y)]] -let[[y << x + y, # v0.12.0+ +let[[y << x + y, print(y)], # body ends here - where((x, 1), - (y, 2))] + where[x << 1, + y << 2]] ``` The let macros implement this by inserting a ``do[...]`` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see [``do`` for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). @@ -194,17 +239,17 @@ The let macros implement this by inserting a ``do[...]`` (see below). In a multi Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a ``let`` form with only one body expression, use three sets of brackets: ```python -let[(x, 1), - (y, 2)][[ +let[x << 1, + y << 2][[ [x, y]]] -let[((x, 1), # v0.12.0+ - (y, 2)) in +let[[x << 1, + y << 2] in [[x, y]]] -let[[[x, y]], # v0.12.0+ - where((x, 1), - (y, 2))] +let[[[x, y]], + where[x << 1, + y << 2]] ``` The outermost brackets delimit the ``let`` form, the middle ones activate multiple-expression mode, and the innermost ones denote a list. @@ -212,17 +257,17 @@ The outermost brackets delimit the ``let`` form, the middle ones activate multip Only brackets are affected; parentheses are interpreted as usual, so returning a literal tuple works as expected: ```python -let[(x, 1), - (y, 2)][ +let[x << 1, + y << 2][ (x, y)] -let[((x, 1), # v0.12.0+ - (y, 2)) in +let[[x << 1, + y << 2] in (x, y)] -let[(x, y), # v0.12.0+ - where((x, 1), - (y, 2))] +let[(x, y), + where[x << 1, + y << 2]] ``` #### Notes @@ -234,9 +279,9 @@ The main difference of the `let` family to Python's own named expressions (a.k.a Nesting utilizes an inside-out macro expansion order: ```python -letrec[(z, 1)][[ +letrec[z << 1][[ print(z), - letrec[(z, 2)][ + letrec[z << 2][ print(z)]]] ``` @@ -254,42 +299,42 @@ Examples: ```python from unpythonic.syntax import macros, dlet, dletseq, dletrec, blet, bletseq, bletrec -@dlet[(x, 0)] +@dlet[x << 0] # up to Python 3.8, use `@dlet(x << 0)` instead def count(): x << x + 1 return x assert count() == 1 assert count() == 2 -@dletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))] +@dletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), + oddp << (lambda x: (x != 0) and evenp(x - 1))] def f(x): return evenp(x) assert f(42) is True assert f(23) is False -@dletseq[(x, 1), - (x, x+1), - (x, x+2)] +@dletseq[x << 1, + x << x + 1, + x << x + 2] def g(a): return a + x assert g(10) == 14 # block versions: the def takes no arguments, runs immediately, and is replaced by the return value. -@blet[(x, 21)] +@blet[x << 21] def result(): return 2*x assert result == 42 -@bletrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))] +@bletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), + oddp << (lambda x: (x != 0) and evenp(x - 1))] def result(): return evenp(42) assert result is True -@bletseq[(x, 1), - (x, x+1), - (x, x+2)] +@bletseq[x << 1, + x << x + 1, + x << x + 2] def result(): return x assert result == 4 @@ -306,31 +351,31 @@ As an exception to the rule, for the purposes of the scope analysis performed by To clarify, here's a sampling from the unit tests: ```python -@dlet[(x, "the env x")] +@dlet[x << "the env x"] def f(): return x assert f() == "the env x" -@dlet[(x, "the env x")] +@dlet[x << "the env x"] def f(): x = "the local x" return x assert f() == "the local x" -@dlet[(x, "the env x")] +@dlet[x << "the env x"] def f(): return x x = "the unused local x" assert f() == "the env x" x = "the global x" -@dlet[(x, "the env x")] +@dlet[x << "the env x"] def f(): global x return x assert f() == "the global x" -@dlet[(x, "the env x")] +@dlet[x << "the env x"] def f(): x = "the local x" del x # deleting a local, ok! @@ -339,7 +384,7 @@ assert f() == "the env x" try: x = "the global x" - @dlet[(x, "the env x")] + @dlet[x << "the env x"] def f(): global x del x # ignored by unpythonic's scope analysis, deletion of globals is too dynamic @@ -365,29 +410,28 @@ def verylongfunctionname(x=1): return x # works as an expr macro -y = let_syntax[(f, verylongfunctionname)][[ # extra brackets: implicit do in body +y = let_syntax[f << verylongfunctionname][[ # extra brackets: implicit do in body print(f()), f(5)]] assert y == 5 -y = let_syntax[(f[a], verylongfunctionname(2*a))][[ # template with formal parameter "a" +y = let_syntax[f[a] << verylongfunctionname(2*a)][[ # template with formal parameter "a" print(f[2]), f[3]]] assert y == 6 -# v0.12.0+ -y = let_syntax[((f, verylongfunctionname)) in +y = let_syntax[[f << verylongfunctionname] in [print(f()), f(5)]] y = let_syntax[[print(f()), f(5)], - where((f, verylongfunctionname))] -y = let_syntax[((f[a], verylongfunctionname(2*a))) in + where[f << verylongfunctionname]] +y = let_syntax[[f[a] << verylongfunctionname(2*a)] in [print(f[2]), f[3]]] y = let_syntax[[print(f[2]), f[3]], - where((f[a], verylongfunctionname(2*a)))] + where[f[a] << verylongfunctionname(2*a)]] # works as a block macro with let_syntax: @@ -441,8 +485,8 @@ After macro expansion completes, ``let_syntax`` has zero runtime overhead; it co > >Within each step, the substitutions are applied **in definition order**: > -> - If the bindings are ``((x, y), (y, z))``, then an ``x`` at the use site transforms to ``z``. So does a ``y`` at the use site. -> - But if the bindings are ``((y, z), (x, y))``, then an ``x`` at the use site transforms to ``y``, and only an explicit ``y`` at the use site transforms to ``z``. +> - If the bindings are ``[x << y, y << z]``, then an ``x`` at the use site transforms to ``z``. So does a ``y`` at the use site. +> - But if the bindings are ``[y << z, x << y]``, then an ``x`` at the use site transforms to ``y``, and only an explicit ``y`` at the use site transforms to ``z``. > >Even in block templates, arguments are always expressions, because invoking a template uses the subscript syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above.
@@ -457,14 +501,12 @@ When used as an expr macro, all bindings are registered first, and then the body The ``abbrev`` macro is otherwise exactly like ``let_syntax``, but it expands outside-in. Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the ``abbrev`` itself expands before any macros invoked in its body. This allows things like: ```python -abbrev[(m, macrowithverylongname)][ +abbrev[m << macrowithverylongname][ m[tree1] if m[tree2] else m[tree3]] - -# v0.12.0+ -abbrev[((m, macrowithverylongname)) in +abbrev[[m << macrowithverylongname] in m[tree1] if m[tree2] else m[tree3]] abbrev[m[tree1] if m[tree2] else m[tree3], - where((m, macrowithverylongname))] + where[m << macrowithverylongname]] ``` which can be useful when writing macros. @@ -517,7 +559,7 @@ A ``local`` declaration comes into effect in the expression following the one wh ```python result = [] -let[(lst, [])][[result.append(lst), # the let "lst" +let[lst << []][[result.append(lst), # the let "lst" local[lst << lst + [1]], # LHS: do "lst", RHS: let "lst" result.append(lst)]] # the do "lst" assert result == [[], [1]] @@ -557,13 +599,13 @@ with multilambda: echo = lambda x: [print(x), x] assert echo("hi there") == "hi there" - count = let[(x, 0)][ + count = let[x << 0][ lambda: [x << x + 1, # x belongs to the surrounding let x]] assert count() == 1 assert count() == 2 - test = let[(x, 0)][ + test = let[x << 0][ lambda: [x << x + 1, local[y << 42], # y is local to the implicit do (x, y)]] @@ -594,14 +636,14 @@ from unpythonic.syntax import macros, namedlambda with namedlambda: f = lambda x: x**3 # assignment: name as "f" assert f.__name__ == "f" - gn, hn = let[(x, 42), (g, None), (h, None)][[ + gn, hn = let[x << 42, g << None, h << None][[ g << (lambda x: x**2), # env-assignment: name as "g" h << f, # still "f" (no literal lambda on RHS) (g.__name__, h.__name__)]] assert gn == "g" assert hn == "f" - foo = let[(f7, lambda x: x) in f7] # let-binding: name as "f7" + foo = let[[f7 << (lambda x: x)] in f7] # let-binding: name as "f7" def foo(func1, func2): assert func1.__name__ == "func1" @@ -626,10 +668,12 @@ The naming is performed using the function ``unpythonic.misc.namelambda``, which - Single-item assignment to a local name, ``f = lambda ...: ...`` + - **Added in v0.15.0**: Named expressions (a.k.a. walrus operator, Python 3.8+), ``f := lambda ...: ...`` + - Expression-assignment to an unpythonic environment, ``f << (lambda ...: ...)`` - Env-assignments are processed lexically, just like regular assignments. - - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). + - Let bindings, ``let[[f << (lambda ...: ...)] in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). - **Added in v0.14.2**: Named argument in a function call, as in ``foo(f=lambda ...: ...)``. @@ -710,7 +754,7 @@ So if we want to use a lambda, we have to create an ``env``, so that we can writ ```python def foo(n0): - return let[(n, n0) in + return let[[n << n0] in (lambda i: n << n + i)] ``` @@ -1580,7 +1624,7 @@ aif[2*21, print("it is falsey")] ``` -Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the ``aif``, and does not exist outside the ``aif``. +Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the ``then`` and ``otherwise`` parts of ``aif``, and anywhere else raises a syntax error at macro expansion time. Any part of ``aif`` may have multiple expressions by surrounding it with brackets (implicit ``do[]``): From 3c731c01b4c92fdccc541ecd2b8b8c0de2d0dc5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 13:31:47 +0300 Subject: [PATCH 217/832] fix `let` bindings update bug in `namedlambda`, `prefix` This bug was introduced when extending the bindings declaration syntax for 0.15.0. --- unpythonic/syntax/lambdatools.py | 3 +++ unpythonic/syntax/letdoutil.py | 13 ++++++++++++- unpythonic/syntax/prefix.py | 3 +++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 322961e4..c385c082 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -301,12 +301,15 @@ def transform(self, tree): return tree # don't recurse! if islet(tree, expanded=False): # let bindings view = UnexpandedLetView(tree) + newbindings = [] for b in view.bindings: b.elts[1], thelambda, match = nameit(getname(b.elts[0]), b.elts[1]) if match: thelambda.body = self.visit(thelambda.body) else: b.elts[1] = self.visit(b.elts[1]) + newbindings.append(b) + view.bindings = newbindings # write the new bindings (important!) view.body = self.visit(view.body) return tree # assumption: no one left-shifts by a literal lambda :) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 0796561c..14b115dd 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -383,7 +383,18 @@ class UnexpandedLetView: the original. The bindings are always presented in this format, regardless of the actual - syntax used in the `let` form. + syntax used in the `let` form. Updates must also be done in this format. + + **CAUTION**: The bindings are only written to the AST when you assign to + the ``bindings`` attribute; in-place updates might not have any effect, + depending on the actual syntax in the original AST (i.e. whether what you + got was actually a reformatted copy). You'll likely want something like this:: + + newbindings = [] + for b in view.bindings: + b.elts[1] = ... # modify it + newbindings.append(b) + view.bindings = newbindings # write the updated bindings to the AST ``body`` (when available) is an AST representing a single expression. If it is an ``ast.List``, it means an implicit ``do[]`` (handled by the diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 6f841d7c..31ffb2a7 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -162,11 +162,14 @@ def transform(self, tree): # let and do have not expanded yet when prefix runs (better that way!). if islet(tree, expanded=False): view = UnexpandedLetView(tree) + newbindings = [] for binding in view.bindings: if type(binding) is not Tuple: raise SyntaxError("prefix: expected a tuple in let binding position") # pragma: no cover _, value = binding.elts # leave name alone, recurse into value binding.elts[1] = self.visit(value) + newbindings.append(binding) + view.bindings = newbindings # write the new bindings (important!) if view.body: view.body = self.visit(view.body) return tree From 6d6401d54ba09febd1ad3ca85267c365783ec5a9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 13:33:12 +0300 Subject: [PATCH 218/832] update examples to use new let bindings syntax Also, test both that syntax and the classic one in the tests. --- README.md | 15 +- doc/dialects/lispython.md | 2 +- doc/dialects/pytkell.md | 4 +- unpythonic/dialects/tests/test_lispython.py | 14 +- unpythonic/dialects/tests/test_listhell.py | 12 +- unpythonic/dialects/tests/test_pytkell.py | 16 +- unpythonic/syntax/lambdatools.py | 8 +- unpythonic/syntax/lazify.py | 20 +- unpythonic/syntax/letdo.py | 6 +- unpythonic/syntax/letdoutil.py | 63 +++--- unpythonic/syntax/tests/test_autoref.py | 4 +- unpythonic/syntax/tests/test_conts.py | 4 +- unpythonic/syntax/tests/test_lambdatools.py | 10 +- unpythonic/syntax/tests/test_lazify.py | 20 +- unpythonic/syntax/tests/test_letdo.py | 232 ++++++++++++-------- unpythonic/syntax/tests/test_letdoutil.py | 58 ++++- unpythonic/syntax/tests/test_tco.py | 14 +- 17 files changed, 294 insertions(+), 208 deletions(-) diff --git a/README.md b/README.md index c9211976..bbf00f76 100644 --- a/README.md +++ b/README.md @@ -445,13 +445,13 @@ As usual in test frameworks, the testing constructs behave somewhat like `assert ```python from unpythonic.syntax import macros, let, letseq, letrec -x = let[((a, 1), (b, 2)) in a + b] -y = letseq[((c, 1), # LET SEQuential, like Scheme's let* - (c, 2 * c), - (c, 2 * c)) in +x = let[[a << 1, b << 2] in a + b] +y = letseq[[c << 1, # LET SEQuential, like Scheme's let* + c << 2 * c, + c << 2 * c] in c] -z = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECursive, like in Scheme - (oddp, lambda x: (x != 0) and evenp(x - 1))) +z = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECursive, like in Scheme + oddp << (lambda x: (x != 0) and evenp(x - 1))] in evenp(42)] ``` @@ -462,7 +462,8 @@ z = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECursiv ```python from unpythonic.syntax import macros, dlet -@dlet((x, 0)) # let-over-lambda for Python +# Up to Python 3.8, use `@dlet(x << 0)` instead +@dlet[x << 0] # let-over-lambda for Python def count(): return x << x + 1 # `name << value` rebinds in the let env assert count() == 1 diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 2b5f4c78..60bd370d 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -171,7 +171,7 @@ assert f(5) == 17 Even Lispython can do no better than this let-over-lambda (here using the haskelly let-in syntax to establish let-bindings): ```python -foo = lambda n0: let[(n, n0) in +foo = lambda n0: let[[n << n0] in (lambda i: n << n + i)] ``` diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index 39e979e3..b91ad174 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -61,9 +61,9 @@ assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), factorials = scanl(mul, 1, s(1, 2, ...)) # 0!, 1!, 2!, ... assert last(take(6, factorials)) == 120 -x = let[(a, 21) in 2 * a] +x = let[[a << 21] in 2 * a] assert x == 42 -x = let[2 * a, where(a, 21)] +x = let[2 * a, where[a << 21]] assert x == 42 ``` diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index cf6ac336..dd7cbf9b 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -32,14 +32,14 @@ def runtests(): test[cdr(c) == 2] # noqa: F821 test[ll(1, 2, 3) == llist((1, 2, 3))] # noqa: F821 - # all unpythonic.syntax let[], letseq[], letrec[] constructs are builtins + # all unpythonic.syntax let[], letseq[], letrec[] constructs are considered dialect builtins # (including the decorator versions, let_syntax and abbrev) - x = let[(a, 21) in 2 * a] # noqa: F821 + x = let[[a << 21] in 2 * a] # noqa: F821 test[x == 42] - x = letseq[((a, 1), # noqa: F821 - (a, 2 * a), # noqa: F821 - (a, 2 * a)) in # noqa: F821 + x = letseq[[a << 1, # noqa: F821 + a << 2 * a, # noqa: F821 + a << 2 * a] in # noqa: F821 a] # noqa: F821 test[x == 4] @@ -62,8 +62,8 @@ def f(k, acc): test[fact(4) == 24] fact(5000) # no crash (and correct result, since Python uses bignums transparently) - t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 + t = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp << (lambda x:(x != 0) and evenp(x - 1))] in # noqa: F821 evenp(10000)] # no crash # noqa: F821 test[t is True] diff --git a/unpythonic/dialects/tests/test_listhell.py b/unpythonic/dialects/tests/test_listhell.py index ce2a0cb9..6d1f8283 100644 --- a/unpythonic/dialects/tests/test_listhell.py +++ b/unpythonic/dialects/tests/test_listhell.py @@ -77,7 +77,7 @@ def g(*args, **kwargs): test[a == 200 and b == 100] with testset("transform of let bindings"): - # the prefix syntax leaves alone the let binding syntax ((name0, value0), ...) + # the prefix syntax leaves alone the let binding syntax even when using tuples, ((name0, value0), ...) a = let[(x, 42)][x << x + 1] test[a == 43] @@ -87,6 +87,14 @@ def double(x): a = let[(x, (double, 21))][x << x + 1] test[a == 43] + # As of v0.15.0, the preferred let bindings syntax is env-assignment, + # so these examples become: + a = let[x << 42][x << x + 1] + test[a == 43] + + a = let[x << (double, 21)][x << x + 1] + test[a == 43] + # similarly, the prefix syntax leaves the "body tuple" of a do alone # (syntax, not semantically a tuple), but recurses into it: with testset("transform of do body"): @@ -96,7 +104,7 @@ def double(x): test[a == 6] # the extra bracket syntax (implicit do) has no danger of confusion, as it's a list, not tuple - a = let[(x, 3)][[ + a = let[x << 3][[ 1, 2, (double, x)]] diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index 894a3d05..55fb016e 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -44,8 +44,8 @@ def addfirst2(a, b, c): # let-bindings are auto-lazified with test["y is unused, so it should not be evaluated"]: - x = let[((x, 42), # noqa: F821 - (y, 1 / 0)) in x] # noqa: F821 + x = let[[x << 42, # noqa: F821 + y << 1 / 0] in x] # noqa: F821 return x == 42 # access `x`, to force the promise # assignments are not (because they can imperatively update existing names) @@ -91,21 +91,21 @@ def f(a, b): with testset("let constructs"): # let-in - x = let[(a, 21) in 2 * a] # noqa: F821 + x = let[[a << 21] in 2 * a] # noqa: F821 test[x == 42] - x = let[((a, 21), # noqa: F821 - (b, 17)) in # noqa: F821 + x = let[[a << 21, # noqa: F821 + b << 17] in # noqa: F821 2 * a + b] # noqa: F821 test[x == 59] # let-where - x = let[2 * a, where(a, 21)] # noqa: F821 + x = let[2 * a, where[a << 21]] # noqa: F821 test[x == 42] x = let[2 * a + b, # noqa: F821 - where((a, 21), # noqa: F821 - (b, 17))] # noqa: F821 + where[a << 21, # noqa: F821 + b << 17]] # noqa: F821 test[x == 59] # nondeterministic evaluation (essentially do-notation in the List monad) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index c385c082..bcb29d6b 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -51,7 +51,7 @@ def multilambda(tree, *, syntax, expander, **kw): echo = lambda x: [print(x), x] assert echo("hi there") == "hi there" - count = let[(x, 0)][ + count = let[x << 0][ lambda: [x << x + 1, x]] assert count() == 1 @@ -93,7 +93,7 @@ def namedlambda(tree, *, syntax, expander, **kw): - Assignments to unpythonic environments, ``f << (lambda ...: ...)`` - - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any + - Let bindings, ``let[[f << (lambda ...: ...)] in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). @@ -105,12 +105,12 @@ def namedlambda(tree, *, syntax, expander, **kw): with namedlambda: f = lambda x: x**3 # assignment: name as "f" - let[(x, 42), (g, None), (h, None)][[ + let[x << 42, g << None, h << None][[ g << (lambda x: x**2), # env-assignment: name as "g" h << f, # still "f" (no literal lambda on RHS) (g(x), h(x))]] - foo = let[(f7, lambda x: x) in f7] # let-binding: name as "f7" + foo = let[[f7 << (lambda x: x)] in f7] # let-binding: name as "f7" The naming is performed using the function ``unpythonic.misc.namelambda``, which will update ``__name__``, ``__qualname__`` and ``__code__.co_name``. diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index ea4ae8b6..fadcac2d 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -309,7 +309,7 @@ def f(lst): with lazify: lst = [] for x in range(3): - lst.append(let[(y, x) in lazy[y]]) + lst.append(let[[y << x] in lazy[y]]) print(lst[0]) # 0 print(lst[1]) # 1 print(lst[2]) # 2 @@ -359,15 +359,15 @@ def add2first(a, b, c): def f(a, b): return a - assert let[((c, 42), - (d, 1/0)) in f(c)(d)] == 42 - assert letrec[((c, 42), - (d, 1/0), - (e, 2*c)) in f(e)(d)] == 84 - - assert letrec[((c, 42), - (d, 1/0), - (e, 2*c)) in [local[x << f(e)(d)], + assert let[[c << 42, + d << 1/0] in f(c)(d)] == 42 + assert letrec[[c << 42, + d << 1/0, + e << 2*c] in f(e)(d)] == 84 + + assert letrec[[c << 42, + d << 1/0, + e << 2*c] in [local[x << f(e)(d)], x/4]] == 21 Works also with continuations. Rules: diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index bc954942..b4a5c3e8 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -62,7 +62,7 @@ # for the macro arguments part. # # So when `args` is empty, this function assumes haskelly let syntax -# `let[(...) in ...]` or `let[..., where(...)]`. In these cases, +# `let[[...] in ...]` or `let[..., where[...]]`. In these cases, # both the bindings and the body reside inside the brackets (i.e., # in the AST contained in the `tree` argument). # @@ -79,7 +79,7 @@ def _destructure_and_apply_let(tree, args, macro_expander, let_transformer, lets if args: bs = canonize_bindings(args, letsyntax_mode=letsyntax_mode) return let_transformer(bindings=bs, body=tree) - # haskelly syntax, let[(...) in ...], let[..., where(...)] + # haskelly syntax, let[[...] in ...], let[..., where[...]] view = UnexpandedLetView(tree) # note "tree" here is only the part inside the brackets return let_transformer(bindings=view.bindings, body=view.body) @@ -100,7 +100,7 @@ def where(tree, *, syntax, **kw): """ if syntax != "name": raise SyntaxError("where (unpythonic.syntax.letdo.where) is a name macro only") # pragma: no cover - raise SyntaxError("where() is only meaningful in a let[body, where((k0, v0), ...)]") # pragma: no cover + raise SyntaxError("where (unpythonic.syntax.letdo.where) is only meaningful in a let[body, where[k0 << v0, ...]]") # pragma: no cover @parametricmacro def let(tree, *, args, syntax, expander, **kw): diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 14b115dd..b784d6ad 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -153,7 +153,7 @@ def islet(tree, expanded=True): return (f"{kind}_decorator", mode) # this call was generated by _let_decorator_impl else: return (f"{kind}_expr", mode) # this call was generated by _let_expr_impl - # dlet[(k0, v0), ...] (usually in a decorator list) + # dlet[k0 << v0, ...] (usually in a decorator list) deconames = ("dlet", "dletseq", "dletrec", "blet", "bletseq", "bletrec") if type(tree) is Subscript and type(tree.value) is Name: # could be a Subscript decorator (Python 3.9+) @@ -168,8 +168,8 @@ def islet(tree, expanded=True): if not type(tree) is Subscript: return False # Note we don't care about the bindings format here. - # let[(k0, v0), ...][body] - # let((k0, v0), ...)[body] + # let[k0 << v0, ...][body] + # let(k0 << v0, ...)[body] # ^^^^^^^^^^^^^^^^^^ macro = tree.value exprnames = ("let", "letseq", "letrec", "let_syntax", "abbrev") @@ -185,8 +185,8 @@ def islet(tree, expanded=True): elif type(macro) is Name: s = macro.id if any(s == x for x in exprnames): - # let[(k0, v0), ...][body] - # let((k0, v0), ...)[body] + # let[k0 << v0, ...][body] + # let(k0 << v0, ...)[body] # ^^^^ expr = _get_subscript_slice(tree) h = _ishaskellylet(expr) @@ -201,19 +201,19 @@ def _ishaskellylet(tree): In other words, detect the part inside the brackets in:: - let[((k0, v0), ...) in body] - let[body, where((k0, v0), ...)] + let[[k0 << v0, ...] in body] + let[body, where[k0 << v0, ...]] To detect the full expression including the ``let[]``, use ``islet`` instead. """ - # let[((k0, v0), ...) in body] - # let[[(k0, v0), ...] in body] + # let[[k0 << v0, ...] in body] + # let[(k0 << v0, ...) in body] def maybeiscontentofletin(tree): return (type(tree) is Compare and len(tree.ops) == 1 and type(tree.ops[0]) is In and type(tree.left) in (List, Tuple)) - # let[body, where((k0, v0), ...)] - # let[body, where[(k0, v0), ...]] + # let[body, where[k0 << v0, ...]] + # let[body, where(k0 << v0, ...)] def maybeiscontentofletwhere(tree): return type(tree) is Tuple and len(tree.elts) == 2 and type(tree.elts[1]) in (Call, Subscript) @@ -339,34 +339,30 @@ class UnexpandedLetView: **Supported formats**:: - dlet[(k0, v0), ...] # decorator - let[(k0, v0), ...][body] # lispy expression - let[((k0, v0), ...) in body] # haskelly expression - let[body, where((k0, v0), ...)] # haskelly expression, inverted - - Lispy expressions are supported also using the old parenthesis syntax - to pass macro arguments:: - - let((k0, v0), ...)[body] # lispy expression + dlet[k0 << v0, ...] # decorator + let[k0 << v0, ...][body] # lispy expression + let[[k0 << v0, ...] in body] # haskelly expression + let[body, where[k0 << v0, ...]] # haskelly expression, inverted In addition, we also support *just the bracketed part* of the haskelly formats. This is to make it easier for the macro interface to destructure these forms (for sending into the ``let`` syntax transformer). So these forms are supported, too:: - ((k0, v0), ...) in body - (body, where((k0, v0), ...)) + [k0 << v0, ...] in body + (body, where[k0 << v0, ...]) - Finally, in any of these, the bindings subform can be in any of the formats: + Finally, in any of these, the bindings subform can actually be in any of + the formats: - ((k0, v0), ...) - ([k0, v0], ...) - [(k0, v0), ...] - [[k0, v0], ...] + [k0 << v0, ...] # preferred, v0.15.0+ (k0 << v0, ...) - [k0 << v0, ...] + [[k0, v0], ...] + [(k0, v0), ...] + ([k0, v0], ...) + ((k0, v0), ...) k, v - k << v + k << v # preferred for a single binding, v0.15.0+ This is a data abstraction that hides the detailed structure of the AST, since there are many alternate syntaxes that can be used for a ``let`` @@ -454,8 +450,8 @@ def _getbindings(self): # Subscript theargs = _get_subscript_slice(thetree) return canonize_bindings(_normalize_macroargs_node(theargs)) - else: # haskelly let, `let[(...) in ...]`, `let[..., where(...)]` - theexpr = self._theexpr_ref() # `(...) in ...`, `..., where(...)` + else: # haskelly let, `let[[...] in ...]`, `let[..., where[...]]` + theexpr = self._theexpr_ref() # `[...] in ...`, `..., where[...]` if t == "in_expr": return canonize_bindings(_normalize_macroargs_node(theexpr.left)) elif t == "where_expr": @@ -535,8 +531,9 @@ class UnexpandedDoView: do0[body0, ...] [...] - The list format is for convenience, for viewing an implicit ``do[]`` in the - body of a ``let`` form. + The list format is for convenience, for viewing an implicit ``do[]`` + (extra bracket syntax) in the body of a ``let`` form before the ``do`` + is actually injected. **Attributes**: diff --git a/unpythonic/syntax/tests/test_autoref.py b/unpythonic/syntax/tests/test_autoref.py index e630e0c0..da9cc031 100644 --- a/unpythonic/syntax/tests/test_autoref.py +++ b/unpythonic/syntax/tests/test_autoref.py @@ -105,9 +105,9 @@ def runtests(): with autoref[e]: x = do[local[a << 2], 2 * a] # noqa: F821 test[x == 4] - y = let[(x, 21) in 2 * x] + y = let[[x << 21] in 2 * x] test[y == 42] - z = let[(x, 21) in 2 * a] # e.a # noqa: F821 + z = let[[x << 21] in 2 * a] # e.a # noqa: F821 test[z == 2] with testset("integration with lazify"): diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index e19af79b..5dd8502a 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -109,8 +109,8 @@ def j1(a, b): with continuations: def j2(a, b): x, y = call_cc[f(a, b)] - return let[((c, a), # noqa: F821 - (d, b)) in f(c, d)] # noqa: F821 + return let[[c << a, # noqa: F821 + d << b] in f(c, d)] # noqa: F821 test[j2(3, 4) == (6, 12)] with testset("if-expression in tail position"): diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index 44632104..0154b163 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -20,13 +20,13 @@ def runtests(): echo = lambda x: [print(x), x] test[echo("hi there") == "hi there"] - count = let[(x, 0)][ # noqa: F821, the `let` macro defines `x` here. + count = let[x << 0][ # noqa: F821, the `let` macro defines `x` here. lambda: [x << x + 1, # noqa: F821 x]] # redundant, but demonstrating multi-expr body. # noqa: F821 test[count() == 1] test[count() == 2] - test1 = let[(x, 0)][ # noqa: F821 + test1 = let[x << 0][ # noqa: F821 lambda: [x << x + 1, # x belongs to the surrounding let # noqa: F821 local[y << 42], # y is local to the implicit do # noqa: F821 (x, y)]] # noqa: F821 @@ -47,14 +47,14 @@ def runtests(): with namedlambda: f1 = lambda x: x**3 # assignment: name as "f1" test[f1.__name__ == "f1"] - gn, hn = let[(x, 42), (g, None), (h, None)][[ # noqa: F821 + gn, hn = let[x << 42, g << None, h << None][[ # noqa: F821 g << (lambda x: x**2), # env-assignment: name as "g" # noqa: F821 h << f1, # still "f1" (RHS is not a literal lambda) # noqa: F821 (g.__name__, h.__name__)]] # noqa: F821 test[gn == "g"] test[hn == "f1"] - foo = let[(f7, lambda x: x) in f7] # let-binding: name as "f7" # noqa: F821 + foo = let[[f7 << (lambda x: x)] in f7] # let-binding: name as "f7" # noqa: F821 test[foo.__name__ == "f7"] warn["NamedExpr test currently disabled for syntactic compatibility with Python 3.6 and 3.7."] @@ -150,7 +150,7 @@ def decorated(*args, **kwargs): # presence of autocurry should not break the output of the outside-in pass with namedlambda: with autocurry: - foo = let[(f7, None) in f7 << (lambda x: x)] # noqa: F821 + foo = let[[f7 << None] in f7 << (lambda x: x)] # noqa: F821 test[foo.__name__ == "f7"] f6 = mypardeco(2, 3, lambda x: x**2) diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 6cfa8ead..261314f6 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -352,26 +352,26 @@ def f14(a, b): with lazify: def f(a, b): return a - test[let[((c, 42), (d, 1 / 0)) in f(c, d)] == 42] + test[let[[c << 42, d << 1 / 0] in f(c, d)] == 42] # a reference on a let binding RHS works like a reference in a function call: just pass it through e = lazy[1 / 0] - test[let[((c, 42), (d, e)) in f(c, d)] == 42] + test[let[[c << 42, d << e] in f(c, d)] == 42] # nested lets - test[letseq[((c, 42), (d, e)) in f(c, d)] == 42] - test[letseq[((a, 2), (a, 2 * a), (a, 2 * a)) in a] == 8] # name shadowing, no infinite loop # noqa: F821, `letseq` defines `a` here. + test[letseq[[c << 42, d << e] in f(c, d)] == 42] + test[letseq[[a << 2, a << 2 * a, a << 2 * a] in a] == 8] # name shadowing, no infinite loop # noqa: F821, `letseq` defines `a` here. b = 2 # let[] should already have taken care of resolving references when lazify expands - test[letseq[((b, 2 * b), (b, 2 * b)) in b] == 8] + test[letseq[[b << 2 * b, b << 2 * b] in b] == 8] test[b == 2] b = lazy[2] # should work also for lazy input - test[letseq[((b, 2 * b), (b, 2 * b)) in b] == 8] + test[letseq[[b << 2 * b, b << 2 * b] in b] == 8] test[b == 2] # letrec injects lambdas into its bindings, so test it too. - test[letrec[((c, 42), (d, e)) in f(c, d)] == 42] + test[letrec[[c << 42, d << e] in f(c, d)] == 42] # various higher-order functions, mostly from unpythonic.fun with testset("interaction with higher-order functions"): @@ -491,10 +491,10 @@ def add2first(a, b, c): def f(a, b): return a - test[let[((c, 42), (d, 1 / 0)) in f(c)(d)] == 42] - test[letrec[((c, 42), (d, 1 / 0), (e, 2 * c)) in f(e)(d)] == 84] + test[let[[c << 42, d << 1 / 0] in f(c)(d)] == 42] + test[letrec[[c << 42, d << 1 / 0, e << 2 * c] in f(e)(d)] == 84] - test[letrec[((c, 42), (d, 1 / 0), (e, 2 * c)) in [local[x << f(e)(d)], # noqa: F821, `letrec` defines `x` here. + test[letrec[[c << 42, d << 1 / 0, e << 2 * c] in [local[x << f(e)(d)], # noqa: F821, `letrec` defines `x` here. x / 2]] == 42] # noqa: F821 # works also with continuations diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index e02d6da4..08f81d6b 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -51,56 +51,56 @@ def runtests(): # Let macros. Lexical scoping supported. with testset("let, letseq, letrec basic usage"): # parallel binding, i.e. bindings don't see each other - test[let[(x, 17), - (y, 23)][ # noqa: F821, `let` defines `y` here. + test[let[x << 17, + y << 23][ # noqa: F821, `let` defines `y` here. (x, y)] == (17, 23)] # noqa: F821 # sequential binding, i.e. Scheme/Racket let* - test[letseq[(x, 1), - (y, x + 1)][ # noqa: F821 + test[letseq[x << 1, + y << x + 1][ # noqa: F821 (x, y)] == (1, 2)] # noqa: F821 - test[letseq[(x, 1), - (x, x + 1)][ # in a letseq, rebinding the same name is ok + test[letseq[x << 1, + x << x + 1][ # in a letseq, rebinding the same name is ok x] == 2] # letrec sugars unpythonic.lispylet.letrec, removing the need for quotes on LHS # and "lambda e: ..." wrappers on RHS (these are inserted by the macro): - test[letrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `letrec` defines `evenp` here. - (oddp, lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 + test[letrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `letrec` defines `evenp` here. + oddp << (lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 evenp(42)] is True] # noqa: F821 # nested letrecs work, too - each environment is internally named by a gensym # so that outer ones "show through": - test[letrec[(z, 9000)][ # noqa: F821 - letrec[(evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - (oddp, lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 + test[letrec[z << 9000][ # noqa: F821 + letrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp << (lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 (evenp(42), z)]] == (True, 9000)] # noqa: F821 with testset("error cases"): # let is parallel binding, doesn't see the X in the same let test_raises[NameError, - let[(X, 1), # noqa: F821 - (y, X + 1)][ # noqa: F821 + let[X << 1, # noqa: F821 + y << X + 1][ # noqa: F821 print(X, y)], # noqa: F821 "should not see the X in the same let"] test_raises[NameError, - letseq[(X, y + 1), # noqa: F821 - (y, 2)][ # noqa: F821 + letseq[X << y + 1, # noqa: F821 + y << 2][ # noqa: F821 (X, y)], # noqa: F821 "y should not yet be defined on the first line"] test_raises[AttributeError, - let[(x, 1), - (x, 2)][ + let[x << 1, + x << 2][ print(x)], "should not be able to rebind the same name in the same let"] # implicit do: an extra set of brackets denotes a multi-expr body with testset("implicit do (extra bracket syntax for multi-expr let body)"): - a = let[(x, 1), - (y, 2)][[ # noqa: F821 + a = let[x << 1, + y << 2][[ # noqa: F821 y << 1337, # noqa: F821 (x, y)]] # noqa: F821 test[a == (1, 1337)] @@ -112,14 +112,14 @@ def runtests(): test[a == [1, 2]] # implicit do works also in letseq, letrec - a = letseq[(x, 1), - (y, x + 1)][[ # noqa: F821 + a = letseq[x << 1, + y << x + 1][[ # noqa: F821 x << 1337, (x, y)]] # noqa: F821 test[a == (1337, 2)] - a = letrec[(x, 1), - (y, x + 1)][[ # noqa: F821 + a = letrec[x << 1, + y << x + 1][[ # noqa: F821 x << 1337, (x, y)]] # noqa: F821 test[a == (1337, 2)] @@ -129,36 +129,36 @@ def runtests(): # (so the z in the inner scope expands to the inner environment's z, # which makes the outer expansion leave it alone): out = [] - letrec[(z, 1)][ # noqa: F821 + letrec[z << 1][ # noqa: F821 begin(out.append(z), # noqa: F821 - letrec[(z, 2)][ # noqa: F821 + letrec[z << 2][ # noqa: F821 out.append(z)])] # (be careful with the parentheses!) # noqa: F821 test[out == [1, 2]] # same using implicit do (extra brackets) out = [] - letrec[(z, 1)][[ # noqa: F821 + letrec[z << 1][[ # noqa: F821 out.append(z), # noqa: F821 - letrec[(z, 2)][ # noqa: F821 + letrec[z << 2][ # noqa: F821 out.append(z)]]] # noqa: F821 test[out == [1, 2]] # lexical scoping: assignment updates the innermost value by that name: out = [] - letrec[(z, 1)][ # noqa: F821 + letrec[z << 1][ # noqa: F821 begin(out.append(z), # outer z # noqa: F821 # assignment to env is an expression, returns the new value out.append(z << 5), # noqa: F821 - letrec[(z, 2)][ # noqa: F821 + letrec[z << 2][ # noqa: F821 begin(out.append(z), # inner z # noqa: F821 out.append(z << 7))], # update inner z # noqa: F821 out.append(z))] # outer z # noqa: F821 test[out == [1, 5, 2, 7, 5]] out = [] - letrec[(x, 1)][ + letrec[x << 1][ begin(out.append(x), - letrec[(z, 2)][ # noqa: F821 + letrec[z << 2][ # noqa: F821 begin(out.append(z), # noqa: F821 out.append(x << 7))], # x only defined in outer letrec, updates that out.append(x))] @@ -166,23 +166,23 @@ def runtests(): # same using implicit do out = [] - letrec[(x, 1)][[ + letrec[x << 1][[ out.append(x), - letrec[(z, 2)][[ # noqa: F821 + letrec[z << 2][[ # noqa: F821 out.append(z), # noqa: F821 out.append(x << 7)]], out.append(x)]] test[out == [1, 2, 7, 7]] # letrec bindings are evaluated sequentially - test[letrec[(x, 1), - (y, x + 2)][ # noqa: F821 + test[letrec[x << 1, + y << x + 2][ # noqa: F821 (x, y)] == (1, 3)] # noqa: F821 # so this is an error (just like in Racket): test_raises[AttributeError, - letrec[(x, y + 1), # noqa: F821, `y` being undefined here is the point of this test. - (y, 2)][ # noqa: F821 + letrec[x << y + 1, # noqa: F821, `y` being undefined here is the point of this test. + y << 2][ # noqa: F821 print(x)], "y should not be yet defined on the first line"] @@ -191,33 +191,33 @@ def runtests(): # # This is the whole point of having a letrec construct, # instead of just let, letseq. - test[letrec[(f, lambda t: t + y + 1), # noqa: F821 - (y, 2)][ # noqa: F821 + test[letrec[f << (lambda t: t + y + 1), # noqa: F821 + y << 2][ # noqa: F821 f(3)] == 6] # noqa: F821 # bindings are evaluated only once - a = letrec[(x, 1), - (y, x + 2)][[ # y computed now, using the current value of x # noqa: F821 + a = letrec[x << 1, + y << x + 2][[ # y computed now, using the current value of x # noqa: F821 x << 1337, # x updated now, no effect on y (x, y)]] # noqa: F821 test[a == (1337, 3)] # lexical scoping: a comprehension or lambda in a let body # shadows names from the surrounding let, but only in that subexpr - test[let[(x, 42)][[ + test[let[x << 42][[ [x for x in range(10)]]] == list(range(10))] - test[let[(x, 42)][[ + test[let[x << 42][[ [x for x in range(10)], x]] == 42] - test[let[(x, 42)][ + test[let[x << 42][ (lambda x: x**2)(10)] == 100] - test[let[(x, 42)][[ + test[let[x << 42][[ (lambda x: x**2)(10), x]] == 42] # let over lambda - in Python! with testset("let over lambda"): - count = let[(x, 0)][ + count = let[x << 0][ lambda: x << x + 1] test[count() == 1] test[count() == 2] @@ -226,7 +226,7 @@ def runtests(): # - sugar around unpythonic.lispylet.dlet et al. # - env is passed implicitly, and named with a gensym (so lexical scoping works) with testset("let over def"): - @dlet((x, 0)) + @dlet(x << 0) def count(): x << x + 1 # assigment to let environment uses the "assignment expr" syntax return x @@ -234,26 +234,26 @@ def count(): test[count() == 2] # nested dlets respect lexical scoping - @dlet((x, 22)) + @dlet(x << 22) def outer(): x << x + 1 - @dlet((x, 41)) + @dlet(x << 41) def inner(): return x << x + 1 return (x, inner()) test[outer() == (23, 42)] # letseq over def - @dletseq((x, 1), - (x, x + 1), - (x, x + 2)) + @dletseq(x << 1, + x << x + 1, + x << x + 2) def g(a): return a + x test[g(10) == 14] # letrec over def - @dletrec((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `dletrec` defines `evenp` here. - (oddp, lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 + @dletrec(evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `dletrec` defines `evenp` here. + oddp << (lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 def f(x): return evenp(x) # noqa: F821 test[f(42) is True] @@ -262,20 +262,20 @@ def f(x): with testset("let block"): # block version # - the def takes no args, runs immediately, replaced with return value - @blet((x, 21)) + @blet(x << 21) def result(): return 2 * x test[result == 42] - @bletseq((x, 1), - (x, x + 1), - (x, x + 2)) # noqa: F823, `bletseq` defines and assigns to `x`. + @bletseq(x << 1, + x << x + 1, + x << x + 2) # noqa: F823, `bletseq` defines and assigns to `x`. def result(): return x test[result == 4] - @bletrec((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `bletrec` defines `evenp` here. - (oddp, lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 + @bletrec(evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `bletrec` defines `evenp` here. + oddp << (lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 def result(): return evenp(42) # noqa: F821 test[result is True] @@ -283,43 +283,43 @@ def result(): # interaction of unpythonic's scoping system with Python's own lexical scoping with testset("integration of let scoping with Python's scoping"): x = "the nonlocal x" - @dlet((x, "the env x")) + @dlet(x << "the env x") def test1(): return x test[test1() == "the env x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test2(): return x # local var assignment not in effect yet # noqa: F823, `dlet` defines `x` here. x = "the unused local x" # noqa: F841, this `x` being unused is the point of this test. # pragma: no cover test[test2() == "the env x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test3(): x = "the local x" return x test[test3() == "the local x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test4(): nonlocal x return x test[test4() == "the nonlocal x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test5(): global x return x test[test5() == "the global x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test6(): class Foo: x = "the classattr x" # name in store context, not the env x return x test[test6() == "the env x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test7(): class Foo: x = "the classattr x" @@ -328,7 +328,7 @@ def doit(self): return (Foo().doit(), x) test[test7() == ("the classattr x", "the env x")] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test8(): class Foo: x = "the classattr x" @@ -361,27 +361,27 @@ def doit(self): # in the standard library, but we also perform some expression-by-expression # analysis to make it possible to refer to the old bindings on the RHS of # "name << value", as well as to support local deletion.) - @dlet((x, "the env x")) + @dlet(x << "the env x") def test9(): x = x + " (copied to local)" # the local x = the env x # noqa: F823 return x # the local x test[test9() == "the env x (copied to local)"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test10(): x = x + " (copied to local)" # noqa: F823 del x # comes into effect for the next statement return x # so this is env's original x test[test10() == "the env x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test11(): x = "the local x" return x # not deleted yet del x # this seems to be optimized out by Python. # pragma: no cover test[test11() == "the local x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test12(): x = "the local x" del x @@ -389,7 +389,7 @@ def test12(): return x test[test12() == "the other local x"] - @dlet((x, "the env x")) + @dlet(x << "the env x") def test13(): x = "the local x" del x @@ -399,7 +399,7 @@ def test13(): with test_raises[NameError, "should have tried to access the deleted nonlocal x"]: x = "the nonlocal x" - @dlet((x, "the env x")) + @dlet(x << "the env x") def test14(): nonlocal x del x # ignored by unpythonic's scope analysis, too dynamic @@ -408,71 +408,71 @@ def test14(): x = "the nonlocal x" # restore the test environment # in do[] (also the implicit do), local[] takes effect from the next item - test[let[(x, "the let x"), - (y, None)][ # noqa: F821 + test[let[x << "the let x", + y << None][ # noqa: F821 do[y << x, # still the "x" of the let # noqa: F821 local[x << "the do x"], # from here on, "x" refers to the "x" of the do (x, y)]] == ("the do x", "the let x")] # noqa: F821 # don't code like this! ...but the scoping mechanism should understand it result = [] - let[(lst, [])][do[result.append(lst), # the let "lst" # noqa: F821 + let[lst << []][do[result.append(lst), # the let "lst" # noqa: F821 local[lst << lst + [1]], # LHS: do "lst", RHS: let "lst" # noqa: F821 result.append(lst)]] # the do "lst" # noqa: F821 test[result == [[], [1]]] # same using implicit do result = [] - let[(lst, [])][[result.append(lst), # noqa: F821 + let[lst << []][[result.append(lst), # noqa: F821 local[lst << lst + [1]], # noqa: F821 result.append(lst)]] # noqa: F821 test[result == [[], [1]]] with testset("haskelly syntax"): - result = let[((foo, 5), # noqa: F821, `let` defines `foo` here. - (bar, 2)) # noqa: F821 + result = let[[foo << 5, # noqa: F821, `let` defines `foo` here. + bar << 2] # noqa: F821 in foo + bar] # noqa: F821 test[result == 7] - result = letseq[((foo, 100), # noqa: F821, `letseq` defines `foo` here. - (foo, 2 * foo), # noqa: F821 - (foo, 4 * foo)) # noqa: F821 + result = letseq[[foo << 100, # noqa: F821, `letseq` defines `foo` here. + foo << 2 * foo, # noqa: F821 + foo << 4 * foo] # noqa: F821 in foo] # noqa: F821 test[result == 800] - result = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `letrec` defines `evenp` here. - (oddp, lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 + result = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `letrec` defines `evenp` here. + oddp << (lambda x: (x != 0) and evenp(x - 1))] # noqa: F821 in [print("hi from letrec-in"), evenp(42)]] # noqa: F821 test[result is True] # inverted let, for situations where a body-first style improves readability: result = let[foo + bar, # noqa: F821, the names in this expression are defined in the `where` clause of the `let`. - where((foo, 5), # noqa: F821, this defines `foo`. - (bar, 2))] # noqa: F821 + where[foo << 5, # noqa: F821, this defines `foo`. + bar << 2]] # noqa: F821 test[result == 7] result = letseq[foo, # noqa: F821 - where((foo, 100), # noqa: F821 - (foo, 2 * foo), # noqa: F821 - (foo, 4 * foo))] # noqa: F821 + where[foo << 100, # noqa: F821 + foo << 2 * foo, # noqa: F821 + foo << 4 * foo]] # noqa: F821 test[result == 800] # can also use the extra bracket syntax to get an implicit do # (note the [] should then enclose the body only). result = letrec[[print("hi from letrec-where"), evenp(42)], # noqa: F821 - where((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - (oddp, lambda x: (x != 0) and evenp(x - 1)))] # noqa: F821 + where[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp << (lambda x: (x != 0) and evenp(x - 1))]] # noqa: F821 test[result is True] - # TODO: for now, with more than one binding the outer parentheses + # With more than one binding the delimiters surrounding the bindings subform # are required, even in this format where they are somewhat redundant. - result = let[((x, 1), (y, 2)) in x + y] # noqa: F821 + result = let[[x << 1, y << 2] in x + y] # noqa: F821 test[result == 3] - # single binding special syntax, no need for outer parentheses - with testset("special syntax for single binding case"): + # single binding special syntax, no need for outer delimiters + with testset("special syntax for single binding case (classic format)"): result = let[x, 1][2 * x] test[result == 2] result = let[(x, 1) in 2 * x] @@ -510,12 +510,50 @@ def quux(): return x test[quux == 1] + with testset("special syntax for single binding case (modern format)"): + result = let[x << 1][2 * x] + test[result == 2] + result = let[[x << 1] in 2 * x] + test[result == 2] + result = let[2 * x, where[x << 1]] + test[result == 2] + + @dlet(x << 1) + def qux(): + return x + test[qux() == 1] + + @dletseq(x << 1) + def qux(): + return x + test[qux() == 1] + + @dletrec(x << 1) + def qux(): + return x + test[qux() == 1] + + @blet(x << 1) + def quux(): + return x + test[quux == 1] + + @bletseq(x << 1) + def quux(): + return x + test[quux == 1] + + @bletrec(x << 1) + def quux(): + return x + test[quux == 1] + with testset("object instance bound to let variable"): # The point is to test whether `s.a` below transforms # correctly to `e.s.a`. class Silly: a = "Ariane 5" - test[let[(s, Silly()) in s.a] == "Ariane 5"] # noqa: F821 + test[let[[s << Silly()] in s.a] == "Ariane 5"] # noqa: F821 if __name__ == '__main__': # pragma: no cover with session(__file__): diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index dffe2f29..871530e4 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -57,40 +57,64 @@ def validate(lst): test[not islet(q[x])] # noqa: F821 test[not islet(q[f()])] # noqa: F821 + test[islet(the[expandrq[let[x << 21][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` + test[islet(the[expandrq[let[[x << 21] in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 + test[islet(the[expandrq[let[2 * x, where[x << 21]]]]) == ("expanded_expr", "let")] # noqa: F821 + test[islet(the[expandrq[let[(x, 21)][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` test[islet(the[expandrq[let[(x, 21) in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 test[islet(the[expandrq[let[2 * x, where(x, 21)]]]) == ("expanded_expr", "let")] # noqa: F821 + with expandrq as testdata: + @dlet(x << 21) # noqa: F821 + def f2(): + return 2 * x # noqa: F821 + test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] + with expandrq as testdata: @dlet((x, 21)) # noqa: F821 def f1(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] + testdata = q[let[x << 21][2 * x]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("lispy_expr", "let")] + testdata = q[let[(x, 21)][2 * x]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("lispy_expr", "let")] # one binding special case for haskelly let-in + testdata = q[let[[x, 21] in 2 * x]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("in_expr", "let")] testdata = q[let[(x, 21) in 2 * x]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("in_expr", "let")] + testdata = q[let[2 * x, where[x, 21]]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("where_expr", "let")] + testdata = q[let[2 * x, where(x, 21)]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("where_expr", "let")] + testdata = q[let[[x << 21, y << 2] in y * x]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("in_expr", "let")] testdata = q[let[((x, 21), (y, 2)) in y * x]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("in_expr", "let")] - testdata = q[let[2 * x, where(x, 21)]] # noqa: F821 - test[islet(the[testdata], expanded=False) == ("where_expr", "let")] - # some other macro invocation test[not islet(the[q[someothermacro((x, 21))[2 * x]]], expanded=False)] # noqa: F821 test[not islet(the[q[someothermacro[(x, 21) in 2 * x]]], expanded=False)] # noqa: F821 - # invalid syntax for haskelly let-in + # invalid syntax for haskelly let-in (no delimiters around bindings subform) testdata = q[let[a in b]] # noqa: F821 test[not islet(the[testdata], expanded=False)] with q as testdata: @dlet((x, 21)) # noqa: F821 - def f2(): + def f3(): + return 2 * x # noqa: F821 + test[islet(the[testdata[0].decorator_list[0]], expanded=False) == ("decorator", "dlet")] + + with q as testdata: + @dlet(x << 21) # noqa: F821 + def f4(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]], expanded=False) == ("decorator", "dlet")] @@ -101,6 +125,24 @@ def f2(): # because otherwise `autocurry` will attempt to curry the AST-lifted # representation, leading to arguably funny but nonsensical things like # `ctx=currycall(ast.Load)`. + with expandrq as testdata: + with autocurry: + let[x << 21][2 * x] # noqa: F821 # note this goes into an ast.Expr + thelet = testdata[0].value + test[islet(the[thelet]) == ("curried_expr", "let")] + + with expandrq as testdata: + with autocurry: + let[[x << 21] in 2 * x] # noqa: F821 + thelet = testdata[0].value + test[islet(the[thelet]) == ("curried_expr", "let")] + + with expandrq as testdata: + with autocurry: + let[2 * x, where[x << 21]] # noqa: F821 + thelet = testdata[0].value + test[islet(the[thelet]) == ("curried_expr", "let")] + with expandrq as testdata: with autocurry: let((x, 21))[2 * x] # noqa: F821 # note this goes into an ast.Expr @@ -267,7 +309,7 @@ def testletdestructuring(testdata): # decorator with q as testdata: @dlet((x, 21), (y, 2)) # noqa: F821 - def f3(): + def f5(): return 2 * x # noqa: F821 # read @@ -348,7 +390,7 @@ def testexpandedletdestructuring(testdata): # decorator with expandrq as testdata: @dlet((x, 21), (y, 2)) # noqa: F821 - def f4(): + def f6(): return 2 * x # noqa: F821 view = ExpandedLetView(testdata[0].decorator_list[0]) test_raises[TypeError, @@ -444,7 +486,7 @@ def testbindings(*expected): # decorator, letrec with expandrq as testdata: @dletrec((x, 21), (y, 2)) # noqa: F821 - def f5(): + def f7(): return 2 * x # noqa: F821 view = ExpandedLetView(testdata[0].decorator_list[0]) test_raises[TypeError, diff --git a/unpythonic/syntax/tests/test_tco.py b/unpythonic/syntax/tests/test_tco.py index 344d2fb5..fd49f3e1 100644 --- a/unpythonic/syntax/tests/test_tco.py +++ b/unpythonic/syntax/tests/test_tco.py @@ -57,24 +57,24 @@ def lamtest(): # works with let constructs with testset("basic usage in let constructs"): - @dletrec((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `dletrec` defines `evenp` here. - (oddp, lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 + @dletrec(evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `dletrec` defines `evenp` here. + oddp << (lambda x: (x != 0) and evenp(x - 1))) # noqa: F821 def g(x): return evenp(x) test[g(9001) is False] def g(x): - return let[(y, 3 * x)][y] # noqa: F821, `let` defines `y` here. + return let[y << 3 * x][y] # noqa: F821, `let` defines `y` here. test[g(10) == 30] def h(x): - return let[(y, 2 * x)][g(y)] # noqa: F821 + return let[y << 2 * x][g(y)] # noqa: F821 test[h(10) == 60] def h(x): - return letseq[(y, x), # noqa: F821, `letseq` defines `y` here. - (y, y + 1), # noqa: F821 - (y, y + 1)][g(y)] # noqa: F821 + return letseq[y << x, # noqa: F821, `letseq` defines `y` here. + y << y + 1, # noqa: F821 + y << y + 1][g(y)] # noqa: F821 test[h(10) == 36] with testset("integration with autoreturn"): From 5487efc15cd5397e06997ae3fedc774c6a8c9c57 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 15:25:08 +0300 Subject: [PATCH 219/832] update examples for let_syntax, abbrev, dlet, blet et al. --- unpythonic/syntax/letsyntax.py | 26 +++++++++++------------ unpythonic/syntax/tests/test_conts_gen.py | 8 +++---- unpythonic/syntax/tests/test_letdoutil.py | 6 ++++-- unpythonic/syntax/tests/test_letsyntax.py | 14 ++++++------ 4 files changed, 28 insertions(+), 26 deletions(-) diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index e4151835..b0acc118 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -33,16 +33,16 @@ def let_syntax(tree, *, args, syntax, expander, **kw): **Expression variant**:: - let_syntax[(lhs, rhs), ...][body] - let_syntax[(lhs, rhs), ...][[body0, ...]] + let_syntax[lhs << rhs, ...][body] + let_syntax[lhs << rhs, ...][[body0, ...]] Alternative haskelly syntax:: - let_syntax[((lhs, rhs), ...) in body] - let_syntax[((lhs, rhs), ...) in [body0, ...]] + let_syntax[[lhs << rhs, ...] in body] + let_syntax[[lhs << rhs, ...] in [body0, ...]] - let_syntax[body, where((lhs, rhs), ...)] - let_syntax[[body0, ...], where((lhs, rhs), ...)] + let_syntax[body, where[lhs << rhs, ...]] + let_syntax[[body0, ...], where[lhs << rhs, ...]] **Block variant**:: @@ -148,7 +148,7 @@ def abbrev(tree, *, args, syntax, expander, **kw): Because this variant expands before any macros in the body, it can locally rename other macros, e.g.:: - abbrev[(m, macrowithverylongname)][ + abbrev[m << macrowithverylongname][ m[tree1] if m[tree2] else m[tree3]] **CAUTION**: Because ``abbrev`` expands outside-in, and does not respect @@ -194,12 +194,12 @@ def block(tree, *, syntax, **kw): # -------------------------------------------------------------------------------- # Syntax transformers -# let_syntax[(lhs, rhs), ...][body] -# let_syntax[(lhs, rhs), ...][[body0, ...]] -# let_syntax[((lhs, rhs), ...) in body] -# let_syntax[((lhs, rhs), ...) in [body0, ...]] -# let_syntax[body, where((lhs, rhs), ...)] -# let_syntax[[body0, ...], where((lhs, rhs), ...)] +# let_syntax[lhs << rhs, ...][body] +# let_syntax[lhs << rhs, ...][[body0, ...]] +# let_syntax[[lhs << rhs, ...] in body] +# let_syntax[[lhs << rhs, ...] in [body0, ...]] +# let_syntax[body, where[lhs << rhs, ...]] +# let_syntax[[body0, ...], where[lhs << rhs, ...]] # # This transformer takes destructured input, with the bindings subform # and the body already extracted, and supplied separately. diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index ef7571b2..48a804f8 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -32,7 +32,7 @@ def runtests(): with testset("a basic generator"): with continuations: # logic to resume after the last executed my_yield, if any - @dlet((k, None)) # noqa: F821, dlet defines the name. + @dlet(k << None) # noqa: F821, dlet defines the name. def g(): if k: # noqa: F821 return k() # noqa: F821 @@ -57,7 +57,7 @@ def my_yield(value, cc): with testset("FP loop based generator"): with continuations: # logic to resume after the last executed my_yield, if any - @dlet((k, None)) # noqa: F821 + @dlet(k << None) # noqa: F821 def g(): if k: # noqa: F821 return k() # noqa: F821 @@ -108,7 +108,7 @@ def my_yieldf(value, cc): cc = identity return value - @dlet((k, None)) # <-- we must still remember this line # noqa: F821 + @dlet(k << None) # <-- we must still remember this line # noqa: F821 def g(): begin_generator_body my_yield(1) @@ -132,7 +132,7 @@ def g(): with block[value] as my_yield: # noqa: F821 call_cc[my_yieldf(value)] # for this to work, let_syntax[] must eliminate its "if 1" blocks. # noqa: F821 with block[myname, body] as make_generator: # noqa: F821, `let_syntax` defines `myname` and `body` when we call `make_generator`. - @dlet((k, None)) # noqa: F821 + @dlet(k << None) # noqa: F821 def myname(): # replaced by the user-supplied name, since "myname" is a template parameter. # logic to resume after the last executed my_yield, if any if k: # noqa: F821 diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index 871530e4..45725233 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -57,23 +57,25 @@ def validate(lst): test[not islet(q[x])] # noqa: F821 test[not islet(q[f()])] # noqa: F821 + # modern notation for bindings test[islet(the[expandrq[let[x << 21][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` test[islet(the[expandrq[let[[x << 21] in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 test[islet(the[expandrq[let[2 * x, where[x << 21]]]]) == ("expanded_expr", "let")] # noqa: F821 + # classic notation for bindings test[islet(the[expandrq[let[(x, 21)][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` test[islet(the[expandrq[let[(x, 21) in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 test[islet(the[expandrq[let[2 * x, where(x, 21)]]]) == ("expanded_expr", "let")] # noqa: F821 with expandrq as testdata: @dlet(x << 21) # noqa: F821 - def f2(): + def f1(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] with expandrq as testdata: @dlet((x, 21)) # noqa: F821 - def f1(): + def f2(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] diff --git a/unpythonic/syntax/tests/test_letsyntax.py b/unpythonic/syntax/tests/test_letsyntax.py index 9b112d0b..13940cdd 100644 --- a/unpythonic/syntax/tests/test_letsyntax.py +++ b/unpythonic/syntax/tests/test_letsyntax.py @@ -25,14 +25,14 @@ def verylongfunctionname(x=1): nonlocal evaluations evaluations += 1 return x - y = let_syntax((f, verylongfunctionname))[[ # extra brackets: implicit do # noqa: F821, `let_syntax` defines `f` here. + y = let_syntax(f << verylongfunctionname)[[ # extra brackets: implicit do # noqa: F821, `let_syntax` defines `f` here. f(), # noqa: F821 f(5)]] # noqa: F821 test[evaluations == 2] test[y == 5] # haskelly syntax - y = let_syntax[((f, verylongfunctionname)) # noqa: F821 + y = let_syntax[[f << verylongfunctionname] # noqa: F821 in [f(), # noqa: F821 f(17)]] # noqa: F821 test[evaluations == 4] @@ -40,14 +40,14 @@ def verylongfunctionname(x=1): y = let_syntax[[f(), # noqa: F821 f(23)], # noqa: F821 - where((f, verylongfunctionname))] # noqa: F821 + where[f << verylongfunctionname]] # noqa: F821 test[evaluations == 6] test[y == 23] # templates # - positional parameters only, no default values # TODO: updating this to use bracket syntax requires changes to `_destructure_and_apply_let`. - y = let_syntax((f[a], verylongfunctionname(2 * a)))[[ # noqa: F821 + y = let_syntax(f[a] << verylongfunctionname(2 * a))[[ # noqa: F821 f[2], # noqa: F821 f[3]]] # noqa: F821 test[evaluations == 8] @@ -57,7 +57,7 @@ def verylongfunctionname(x=1): class Silly: realthing = 42 # This test will either pass, or error out with an AttributeError. - test[let_syntax[((alias, realthing)) in Silly.alias] == 42] # noqa: F821 + test[let_syntax[[alias << realthing] in Silly.alias] == 42] # noqa: F821 with testset("block variant"): with let_syntax: @@ -181,14 +181,14 @@ def alias(): test[y == 5] # haskelly syntax - y = abbrev[((f, verylongfunctionname)) # noqa: F821 + y = abbrev[[f << verylongfunctionname] # noqa: F821 in [f(), # noqa: F821 f(17)]] # noqa: F821 test[y == 17] y = abbrev[[f(), # noqa: F821 f(23)], # noqa: F821 - where((f, verylongfunctionname))] # noqa: F821 + where[f << verylongfunctionname]] # noqa: F821 test[y == 23] # in abbrev, outer expands first, so in the test, From 61896ca81f6d5153b5f3b21ce1f3b5493e0aebd1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 15:36:35 +0300 Subject: [PATCH 220/832] update a few stray letrec examples --- doc/dialects/lispython.md | 4 ++-- unpythonic/dialects/tests/test_lispython.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 60bd370d..b462c343 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -46,8 +46,8 @@ def factorial(n): assert factorial(4) == 24 factorial(5000) # no crash -t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), - (oddp, lambda x: (x != 0) and evenp(x - 1))) in +t = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), + oddp << (lambda x: (x != 0) and evenp(x - 1))] in evenp(10000)] assert t is True diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index dd7cbf9b..35988992 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -101,8 +101,8 @@ def f(k, acc): # # With `mcpyrate` this shouldn't matter, but we're keeping the example. with testset("autonamed letrec lambdas, multiple-expression let body"): - t = letrec[((evenp, lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - (oddp, lambda x:(x != 0) and evenp(x - 1))) in # noqa: F821 + t = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp << (lambda x:(x != 0) and evenp(x - 1))] in # noqa: F821 [local[x << evenp(100)], # noqa: F821, multi-expression let body is a do[] environment (x, evenp.__name__, oddp.__name__)]] # noqa: F821 test[t == (True, "evenp", "oddp")] From bdb0dbdb0986fc56035a2039f16504f4262e8ecf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 15:36:48 +0300 Subject: [PATCH 221/832] fix macro ordering in docstring examples --- unpythonic/syntax/lazify.py | 4 ++-- unpythonic/syntax/tests/test_lazify.py | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index fadcac2d..01d9dcec 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -352,7 +352,7 @@ def g(x): Introducing the *HasThon* programming language (it has 100% more Thon than popular brands):: - with autocurry, lazify: # or continuations, autocurry, lazify if you want those + with lazify, autocurry: def add2first(a, b, c): return a + b assert add2first(2)(3)(1/0) == 5 @@ -383,7 +383,7 @@ def f(a, b): Example:: - with continuations, lazify: + with lazify, continuations: k = None def setk(*args, cc): nonlocal k diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 261314f6..6ab71ff9 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -477,7 +477,6 @@ def withec2(ec): test[withec2 == 42] # Introducing the HasThon programming language. - # For a continuation-enabled HasThon, use "with lazify, autocurry, continuations". # If you want to play around with this idea, see `unpythonic.dialects.pytkell`. with testset("HasThon, with 100% more Thon than popular brands"): with lazify, autocurry: From d98d8cea64651d8182240d07b89b08a744abeb45 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 15:37:08 +0300 Subject: [PATCH 222/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 4d88b591..a63e5e73 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,8 +69,6 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: Check expansion order of several macros in the same `with` statement - # TODO: Have a common base class for all `unpythonic` `ASTMarker`s? # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. From 8cc4bf55a46afdbc335bd663f982643d9ecac671 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 15:59:45 +0300 Subject: [PATCH 223/832] improve let_syntax docs --- doc/macros.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 62d84d2f..d2b1dc28 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -399,7 +399,13 @@ else: ### ``let_syntax``, ``abbrev``: syntactic local bindings -Locally splice code at macro expansion time (it's almost like inlining functions): +**Note v0.15.0.** *Now that we use `mcpyrate` as the macro expander, `let_syntax` and `abbrev` are not really needed. We are keeping them mostly for backwards compatibility, and because they exercise a different feature set in the macro expander, making the existence of these constructs particularly useful for system testing.* + +*To define macros in the same module that uses them, see [multi-phase compilation](https://github.com/Technologicat/mcpyrate/blob/master/doc/compiler.md#multi-phase-compilation) in the [compiler documentation](https://github.com/Technologicat/mcpyrate/blob/master/doc/compiler.md). Using [run-time compiler access](https://github.com/Technologicat/mcpyrate/blob/master/doc/compiler.md#invoking-the-compiler-at-run-time), you can even create a macro definition module at run time (e.g. from a [quasiquoted](https://github.com/Technologicat/mcpyrate/blob/master/doc/quasiquotes.md) block) and inject it to `sys.modules`, allowing other code to import and use those macros. See the [compiler tests](https://github.com/Technologicat/mcpyrate/blob/master/mcpyrate/test/test_compiler.py) for examples.* + +*To rename existing macros, you can as-import them. As of `unpythonic` v0.15.0, doing so for `unpythonic.syntax` constructs is not recommended, though, because there is still a lot of old analysis code in the macro implementations that may scan for the original name. This may or may not be fixed in a future release.* + +These constructs allow to locally splice code at macro expansion time (it's almost like inlining functions): #### ``let_syntax`` @@ -435,10 +441,14 @@ y = let_syntax[[print(f[2]), # works as a block macro with let_syntax: + # with block as name: + # with block[a0, ...] as name: with block[a, b, c] as makeabc: # capture a block of statements lst = [a, b, c] makeabc(3 + 4, 2**3, 3 * 3) assert lst == [7, 8, 9] + # with expr as name: + # with expr[a0, ...] as name: with expr[n] as nth: # capture a single expression lst[n] assert nth(2) == 9 @@ -511,9 +521,9 @@ abbrev[m[tree1] if m[tree2] else m[tree3], which can be useful when writing macros. -**CAUTION**: ``let_syntax`` is essentially a toy macro system within the real macro system. The usual caveats of macro systems apply. Especially, we support absolutely no form of hygiene. Be very, very careful to avoid name conflicts. +**CAUTION**: ``let_syntax`` is essentially a toy macro system within the real macro system. The usual caveats of macro systems apply. Especially, ``let_syntax`` and ``abbrev`` support absolutely no form of hygiene. Be very, very careful to avoid name conflicts. -The ``let_syntax`` macro is meant for simple local substitutions where the elimination of repetition can shorten the code and improve its readability. If you need to do something complex (or indeed save a definition and reuse it somewhere else, non-locally), write a real macro directly in `mcpyrate`. +The ``let_syntax`` macro is meant for simple local substitutions where the elimination of repetition can shorten the code and improve its readability, in cases where the final "unrolled" code should be written out at compile time. If you need to do something complex (or indeed save a definition and reuse it somewhere else, non-locally), write a real macro directly in `mcpyrate`. This was inspired by Racket's [``let-syntax``](https://docs.racket-lang.org/reference/let.html) and [``with-syntax``](https://docs.racket-lang.org/reference/stx-patterns.html). From 399fefe9c7d06c613e79a6fd5494b1e92a96a219 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 8 May 2021 15:59:54 +0300 Subject: [PATCH 224/832] improve simplelet docs --- doc/macros.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index d2b1dc28..b2bdbd1f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -532,9 +532,16 @@ This was inspired by Racket's [``let-syntax``](https://docs.racket-lang.org/refe As a bonus, we provide classical simple ``let`` and ``letseq``, wholly implemented as AST transformations, providing true lexical variables but no assignment support (because in Python, assignment is a statement) or multi-expression body support. Just like in Lisps, this version of ``letseq`` (Scheme/Racket ``let*``) expands into a chain of nested ``let`` expressions, which expand to lambdas. -These are provided in the separate module ``unpythonic.syntax.simplelet``, import them with the line: +These are provided in the separate module ``unpythonic.syntax.simplelet``, and are not part of the `unpythonic.syntax` macro API. For simplicity, they support only the lispy list syntax in the bindings subform (using brackets, specifically!), and no haskelly syntax at all: -``from unpythonic.syntax.simplelet import macros, let, letseq``. +```python +from unpythonic.syntax.simplelet import macros, let, letseq + +let[[x, 42], [y, 23]][...] +let[[x, 42]][...] +letseq[[x, 1], [x, x + 1]][...] +letseq[[x, 1]][...] +``` ## Sequencing From b5cfc6382f16b29b55e88fb40234b9ee232f1beb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 9 May 2021 01:28:02 +0300 Subject: [PATCH 225/832] update porting TODO comments --- unpythonic/syntax/__init__.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index a63e5e73..034dca54 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -69,16 +69,12 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: Have a common base class for all `unpythonic` `ASTMarker`s? - # TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. -# TODO: Get rid of `UnpythonicExpandedMacroMarker`, use `ASTMarker`. We don't really need to keep markers -# TODO: until run time. This requires a hook in the expander for custom postprocessors, so we can auto-remove -# TODO: our AST markers when macro expansion of a module is complete. - # TODO: Consider using run-time compiler access in macro tests, like `mcpyrate` itself does. This compartmentalizes testing so that the whole test module won't crash on a macro-expansion error. +# TODO: Return a compile-time marker from all block macros? + # TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. # TODO: 0.16: Add call-macros to `mcpyrate`. This allows the whole expression of `kw()`/`where()` to be detected as a macro invocation. (First, think whether this is a good idea.) From 33a2db921dabbff85c2bff79e1beec2b942871b9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 9 May 2021 01:30:02 +0300 Subject: [PATCH 226/832] use mcpyrate's ASTMarkers for data-driven communication between macros --- CHANGELOG.md | 4 +- unpythonic/syntax/__init__.py | 14 +++- unpythonic/syntax/autocurry.py | 2 +- unpythonic/syntax/autoref.py | 97 ++++++++++------------ unpythonic/syntax/lazify.py | 11 ++- unpythonic/syntax/tailtools.py | 22 ++--- unpythonic/syntax/tests/test_util.py | 33 +------- unpythonic/syntax/util.py | 120 +++++++-------------------- 8 files changed, 109 insertions(+), 194 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 203fce74..f121665d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -131,9 +131,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. -- Rename the internal utility class `unpythonic.syntax.util.ASTMarker` to `UnpythonicExpandedMacroMarker`, to explicitly have a class name different from `mcpyrate.markers.ASTMarker`, because these represent semantically different things. - - `mcpyrate`'s `ASTMarker`s are a macro-expansion-time data-driven communication feature to allow macros to easily work together, and are deleted from the AST before handing the AST to Python's `compile` function. (If you're curious, `unpythonic` uses some of those markers itself; grep the codebase for `ASTMarker`.) - - `unpythonic`'s `UnpythonicExpandedMacroMarker`s remain in the AST at run time. +- Remove the internal utility class `unpythonic.syntax.util.ASTMarker`. We now have `mcpyrate.markers.ASTMarker`, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time. - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 034dca54..0bf37070 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -4,8 +4,6 @@ Requires `mcpyrate`. """ -from ..dynassign import make_dynvar - # -------------------------------------------------------------------------------- # This module only re-exports the macro interfaces so the macros can be imported # by `from unpythonic.syntax import macros, ...`. The submodules contain the actual @@ -99,8 +97,18 @@ from .tailtools import * # noqa: F401, F403 from .testingtools import * # noqa: F401, F403 +# -------------------------------------------------------------------------------- +# Initialization code, not really meant for export. + +from ..dynassign import make_dynvar as _make_dynvar + # We use `dyn` to pass the `expander` parameter to the macro implementations. class _NoExpander: def __getattr__(self, k): # Make the dummy error out whenever we attempt to do anything with it. raise NotImplementedError("Macro expander instance has not been set in `dyn`.") -make_dynvar(_macro_expander=_NoExpander()) +_make_dynvar(_macro_expander=_NoExpander()) + +# Set up `unpythonic`'s AST markers to be deleted by the macro expander's global postprocessor. +# This way we can use AST markers for data-driven internal communication between macros. +from . import util +util.register_postprocessor_hook() diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 602021e4..5eb775a7 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -85,7 +85,7 @@ def transform(self, tree): return tree hascurry = self.state.hascurry - if type(tree) is Call and not isx(tree.func, "ExpandedAutorefMarker"): + if type(tree) is Call: if has_curry(tree): # detect decorated lambda with manual curry # the lambda inside the curry(...) is the next Lambda node we will descend into. hascurry = True diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index a36b0535..8fdbc407 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -3,7 +3,7 @@ __all__ = ["autoref"] -from ast import (Name, Load, Call, Lambda, With, Constant, arg, +from ast import (Name, Load, Call, Lambda, arg, Attribute, Subscript, Store, Del) from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 @@ -13,9 +13,9 @@ from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer -from .astcompat import getconstant, Str +from .astcompat import getconstant from .nameutil import isx -from .util import wrapwith, ExpandedAutorefMarker +from .util import ExpandedAutorefMarker from .letdoutil import isdo, islet, ExpandedDoView, ExpandedLetView from .testingtools import _test_function_names @@ -43,21 +43,25 @@ # # One possible clean-ish implementation is:: # -# with ExpandedAutorefMarker("o"): # no-op at runtime -# x # --> (lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))) -# x.a # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x")))).a -# x[s] # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))))[s] -# o # --> o (can only occur if an as-part is supplied) -# with ExpandedAutorefMarker("p"): -# x # --> (lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))) -# x.a # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))).a -# x[s] # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x")))[s] -# # when the inner autoref expands, it doesn't know about the outer one, so we will get this: -# o # --> (lambda _ar314: _ar314[1] if _ar314[0] else o)(_autoref_resolve((p, "o"))) -# o.x # --> ((lambda _ar314: _ar314[1] if _ar314[0] else o)(_autoref_resolve((p, "o")))).x -# o[s] # --> ((lambda _ar314: _ar314[1] if _ar314[0] else o)(_autoref_resolve((p, "o"))))[s] -# # the outer autoref needs the marker to know to skip this (instead of looking up o.p): -# p # --> p +# $ASTMarker: +# varname: 'o' +# body: +# x # --> (lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))) +# x.a # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x")))).a +# x[s] # --> ((lambda _ar271: _ar271[1] if _ar271[0] else x)(_autoref_resolve((o, "x"))))[s] +# o # --> o (can only occur if an as-part is supplied) +# $ASTMarker: +# varname: 'p' +# body: +# x # --> (lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))) +# x.a # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))).a +# x[s] # --> ((lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x")))[s] +# # when the inner autoref expands, it doesn't know about the outer one, so we will get this: +# o # --> (lambda _ar314: _ar314[1] if _ar314[0] else o)(_autoref_resolve((p, "o"))) +# o.x # --> ((lambda _ar314: _ar314[1] if _ar314[0] else o)(_autoref_resolve((p, "o")))).x +# o[s] # --> ((lambda _ar314: _ar314[1] if _ar314[0] else o)(_autoref_resolve((p, "o"))))[s] +# # the outer autoref needs the marker to know to skip this (instead of looking up o.p): +# p # --> p # # The lambda is needed, because the lexical-variable lookup for ``x`` must occur at the use site, # and it can only be performed by Python itself. We could modify ``_autoref_resolve`` to take @@ -77,12 +81,6 @@ # In reality, we also capture-and-assign the autoref'd expr into a gensym'd variable (instead of referring # to ``o`` and ``p`` directly), so that arbitrary expressions can be autoref'd without giving them # a name in user code. -# -# TODO: Consider whether we could use a `mcpyrate.markers.ASTMarker` (which could -# TODO: be deleted before the code reaches run time, instead of leaving it in like -# TODO: `UnpythonicExpandedMacroMarker`s are). May need a postprocess hook in the -# TODO: expander, so that we could register a function that deletes autoref markers -# TODO: at the expander's global postprocess pass. @parametricmacro def autoref(tree, *, args, syntax, expander, **kw): @@ -175,20 +173,6 @@ def _autoref(block_body, args, asname): o = asname.id if asname else gensym("_o") # Python itself guarantees asname to be a bare Name. - # TODO: We can't use `unpythonic.syntax.util.isexpandedmacromarker` here, because it - # TODO: doesn't currently understand markers with arguments. Extend it? - # - # with ExpandedAutorefMarker("_o42"): - def isexpandedautorefblock(tree): - if not (type(tree) is With and len(tree.items) == 1): - return False - ctxmanager = tree.items[0].context_expr - return (type(ctxmanager) is Call and - isx(ctxmanager.func, "ExpandedAutorefMarker") and - len(ctxmanager.args) == 1 and type(ctxmanager.args[0]) in (Constant, Str)) # Python 3.8+: ast.Constant - def getreferent(tree): - return getconstant(tree.items[0].context_expr.args[0]) - # (lambda _ar314: _ar314[1] if _ar314[0] else x)(_autoref_resolve((p, o, "x"))) def isautoreference(tree): return (type(tree) is Call and @@ -237,8 +221,8 @@ def transform(self, tree): elif isdo(tree): view = ExpandedDoView(tree) self.generic_withstate(tree, referents=referents + [view.body[0].args.args[0].arg]) # lambda e14: ... - elif isexpandedautorefblock(tree): - self.generic_withstate(tree, referents=referents + [getreferent(tree)]) + elif isinstance(tree, ExpandedAutorefMarker): + self.generic_withstate(tree, referents=referents + [tree.varname]) elif isautoreference(tree): # generated by an inner already expanded autoref block thename = getconstant(get_resolver_list(tree)[-1]) if thename in referents: @@ -250,11 +234,15 @@ def transform(self, tree): # # expands to: # - # with ExpandedAutorefMarker('_o5'): - # _o5 = e - # with ExpandedAutorefMarker('_o4'): - # _o4 = (lambda _ar13: (_ar13[1] if _ar13[0] else e2))(_autoref_resolve((_o5, 'e2'))) - # (lambda _ar9: (_ar9[1] if _ar9[0] else e))(_autoref_resolve((_o4, _o5, 'e'))) + # $ASTMarker: + # varname: '_o5' + # body: + # _o5 = e + # $ASTMarker: + # varname: '_o4' + # body: + # _o4 = (lambda _ar13: (_ar13[1] if _ar13[0] else e2))(_autoref_resolve((_o5, 'e2'))) + # (lambda _ar9: (_ar9[1] if _ar9[0] else e))(_autoref_resolve((_o4, _o5, 'e'))) # # so there's no "e" as referent; the actual referent has a gensymmed name. # Inside the body of the inner autoref, looking up "e" in e2 before falling @@ -269,11 +257,15 @@ def transform(self, tree): # # expands to: # - # with ExpandedAutorefMarker('outer'): - # outer = e - # with ExpandedAutorefMarker('inner'): - # inner = (lambda _ar17: (_ar17[1] if _ar17[0] else e2))(_autoref_resolve((outer, 'e2'))) - # outer # <-- !!! + # $ASTMarker: + # varname: 'outer' + # body: + # outer = e + # $ASTMarker: + # varname: 'inner' + # body: + # inner = (lambda _ar17: (_ar17[1] if _ar17[0] else e2))(_autoref_resolve((outer, 'e2'))) + # outer # <-- !!! # # Now this case is triggered; we get a bare `outer` inside the inner body. # TODO: Whether this wart is a good idea is another question... @@ -284,7 +276,7 @@ def transform(self, tree): else: add_to_resolver_list(tree, q[n[o]]) # _autoref_resolve((p, "x")) --> _autoref_resolve((p, o, "x")) return tree - elif type(tree) is Call and isx(tree.func, "ExpandedAutorefMarker"): # nested autorefs + elif isinstance(tree, ExpandedAutorefMarker): # nested autorefs return tree elif type(tree) is Name and (type(tree.ctx) is Load or not tree.ctx) and tree.id not in referents: tree = makeautoreference(tree) @@ -309,5 +301,4 @@ def transform(self, tree): for stmt in block_body: newbody.append(AutorefTransformer(referents=always_skip + [o]).visit(stmt)) - return wrapwith(item=q[h[ExpandedAutorefMarker](u[o])], - body=newbody) + return ExpandedAutorefMarker(body=newbody, varname=o) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 01d9dcec..e89665ea 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -14,7 +14,7 @@ from mcpyrate.walkers import ASTTransformer from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, - isx, getname, is_decorator, wrapwith) + isx, getname, is_decorator) from .letdoutil import islet, isdo, ExpandedLetView from .nameutil import is_unexpanded_expr_macro from ..lazyutil import Lazy, passthrough_lazy_args, force, force1, maybe_force_args @@ -707,7 +707,7 @@ def transform_starred(tree, dstarred=False): # _autoref_resolve doesn't need any special handling elif (isdo(tree) or is_decorator(tree.func, "namelambda") or any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, _expanded_lazy_name) or - any(isx(tree.func, s) for s in ("_autoref_resolve", "ExpandedAutorefMarker"))): + isx(tree.func, "_autoref_resolve")): # here we know the operator (.func) to be one of specific names; # don't transform it to avoid confusing lazyrec[] (important if this # is an inner call in the arglist of an outer, lazy call, since it @@ -831,7 +831,10 @@ def transform_starred(tree, dstarred=False): # The second strict callee may get promises instead of values, because the # strict trampoline does not have the maybe_force_args (that usually forces the args # when lazy code calls into strict code). - return wrapwith(item=q[h[dyn.let](_build_lazy_trampoline=True)], - body=newbody) + with q as quoted: + with h[dyn.let](_build_lazy_trampoline=True): + with a: + newbody + return quoted # ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index aff82cd0..6836a655 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -21,7 +21,6 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym -from mcpyrate.markers import ASTMarker from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer, ASTVisitor @@ -31,7 +30,8 @@ from .util import (isx, isec, detect_callec, detect_lambda, has_tco, sort_lambda_decorators, - suggest_decorator_index, ExpandedContinuationsMarker, wrapwith, isexpandedmacromarker) + suggest_decorator_index, + UnpythonicASTMarker, ExpandedContinuationsMarker) from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView from ..dynassign import dyn @@ -653,7 +653,7 @@ def call_cc(tree, **kw): """ if _continuations_level.value < 1: raise SyntaxError("call_cc[] is only meaningful in a `with continuations` block.") # pragma: no cover, not meant to hit the expander (expanded away by `with continuations`) - return UnpythonicCallCcMarker(tree) + return CallCcMarker(body=tree) # -------------------------------------------------------------------------------- @@ -707,7 +707,7 @@ def _tco(block_body): for stmt in block_body: # skip nested, already expanded "with continuations" blocks # (needed to support continuations in the Lispython dialect, which applies tco globally) - if isexpandedmacromarker("ExpandedContinuationsMarker", stmt): + if isinstance(stmt, ExpandedContinuationsMarker): new_block_body.append(stmt) continue @@ -765,9 +765,9 @@ def cc(value): _continuations_level = NestingLevelTracker() # for checking validity of call_cc[] -class UnpythonicContinuationsMarker(ASTMarker): +class ContinuationsMarker(UnpythonicASTMarker): """AST marker related to the unpythonic's continuations (call_cc) subsystem.""" -class UnpythonicCallCcMarker(UnpythonicContinuationsMarker): +class CallCcMarker(ContinuationsMarker): """AST marker denoting a `call_cc[]` invocation.""" @@ -867,7 +867,7 @@ def data_cb(tree): # transform an inert-data return value into a tail-call to c def iscallcc(tree): if type(tree) not in (Assign, Expr): return False - return isinstance(tree.value, UnpythonicCallCcMarker) + return isinstance(tree.value, CallCcMarker) def split_at_callcc(body): if not body: return [], None, [] @@ -921,7 +921,7 @@ def maybe_starred(expr): # return expr.id or set starget else: raise SyntaxError(f"call_cc[]: expected an assignment or a bare expr, got {stmt}") # pragma: no cover # extract the function call(s) - if not isinstance(stmt.value, UnpythonicCallCcMarker): # both Assign and Expr have a .value + if not isinstance(stmt.value, CallCcMarker): # both Assign and Expr have a .value assert False # we should get only valid call_cc[] invocations that pass the `iscallcc` test # pragma: no cover theexpr = stmt.value.body # discard the AST marker if not (type(theexpr) in (Call, IfExp) or (type(theexpr) in (Constant, NameConstant) and getconstant(theexpr) is None)): @@ -1102,7 +1102,8 @@ def transform(self, tree): # set up the default continuation that just returns its args # (the top-level "cc" is only used for continuations created by call_cc[] at the top level of the block) - new_block_body = [Assign(targets=[q[n["cc"]]], value=q[h[identity]])] + with q as new_block_body: + cc = h[identity] # noqa: F841, only quoted # transform all defs (except the chaining handler), including those added by call_cc[]. for stmt in block_body: @@ -1116,8 +1117,7 @@ def transform(self, tree): # Leave a marker so "with tco", if applied, can ignore the expanded "with continuations" block # (needed to support continuations in the Lispython dialect, since it applies tco globally.) - return wrapwith(item=q[h[ExpandedContinuationsMarker]], - body=new_block_body) + return ExpandedContinuationsMarker(body=new_block_body) # ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tests/test_util.py b/unpythonic/syntax/tests/test_util.py index 18ad316c..738f09dc 100644 --- a/unpythonic/syntax/tests/test_util.py +++ b/unpythonic/syntax/tests/test_util.py @@ -14,10 +14,9 @@ suggest_decorator_index, is_lambda_decorator, is_decorated_lambda, destructure_decorated_lambda, sort_lambda_decorators, - transform_statements, eliminate_ifones, - wrapwith, isexpandedmacromarker) + transform_statements, eliminate_ifones) -from ast import Call, Name, Constant, Expr, With, withitem +from ast import Call, Name, Constant, Expr from ...ec import call_ec, throw # just so hq[] captures them, like in real code @@ -256,34 +255,6 @@ def ishello(tree): result = eliminate_ifones(eliminate_ifones_testdata8) test[len(result) == 1 and ishello(result[0])] - with testset("wrapwith"): - with q as wrapwith_testdata: - 42 - wrapped = wrapwith(q[n["ExampleContextManager"]], wrapwith_testdata) - test[type(wrapped) is list] - thewith = wrapped[0] - test[type(thewith) is With] - test[type(thewith.items[0]) is withitem] - ctxmanager = thewith.items[0].context_expr - test[type(ctxmanager) is Name] - test[ctxmanager.id == "ExampleContextManager"] - firststmt = thewith.body[0] - test[type(firststmt) is Expr] - test[type(firststmt.value) in (Constant, Num)] # Python 3.8+: ast.Constant - test[getconstant(firststmt.value) == 42] # Python 3.8+: ast.Constant - - with testset("isexpandedmacromarker"): - with q as ismarker_testdata1: - with ExampleMarker: # noqa: F821 - ... - with q as ismarker_testdata2: - with NotAMarker1, NotAMarker2: # noqa: F821 - ... - test[isexpandedmacromarker("ExampleMarker", ismarker_testdata1[0])] - test[not isexpandedmacromarker("AnotherMarker", ismarker_testdata1[0])] # right AST node type, different marker - test[not isexpandedmacromarker("NotAMarker1", ismarker_testdata2[0])] # a marker must be the only ctxmanager in the `with` - test[not isexpandedmacromarker("ExampleMarker", q["surprise!"])] # wrong AST node type - if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 42daff9a..78c697d7 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -11,15 +11,15 @@ "has_tco", "has_curry", "has_deco", "sort_lambda_decorators", "suggest_decorator_index", "eliminate_ifones", "transform_statements", - "wrapwith", - "isexpandedmacromarker", "UnpythonicExpandedMacroMarker", + "UnpythonicASTMarker", "UnpythonicExpandedMacroMarker", "ExpandedContinuationsMarker", "ExpandedAutorefMarker"] from functools import partial -from ast import (Call, Lambda, FunctionDef, AsyncFunctionDef, - If, With, withitem, stmt) +from ast import Call, Lambda, FunctionDef, AsyncFunctionDef, If, stmt +from mcpyrate.core import add_postprocessor +from mcpyrate.markers import ASTMarker, delete_markers from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer, ASTVisitor @@ -408,92 +408,36 @@ def transform(self, tree): return tree return StatementTransformer().visit(body) -def wrapwith(item, body): - """Wrap ``body`` with a single-item ``with`` block, using ``item``. +# -------------------------------------------------------------------------------- +# AST markers. - ``item`` must be an expr, used as ``context_expr`` of the ``withitem`` node. +class UnpythonicASTMarker(ASTMarker): + """Base class for all AST markers used by `unpythonic`.""" +class UnpythonicExpandedMacroMarker(UnpythonicASTMarker): + """AST marker base class for expanded `unpythonic.syntax` macros.""" - ``body`` must be a ``list`` of AST nodes. - - Syntax transformer. Returns the wrapped body. - - This function is intended to be called from macro implementations. We leave out - the source location information, so that the macro expander can auto-fill it. - """ - wrapped = With(items=[withitem(context_expr=item, optional_vars=None)], - body=body) - return [wrapped] - -def isexpandedmacromarker(typename, tree): - """Return whether tree is a specific expanded macro AST marker. Used by block macros. - - That is, whether ``tree`` is a ``with`` block with a single context manager, - which is represented by a ``Name`` whose ``id`` matches the given ``typename``. - - Example. If ``tree`` is the AST for the following code:: - - with ExpandedContinuationsMarker: - ... - - then ``isexpandedmacromarker("ExpandedContinuationsMarker", tree)`` returns ``True``. - - **NOTE**: The markers this function detects remain in the AST at run time; - they inherit from `unpythonic.syntax.util.UnpythonicExpandedMacroMarker`. - They are semantically different from `mcpyrate.markers.ASTMarker`, which - are compiled away (and must all be deleted before handing the AST over to - Python's `compile`). - """ - if type(tree) is not With or len(tree.items) != 1: - return False - ctxmanager = tree.items[0].context_expr - return isx(ctxmanager, typename) - -# We use a custom metaclass to make __enter__ and __exit__ callable on the class -# instead of requiring an instance. -# -# Note ``thing.dostuff(...)`` means ``Thing.dostuff(thing, ...)``; the method -# is looked up *on the class* of the instance ``thing``, not on the instance -# itself. Hence, to make method lookup succeed when we have no instance, the -# method should be defined on the class of the class, i.e. *on the metaclass*. -# https://stackoverflow.com/questions/20247841/using-delitem-with-a-class-object-rather-than-an-instance-in-python -class UnpythonicExpandedMacroMarker(type): - """Metaclass for AST markers used by block macros. - - This can be used by block macros to tell other block macros that a section - of the AST is an already-expanded block of a given kind (so that others can - tune their processing or skip it, as appropriate). At run time a marker - does nothing. - - The difference to `mcpyrate.markers.ASTMarker` is that `mcpyrate`'s is a - compile-time thing only (and must be deleted from the AST before the AST - is handed over to Python's `compile`), whereas this one remains in the - AST at run time. - - Usage:: - - with SomeMarker: - ... # expanded code goes here - - We provide a custom metaclass so that there is no need to instantiate - ``SomeMarker``; suitable no-op ``__enter__`` and ``__exit__`` methods - are defined on the metaclass, so e.g. ``SomeMarker.__enter__`` is valid. - """ - def __enter__(cls): - pass # pragma: no cover - def __exit__(cls, exctype, excvalue, traceback): - pass # pragma: no cover - -class ExpandedContinuationsMarker(metaclass=UnpythonicExpandedMacroMarker): +class ExpandedContinuationsMarker(UnpythonicExpandedMacroMarker): """AST marker for an expanded "with continuations" block.""" - pass # pragma: no cover -# This one must be "instantiated", because we need to pass information at -# macro expansion time using the ctor call syntax, e.g. `ExpandedAutorefMarker("o")`. -class ExpandedAutorefMarker(metaclass=UnpythonicExpandedMacroMarker): +class ExpandedAutorefMarker(UnpythonicExpandedMacroMarker): """AST marker for an expanded "with autoref[o]" block.""" - def __init__(self, varname): - self.varname = varname # not needed, but doesn't hurt either. - def __enter__(cls): - pass # pragma: no cover - def __exit__(cls, exctype, excvalue, traceback): - pass # pragma: no cover + def __init__(self, body, varname): + super().__init__(body) + self.varname = varname + self._fields += ["varname"] + +# The point of having these two functions is: +# - `__init__` must explicitly enable the hook, thus making its existence +# obvious, since the entry-point source file for the macro layer has an +# obvious function call, instead of having the hook secretly registered +# by an innocuous-looking utility module. +# +# - We could register `partial(delete_markers, cls=UnpythonicASTMarker)`, +# but then we would be unable to `remove_postprocessor` it later, because +# the function object itself is the key used for unregistering. Currently +# we don't need to do that, but it's nice to have the possibility. +def register_postprocessor_hook(): + """Set up global postprocessor hook for `mcpyrate` to nuke `unpythonic`'s AST markers from the final tree.""" + add_postprocessor(_delete_unpythonic_ast_markers) +def _delete_unpythonic_ast_markers(tree): + return delete_markers(tree, cls=UnpythonicASTMarker) From b6c002c0ac73d703f8f3d94b3dbc81b8ad664114 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 9 May 2021 01:41:15 +0300 Subject: [PATCH 227/832] upgrade mcpyrate --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index c33fd1df..4fe57592 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -mcpyrate>=3.4.1 +mcpyrate>=3.5.0 sympy>=1.4 From 4c488b15fcc48ad7e18fe85ffba1e202e798bd42 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 9 May 2021 02:21:23 +0300 Subject: [PATCH 228/832] finalize porting TODO comments --- unpythonic/syntax/__init__.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 0bf37070..8a78ddda 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -67,21 +67,21 @@ # If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! -# TODO: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. +# TODO: 0.16: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. -# TODO: Consider using run-time compiler access in macro tests, like `mcpyrate` itself does. This compartmentalizes testing so that the whole test module won't crash on a macro-expansion error. +# TODO: 0.16: Consider using run-time compiler access in macro tests, like `mcpyrate` itself does. This compartmentalizes testing so that the whole test module won't crash on a macro-expansion error. -# TODO: Return a compile-time marker from all block macros? +# TODO: 0.16: Return a compile-time marker from all block macros? Currently only macros that need to emit a marker for a specific reason (for working together with some specific macro) do so, namely `autoref` and `continuations`. # TODO: 0.16: move `scoped_transform` to `mcpyrate` as `ScopedASTTransformer` and `ScopedASTVisitor`. # TODO: 0.16: Add call-macros to `mcpyrate`. This allows the whole expression of `kw()`/`where()` to be detected as a macro invocation. (First, think whether this is a good idea.) -# TODO: Something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead. +# TODO: 0.16: Something like `unpythonic.syntax.nameutil` should probably live in `mcpyrate` instead. -# TODO: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... +# TODO: 0.16: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... -# TODO: Change decorator macro invocations to use [] instead of () to pass macro arguments. Requires Python 3.9. +# TODO: Far future: Change decorator macro invocations to use [] instead of () to pass macro arguments. Requires Python 3.9, so the earliest time to do this is when 3.9 becomes the minimum Python version for `unpythonic`. from .autocurry import * # noqa: F401, F403 from .autoref import * # noqa: F401, F403 From ab9c8363a6a4d1c64ca7fdc547850c707d04c773 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 9 May 2021 02:23:09 +0300 Subject: [PATCH 229/832] replace countlines.py with more advanced version from mcpyrate --- countlines.py | 83 +++++++++++++++++++++++++++++---------------------- 1 file changed, 48 insertions(+), 35 deletions(-) diff --git a/countlines.py b/countlines.py index 187a16c4..ab757165 100644 --- a/countlines.py +++ b/countlines.py @@ -1,60 +1,73 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -"""Estimate project size in lines of code. +"""Estimate project size in lines of code.""" -Ignores blank lines, docstrings, and whole-line comments.""" +# TODO: add sorting options: name, code count, SLOC count, code ratio. import os import re from operator import itemgetter def listpy(path): - return list(sorted(fn for fn in os.listdir(path) if fn.endswith(".py"))) + return list(sorted(filename for filename in os.listdir(path) if filename.endswith(".py"))) -def loc(code, blanks, docstrings, comments): # blanks et al.: include this item? +def count_sloc(code, *, blanks, docstrings, comments): + """blanks et al.: include this item?""" if not docstrings: # TODO: make sure it's a docstring (and not some other """...""" string) code = re.sub(r'""".*?"""', r'', code, flags=(re.MULTILINE + re.DOTALL)) + code = re.sub(r"'''.*?'''", r'', code, flags=(re.MULTILINE + re.DOTALL)) lines = code.split("\n") if not blanks: lines = [line for line in lines if line.strip()] if not comments: - # TODO: removes only whole-line comments. - lines = [line for line in lines if not line.strip().startswith("#")] + lines = [line for line in lines if not line.strip().startswith("#")] # ignore whole-line comments return len(lines) -def analyze(items, blanks=False, docstrings=False, comments=False): - grandtotal = 0 - for name, p in items: - path = os.path.join(*p) - files = listpy(path) - ns = [] - for fn in files: - with open(os.path.join(path, fn), "rt", encoding="utf-8") as f: +def report(paths): + print(f"Code size for {os.getcwd()}") + def format_name(s, width=25): + return s.ljust(width) + def format_number(n, width=5): + return str(n).rjust(width) + def format_path(s): # ./subdir/something + def label(s): + if s == ".": + return "top level" + return s[2:] + return format_name(label(s)) + codes_grandtotal = 0 + slocs_grandtotal = 0 + for path in paths: + filenames = listpy(path) + results = [] + for filename in filenames: + with open(os.path.join(path, filename), "rt", encoding="utf-8") as f: content = f.read() - ns.append(loc(content, blanks, docstrings, comments)) - # report - print(f"{name}:") - for fn, n in sorted(zip(files, ns), key=itemgetter(1)): - print(f" {fn} {n}") - grouptotal = sum(ns) - print(f" total for {name} {grouptotal}") - grandtotal += grouptotal - print(f"grand total {grandtotal}") + code = count_sloc(content, blanks=False, docstrings=False, comments=False) + sloc = count_sloc(content, blanks=True, docstrings=True, comments=True) + results.append((code, sloc)) + + if results: + codes, slocs = zip(*results) + codes = sum(codes) + slocs = sum(slocs) + print(f"\n {format_path(path)} {format_number(codes)} / {format_number(slocs)} {int(codes / slocs * 100):d}% code") + for filename, (code, sloc) in sorted(zip(filenames, results), key=itemgetter(1)): + print(f" {format_name(filename)} {format_number(code)} / {format_number(sloc)} {int(code / sloc * 100):d}% code") + codes_grandtotal += codes + slocs_grandtotal += slocs + print(f"\n{format_name('Total')} {format_number(codes_grandtotal)} / {format_number(slocs_grandtotal)} {int(codes_grandtotal / slocs_grandtotal * 100):d}% code") def main(): - items = (("top level", ["."]), - ("regular code", ["unpythonic"]), - ("regular code tests", ["unpythonic", "tests"]), - ("testing framework (not counting macros)", ["unpythonic", "test"]), - ("REPL/networking code", ["unpythonic", "net"]), - ("REPL/networking tests", ["unpythonic", "net", "tests"]), - ("macros", ["unpythonic", "syntax"]), - ("macro tests", ["unpythonic", "syntax", "tests"])) - print("Raw (with blanks, docstrings and comments)") - analyze(items, blanks=True, docstrings=True, comments=True) - print("\nFiltered (non-blank code lines only)") - analyze(items) + blacklist = [".git", "build", "dist", "__pycache__", "00_stuff"] + paths = [] + for root, dirs, files in os.walk("."): + paths.append(root) + for x in blacklist: + if x in dirs: + dirs.remove(x) + report(sorted(paths)) if __name__ == '__main__': main() From 3b5e5aff3ba3bd758151b7bf5aa5f2abb07cd82f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 01:49:27 +0300 Subject: [PATCH 230/832] update changelog --- CHANGELOG.md | 161 +++++++++++++++++++++++++++------------------------ 1 file changed, 85 insertions(+), 76 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f121665d..63b663a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,12 +1,13 @@ -**0.15.0** (in progress; updated 5 May 2021) - *"We say 'howdy' around these parts"* edition: +**0.15.0** (in progress; updated 10 May 2021) - *"We say 'howdy' around these parts"* edition: -Beside introducing **dialects** (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This unfortunately introduces some breaking changes; see below. While at it, we have also taken the opportunity to make also any previously scheduled breaking changes. +Beside introducing **dialects** (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This introduces some breaking changes, so we have also taken the opportunity to apply any such that were previously scheduled. **IMPORTANT**: -- **Minimum Python language version is now 3.6**. -- The optional **macro expander is now [`mcpyrate`](https://github.com/Technologicat/mcpyrate)**. -- For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). +- Minimum Python language version is now 3.6. + - We support 3.6, 3.7, 3.8, 3.9 and PyPy3 (language versions 3.6 and 3.7). + - For future plans, see our [Python language version support status](https://github.com/Technologicat/unpythonic/issues/1). +- The optional macro expander is now [`mcpyrate`](https://github.com/Technologicat/mcpyrate). If you still need `unpythonic` for Python 3.4 or 3.5, use version 0.14.3, which is the final version of `unpythonic` that supports those language versions. @@ -18,9 +19,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Dialects!** New module `unpythonic.dialects`, providing [some example dialects](doc/dialects.md) that demonstrate what can be done with a [dialects system](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) (i.e. full-module code transformer) together with a kitchen-sink language extension macro package such as `unpythonic`. - These dialects have been moved from the now-obsolete [`pydialect`](https://github.com/Technologicat/pydialect) project and ported to use [`mcpyrate`](https://github.com/Technologicat/mcpyrate). -- **Python 3.8 and 3.9 support added.** - -- Robustness: several auxiliary syntactic constructs now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. +- **Improved robustness**: several auxiliary syntactic constructs now detect *at macro expansion time* if they appear outside any valid lexical context, and raise `SyntaxError` (with a descriptive message) if so. - The full list is: - `call_cc[]`, for `with continuations` - `it`, for `aif[]` @@ -30,7 +29,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `with expr`/`with block`, for `with let_syntax`/`with abbrev` - Previously these constructs could only raise an error at run time, and not all of them could detect the error even then. -- For syntactic consistency, allow env-assignment notation and brackets to declare bindings in the `let` family of macros. The preferred syntaxes for the `let` macro are now: +- **Syntactic consistency**: allow env-assignment notation and brackets to declare bindings in the `let` family of macros. The preferred syntaxes for the `let` macro are now: ```python let[x << 42, y << 9001][...] # lispy expr @@ -45,97 +44,107 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi ``` Similarly for `letseq`, `letrec`, and the decorator versions; and for the expr forms of `let_syntax`, `abbrev`. The reason for preferring this notation is that it is consistent with both `unpythonic`'s env-assignments (`let` bindings live in an `env`) and the use of brackets to denote macro invocations. - However, all the following are also accepted, with the meaning exactly the same: + To ease backwards compatibility, we still accept the syntax used up to v0.14.3, too. + + Also, from symmetry and usability viewpoints, if a mix of brackets and parentheses are used, it hardly makes sense to require some specific mix - so this has been extended so that the choice of delimiter doesn't matter. All the following are also accepted, with the meaning exactly the same as above: ```python - let[(x << 42, y << 9001) in ...] - let[..., where(x << 42, y << 9001)] - let[[x, 42], [y, 9001]][...] + let[[x, 42], [y, 9001]][...] # best visual consistency let[(x, 42), (y, 9001)][...] - let[[[x, 42], [y, 9001]] in ...] + let([x, 42], [y, 9001])[...] + let((x, 42), (y, 9001))[...] # like up to v0.14.3 + let[[[x, 42], [y, 9001]] in ...] # best visual consistency let[[(x, 42), (y, 9001)] in ...] let[([x, 42], [y, 9001]) in ...] - let[((x, 42), (y, 9001)) in ...] - let[..., where[[x, 42], [y, 9001]]] + let[((x, 42), (y, 9001)) in ...] # like up to v0.14.3 + let[(x << 42, y << 9001) in ...] + let[..., where[[x, 42], [y, 9001]]] # best visual consistency let[..., where[(x, 42), (y, 9001)]] let[..., where([x, 42], [y, 9001])] - let[..., where((x, 42), (y, 9001))] + let[..., where((x, 42), (y, 9001))] # like up to v0.14.3 + let[..., where(x << 42, y << 9001)] ``` For a single binding, these are also accepted: ```python let[x, 42][...] - let[(x << 42) in ...] + let(x, 42)[...] # like up to v0.14.3 let[[x, 42] in ...] - let[(x, 42) in ...] - let[..., where(x << 42)] + let[(x, 42) in ...] # like up to v0.14.3 + let[(x << 42) in ...] let[..., where[x, 42]] - let[..., where(x, 42)] + let[..., where(x, 42)] # like up to v0.14.3 + let[..., where(x << 42)] ``` + These alternate syntaxes will be supported at least as long as we accept parentheses to pass macro arguments; but in new code, please use the preferred syntaxes. -- `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - -- Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. - -- All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) +- **Miscellaneous.** + - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) + - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. + - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) + - Python 3.8 and 3.9 support added. **Non-breaking changes**: -- The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - -- Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. The `continuations` macro also outputs a hygienically captured `aif` when transforming an `or` expression that occurs in tail position. - - This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) - - The implicit do (extra bracket syntax) also expands as a hygienically captured `do`, but e.g. in `let[]` it will then expand immediately (due to `let`'s inside-out expansion order) before control returns to the macro stepper. If you want to see the implicit `do[]` invocation, use the `"detailed"` mode of the stepper, which shows individual macro invocations even when expanding inside-out: `step_expansion["detailed"][...]`, `with step_expansion["detailed"]:`. +- **Changes to how some macros expand.** + - Some macros, notably `letseq`, `do0`, and `lazyrec`, now expand into hygienic macro captures of other macros. The `continuations` macro also outputs a hygienically captured `aif` when transforming an `or` expression that occurs in tail position. + - This allows `mcpyrate.debug.step_expansion` to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it *might* be a good idea.) + - The implicit do (extra bracket syntax) also expands as a hygienically captured `do`, but e.g. in `let[]` it will then expand immediately (due to `let`'s inside-out expansion order) before control returns to the macro stepper. If you want to see the implicit `do[]` invocation, use the `"detailed"` mode of the stepper, which shows individual macro invocations even when expanding inside-out: `step_expansion["detailed"][...]`, `with step_expansion["detailed"]:`. -- CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). + - The `do[]` and `do0[]` macros now expand outside-in. The main differences from a user perspective are: + - Any source code captures (such as those performed by `test[]`) show the expanded output of `do` and `do0`, because that's what they receive. (For tests, you may want to use the macro `with expand_testing_macros_first`, which see.) + - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. -- CI: Test coverage improved to 94%. +- **Miscellaneous.** + - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. + - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). + - CI: Test coverage improved to 94%. **Breaking changes**: -- Migrate to the `mcpyrate` macro expander; **MacroPy support dropped**. - - **Macro arguments are now passed using brackets**, `macroname[args]`, instead of parentheses. - - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. - - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. - - `mcpyrate` should report test coverage for macro-using code correctly; no need for `# pragma: no cover` in block macro invocations or in quasiquoted code. - -- The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - - **Any imports of these constructs in user code should be modified to point to the new locations.** - - Unlike `macropy`'s `Lazy`, our `Lazy` does not define `__call__`; instead, it defines the method `force`, which has the same effect (it computes if necessary, and then returns the value of the promise). - - The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else it is available to be used as a regular variable. - - `f[]` now respects nesting: an invocation of `f[]` will not descend into another nested `f[]`. - - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. - - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import. Now you **must** import also the macro `f` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `f` is currently bound to. - -- **Rename the `curry` macro** to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. - -- As promised, names deprecated during 0.14.x have been removed. Old name on the left, new name on the right: - - `m` → `imathify` - - `mg` → `gmathify` - - `setescape` → `catch` - - `escape` → `throw` - - `getvalue`, `runpipe` → `exitpipe` (combined into one) - - **CAUTION**: Now `exitpipe` is an `unpythonic.symbol.sym` (like a Lisp symbol). This is not compatible with existing, pickled `exitpipe` instances; it used to be an instance of the class `Getvalue`, which has been removed. (There's not much reason to pickle an `exitpipe` instance, but we're mentioning this for the sake of completeness.) - -- Drop support for deprecated argument format for `raisef`. Now the usage is `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. - -- Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. - -- Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). This fixes a `flake8` E741 warning, as well as is more descriptive. - -- Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) - -- The `do[]` and `do0[]` macros now expand outside-in. The main differences from a user perspective are: - - Any source code captures (such as those performed by `test[]`) show the expanded output of `do` and `do0`, because that's what they receive. - - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. - -- The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - -- Remove the internal utility class `unpythonic.syntax.util.ASTMarker`. We now have `mcpyrate.markers.ASTMarker`, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time. - -- Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - -- Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. +- **New macro expander `mcpyrate`; MacroPy support dropped**. + - **API differences.** + - Macro arguments are now passed using brackets, `macroname[args][...]`, `with macroname[args]`, `@macroname[args]`, instead of parentheses. + - Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. `@dlet[(x, 42)]` is a syntax error, but `@dlet((x, 42))` is fine). This has been fixed in Python 3.9. + - If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for `unpythonic`. + - If you write your own macros, note `mcpyrate` is not drop-in compatible with MacroPy or `mcpy`. See [its documentation](https://github.com/Technologicat/mcpyrate#documentation) for details. + - **Behavior differences.** + - `mcpyrate` should report test coverage for macro-using code correctly; no need for `# pragma: no cover` in block macro invocations or in quasiquoted code. + +- **Previously scheduled API changes**. + - As promised, names deprecated during 0.14.x have been removed. Old name on the left, new name on the right: + - `m` → `imathify` (consistency with the rest of `unpythonic`) + - `mg` → `gmathify` (consistency with the rest of `unpythonic`) + - `setescape` → `catch` (Lisp family standard name) + - `escape` → `throw` (Lisp family standard name) + - `getvalue`, `runpipe` → `exitpipe` (combined into one) + - **CAUTION**: `exitpipe` already existed in v0.14.3, but beginning with v0.15.0, it is now an `unpythonic.symbol.sym` (like a Lisp symbol). This is not compatible with existing, pickled `exitpipe` instances; it used to be an instance of the class `Getvalue`, which has been removed. (There's not much reason to pickle an `exitpipe` instance, but we're mentioning this for the sake of completeness.) + - Drop support for deprecated argument format for `raisef`. Now the usage is `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. + +- **Other backward-incompatible API changes.** + - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. + - **API differences.** + - The macros `lazy` and `f` can be imported from the syntax interface module, `unpythonic.syntax`, and the class `Lazy` is available at the top level of `unpythonic`. + - Unlike `macropy`'s `Lazy`, our `Lazy` does not define `__call__`; instead, it defines the method `force`, which has the same effect (it computes if necessary, and then returns the value of the promise). + - When you import the macro `quicklambda`, you **must** import also the macro `f`. + - The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else it is available to be used as a regular variable. + - **Behavior differences.** + - `f[]` now respects nesting: an invocation of `f[]` will not descend into another nested `f[]`. + - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. + - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import, by internally querying the expander to determine the name(s) the macro `f` is currently bound to. + - Rename the `curry` macro to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. + - Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) + - Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. + - This was an oversight when this function was added; most other functions in `unpythonic.it` have been curry-friendly from the beginning. + - Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). + - These are mostly used internally, but technically a part of the public API. + - This change fixes a `flake8` [E741](https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes) warning, and the new name for the parameter is more descriptive. + +- **Miscellaneous.** + - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + - Remove the internal utility class `unpythonic.syntax.util.ASTMarker`. We now have `mcpyrate.markers.ASTMarker`, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time. + - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. + - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. **Fixed**: From 0b4e9b132f910f4cc6ee6dcbcebd6807e46dd037 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 01:58:31 +0300 Subject: [PATCH 231/832] update "coming soon" notice --- README.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index bbf00f76..e478a80d 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,11 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f **As of April 2021, `unpythonic` 0.15 is Coming Soon™.** -As of [7bb1198](https://github.com/Technologicat/unpythonic/commit/7bb1198605087f1dd7ca292e33afd53e5aa9721d), the initial porting effort of `unpythonic` to Python 3.8 and the new [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander is complete. In fact, if you want to play around with 0.15-pre, the code is already in `master`. +As of [3b5e5af](https://github.com/Technologicat/unpythonic/commit/3b5e5aff3ba3bd758151b7bf5aa5f2abb07cd82f), the code itself is in a releasable state, and it is already in `master`. All that remains is an extensive documentation review. The changelog is known to be up to date, but something may still need an update in all the other parts of documentation. -The codebase already fully works on all supported Python versions, and passes all automated tests. However, I plan to take the opportunity to polish certain parts before release, and this may take a while. A living TODO list can be found at the beginning of [`unpythonic/syntax/__init__.py`](unpythonic/syntax/__init__.py). Be aware that the plan is tentative, and items might not be listed in a reasonable order. Some of the planned changes might not make the cut for 0.15. +The new version requires Python 3.6 or above, and optionally the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander. Python 3.4 and 3.5, and the MacroPy macro expander, are no longer supported by `unpythonic`. -For details, see [the 0.15 milestone](https://github.com/Technologicat/unpythonic/milestone/1). - -I'm also considering renaming 0.15 to 1.0, since the codebase is mostly stable at this point, and we have already adhered to [semantic versioning](https://semver.org/) since 2019, anyway (albeit with a leading zero). +The release will be numbered **0.15.0**, even though the codebase is mostly stable at this point, and we have already adhered to [semantic versioning](https://semver.org/) since 2019 (albeit with a leading zero). The reason is that the next major version has been known under this development version number for such a long time that it makes no sense to renumber it now. ### Dependencies From 4c36b1526c59d9f9072c3e6d6e8b760d29d45f36 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 01:58:51 +0300 Subject: [PATCH 232/832] oops, update the month too --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e478a80d..e669c074 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f ### New version soon! -**As of April 2021, `unpythonic` 0.15 is Coming Soon™.** +**As of May 2021, `unpythonic` 0.15 is Coming Soon™.** As of [3b5e5af](https://github.com/Technologicat/unpythonic/commit/3b5e5aff3ba3bd758151b7bf5aa5f2abb07cd82f), the code itself is in a releasable state, and it is already in `master`. All that remains is an extensive documentation review. The changelog is known to be up to date, but something may still need an update in all the other parts of documentation. From bec6d0cf4fdcabeb68245868f2d8cb12bc7f2b10 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 02:00:31 +0300 Subject: [PATCH 233/832] simplify explanation of supported versions since we have CI --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index e669c074..573782bd 100644 --- a/README.md +++ b/README.md @@ -26,9 +26,7 @@ None required. - [mcpyrate](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -The officially supported language versions are **CPython 3.8** and **PyPy3** (language version 3.7). [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). - -The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. +The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation From f8356b01805815206881c43e93149023098cf8fd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 02:01:12 +0300 Subject: [PATCH 234/832] style: mcpyrate is a package name. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 573782bd..8967b41d 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ The release will be numbered **0.15.0**, even though the codebase is mostly stab None required. - - [mcpyrate](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. + - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). From d38af7700f9c7a350f89f215ef4ba41127ab3b08 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 02:31:57 +0300 Subject: [PATCH 235/832] update design-notes.md --- doc/design-notes.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index e984fbd1..6da423ce 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -279,26 +279,25 @@ More on type systems: - For the interested reader, grep the source code for ``userlambdas``. - Support a limited form of *decorated lambdas*, i.e. trees of the form ``f(g(h(lambda ...: ...)))``. - The macros will reorder a chain of lambda decorators (i.e. nested calls) to use the correct ordering, when only known decorators are used on a literal lambda. - - This allows some combos such as ``tco``, ``unpythonic.fploop.looped``, ``curry``. + - This allows some combos such as ``tco``, ``unpythonic.fploop.looped``, ``autocurry``. - Only decorators provided by ``unpythonic`` are recognized, and only some of them are supported. For details, see ``unpythonic.regutil``. - If you need to combo ``unpythonic.fploop.looped`` and ``unpythonic.ec.call_ec``, use ``unpythonic.fploop.breakably_looped``, which does exactly that. - The problem with a direct combo is that the required ordering is the trampoline (inside ``looped``) outermost, then ``call_ec``, and then the actual loop, but because an escape continuation is only valid for the dynamic extent of the ``call_ec``, the whole loop must be run inside the dynamic extent of the ``call_ec``. - ``unpythonic.fploop.breakably_looped`` internally inserts the ``call_ec`` at the right step, and gives you the ec as ``brk``. - For the interested reader, look at ``unpythonic.syntax.util``. - - ``namedlambda`` is a two-pass macro. In the outside-in pass, it names lambdas inside ``let[]`` expressions before they are expanded away. The inside-out pass of ``namedlambda`` must run after ``autocurry`` to analyze and transform the auto-curried code produced by ``with autocurry``. In most cases, placing ``namedlambda`` in a separate outer ``with`` block runs both operations in the correct order. + - ``namedlambda`` is a two-pass macro. In the outside-in pass, it names lambdas inside ``let[]`` expressions before they are expanded away. The inside-out pass of ``namedlambda`` must run after ``autocurry`` to analyze and transform the auto-curried code produced by ``with autocurry``. - - ``autoref`` does not need in its output to be curried (hence after ``curry`` to gain some performance), but needs to run before ``lazify``, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what ``namedlambda`` does, so it does not matter in which exact order these two run. + - ``autoref`` does not need in its output to be curried (hence after ``autocurry`` to gain some performance), but needs to run before ``lazify``, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what ``namedlambda`` does, so it does not matter in which exact order these two run. - ``lazify`` is a rather invasive rewrite that needs to see the output from most of the other macros. - ``envify`` needs to see the output of ``lazify`` in order to shunt function args into an unpythonic ``env`` without triggering the implicit forcing. - - Some of the block macros can be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``autocurry`` and ``namedlambda``) require their own ``with`` statement. - - This was the case with MacroPy [[issue report](https://github.com/azazel75/macropy/issues/21)] [[PR](https://github.com/azazel75/macropy/pull/22)]. Should work in `mcpyrate`, but needs testing. - - If something goes wrong in the expansion of one block macro in a ``with`` statement that specifies several block macros, surprises may occur. - - When in doubt, use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. - - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. + - With MacroPy, it used to be so that some of the block macros could be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``autocurry`` and ``namedlambda``) required their own ``with`` statement. In `mcpyrate`, block macros can be comboed in the same ``with`` statement (and expansion order is *left-to-right*). + - See the relevant [[issue report](https://github.com/azazel75/macropy/issues/21)] and [[PR](https://github.com/azazel75/macropy/pull/22)]. + - When in doubt, you can use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. In ``mcpyrate``, this is almost equivalent to having the macros invoked in a single ``with`` statement, in the same order. + - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. (And of course, while testing, try to keep the input as simple as possible.) ## Miscellaneous notes From afb08765a78410a38d0e751a8f68d1f9825474a2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 02:32:29 +0300 Subject: [PATCH 236/832] styling --- doc/design-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 6da423ce..3829d1a2 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -295,7 +295,7 @@ More on type systems: - ``envify`` needs to see the output of ``lazify`` in order to shunt function args into an unpythonic ``env`` without triggering the implicit forcing. - With MacroPy, it used to be so that some of the block macros could be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``autocurry`` and ``namedlambda``) required their own ``with`` statement. In `mcpyrate`, block macros can be comboed in the same ``with`` statement (and expansion order is *left-to-right*). - - See the relevant [[issue report](https://github.com/azazel75/macropy/issues/21)] and [[PR](https://github.com/azazel75/macropy/pull/22)]. + - See the relevant [issue report](https://github.com/azazel75/macropy/issues/21) and [PR](https://github.com/azazel75/macropy/pull/22). - When in doubt, you can use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. In ``mcpyrate``, this is almost equivalent to having the macros invoked in a single ``with`` statement, in the same order. - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. (And of course, while testing, try to keep the input as simple as possible.) From c06a5e179433d04438e2c0412ab7f81b24b1931f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 02:37:22 +0300 Subject: [PATCH 237/832] wording --- doc/design-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 3829d1a2..183ca1d5 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -80,7 +80,7 @@ If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.co Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). -Of course, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. +In general, I like Python, and my hat's off to the devs. It's no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the above particular points, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. From 7713ac96ad858e727abd29d0c7becc295f73740c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:05:10 +0300 Subject: [PATCH 238/832] improve "What Belongs in Python?" --- doc/design-notes.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 183ca1d5..3954bea2 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -80,7 +80,15 @@ If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.co Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). -In general, I like Python, and my hat's off to the devs. It's no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the above particular points, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. +In general, I like Python, and my hat's off to the devs. It's no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the above particular points, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. I think that with macros, Python can be so much more than just a beginner's language, and that language-level extensibility is the logical endpoint of that. + +Of the particular points, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hanging when entering an accidentally infinite mutual recursion. But sometimes, I'd like to enable TCO selectively. If you ask for it, you know what to expect. Well, `unpythonic.syntax` has `with tco` precisely for that. + +I think multi-expression `lambda` is, on the surface, a good idea, but really the issue is that the `lambda` construct is broken. We would be much better off if `def` was an expression. Most of the time, anonymous functions aren't such a great idea, but defining closures inline is - sometimes in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. + +The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline. It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multiple-expression lambdas (to define the actions inline), I don't know what is. + +True multi-shot continuations... `unpythonic.syntax` has `with continuations` precisely for that, but I'm not sure if I'll ever use it in production code. However, it's something that's great to have for teaching the concept in a programming course, when teaching in Python. For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. Python's generators) are often all that's needed to simplify certain patterns, especially those involving backtracking. I'm a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. From 72a39bf408f7f07bcc97b7523d6042fafc66742b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:11:03 +0300 Subject: [PATCH 239/832] wording --- doc/design-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 3954bea2..a026cda2 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -86,13 +86,13 @@ Of the particular points, in my opinion TCO should at least be an option. I like I think multi-expression `lambda` is, on the surface, a good idea, but really the issue is that the `lambda` construct is broken. We would be much better off if `def` was an expression. Most of the time, anonymous functions aren't such a great idea, but defining closures inline is - sometimes in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. -The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline. It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multiple-expression lambdas (to define the actions inline), I don't know what is. +The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline. It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multi-expression lambdas (to define the actions inline), I don't know what is. True multi-shot continuations... `unpythonic.syntax` has `with continuations` precisely for that, but I'm not sure if I'll ever use it in production code. However, it's something that's great to have for teaching the concept in a programming course, when teaching in Python. For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. Python's generators) are often all that's needed to simplify certain patterns, especially those involving backtracking. I'm a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. -It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multiple-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I don't want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) +It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I don't want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) ## Killer features of Common Lisp From 74ee33e044efcdb66fa7c59d953a7b216fe4fbf7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:15:32 +0300 Subject: [PATCH 240/832] platform info now in README --- doc/design-notes.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index a026cda2..28821a5f 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -36,8 +36,6 @@ The main design considerations of `unpythonic` are simplicity, robustness, and m The whole library is pure Python. No foreign extensions are required. We also try to avoid depending on anything beyond "the Python standard", to help `unpythonic` run on any conforming Python implementation. (Provided its AST representation is sufficiently similar to CPython's, to allow the macros to work.) -As of this writing (0.14.2), we test on CPython 3.6, and consider it as the primary target platform. However, if anything fails to work on another 3.6-compliant Python 3 such as [PyPy3](https://doc.pypy.org/en/latest/index.html) ([version 2.3.1 or later](http://pypy.org/compat.html)), issue reports and pull requests are welcome. - The library is split into **two layers**, providing **three kinds of features**: - Pure Python (e.g. batteries for `itertools`), From 548a06a688a59ecbbc06da0902ed5ae7c4990bc2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:16:27 +0300 Subject: [PATCH 241/832] update for 0.15 --- doc/design-notes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 28821a5f..e50a14b3 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -36,11 +36,12 @@ The main design considerations of `unpythonic` are simplicity, robustness, and m The whole library is pure Python. No foreign extensions are required. We also try to avoid depending on anything beyond "the Python standard", to help `unpythonic` run on any conforming Python implementation. (Provided its AST representation is sufficiently similar to CPython's, to allow the macros to work.) -The library is split into **two layers**, providing **three kinds of features**: +The library is split into **three layers**, providing **four kinds of features**: - Pure Python (e.g. batteries for `itertools`), - Macros driving a pure-Python core (e.g. `do`, `let`), - Pure macros (e.g. `continuations`, `lazify`, `dbg`). + - Whole-module transformations, a.k.a. dialects. We believe syntactic macros are [*the nuclear option of software engineering*](https://www.factual.com/blog/thinking-in-clojure-for-java-programmers-part-2/). Accordingly, we aim to [minimize macro magic](https://macropy3.readthedocs.io/en/latest/discussion.html#minimize-macro-magic). If a feature can be implemented - *with a level of usability on par with pythonic standards* - without resorting to macros, then it belongs in the pure-Python layer. (The one exception is when building the feature as a macro is the *simpler* solution. Consider `unpythonic.amb.forall` (overly complicated, to avoid macros) vs. `unpythonic.syntax.forall` (a clean macro-based design of the same feature) as an example. Keep in mind [ZoP](https://www.python.org/dev/peps/pep-0020/) §17 and §18.) From 164683bceec5faeabfb2a6829a31bc2d99429078 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:17:41 +0300 Subject: [PATCH 242/832] wording --- doc/design-notes.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index e50a14b3..c9283e7a 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -38,10 +38,13 @@ The whole library is pure Python. No foreign extensions are required. We also tr The library is split into **three layers**, providing **four kinds of features**: - - Pure Python (e.g. batteries for `itertools`), - - Macros driving a pure-Python core (e.g. `do`, `let`), - - Pure macros (e.g. `continuations`, `lazify`, `dbg`). - - Whole-module transformations, a.k.a. dialects. + - `unpythonic` + - Pure Python (e.g. batteries for `itertools`), + - `unpythonic.syntax` + - Macros driving a pure-Python core (e.g. `do`, `let`), + - Pure macros (e.g. `continuations`, `lazify`, `dbg`). + - `unpythonic.dialects` + - Whole-module transformations, a.k.a. dialects. We believe syntactic macros are [*the nuclear option of software engineering*](https://www.factual.com/blog/thinking-in-clojure-for-java-programmers-part-2/). Accordingly, we aim to [minimize macro magic](https://macropy3.readthedocs.io/en/latest/discussion.html#minimize-macro-magic). If a feature can be implemented - *with a level of usability on par with pythonic standards* - without resorting to macros, then it belongs in the pure-Python layer. (The one exception is when building the feature as a macro is the *simpler* solution. Consider `unpythonic.amb.forall` (overly complicated, to avoid macros) vs. `unpythonic.syntax.forall` (a clean macro-based design of the same feature) as an example. Keep in mind [ZoP](https://www.python.org/dev/peps/pep-0020/) §17 and §18.) From 03e35f402065481a905d2b8113dfa02006088a5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:18:42 +0300 Subject: [PATCH 243/832] oops, mention `unpythonic.net`, too --- doc/design-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index c9283e7a..357fc368 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -38,7 +38,7 @@ The whole library is pure Python. No foreign extensions are required. We also tr The library is split into **three layers**, providing **four kinds of features**: - - `unpythonic` + - `unpythonic`, `unpythonic.net` - Pure Python (e.g. batteries for `itertools`), - `unpythonic.syntax` - Macros driving a pure-Python core (e.g. `do`, `let`), From 0359be0d259e495fd2ece1144b0f5406628179d9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 03:20:10 +0300 Subject: [PATCH 244/832] wording --- doc/design-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 357fc368..2790c160 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -86,7 +86,7 @@ In general, I like Python, and my hat's off to the devs. It's no mean feat to cr Of the particular points, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hanging when entering an accidentally infinite mutual recursion. But sometimes, I'd like to enable TCO selectively. If you ask for it, you know what to expect. Well, `unpythonic.syntax` has `with tco` precisely for that. -I think multi-expression `lambda` is, on the surface, a good idea, but really the issue is that the `lambda` construct is broken. We would be much better off if `def` was an expression. Most of the time, anonymous functions aren't such a great idea, but defining closures inline is - sometimes in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. +I think multi-expression `lambda` is, on the surface, a good idea, but really the issue is that the `lambda` construct is broken. We would be much better off if `def` was an expression. Most of the time, anonymous functions aren't such a great idea, but defining closures inline is - and sometimes, the obvious solution is to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline. It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multi-expression lambdas (to define the actions inline), I don't know what is. From ee655fc5b0ca1ea0a2fb77d4e78cade43de645cd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:03:26 +0300 Subject: [PATCH 245/832] oops --- unpythonic/syntax/lambdatools.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index bcb29d6b..09845789 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -275,8 +275,9 @@ def iscallwithnamedargs(tree): # it as `myname` (str); else return `tree` as-is. def nameit(myname, tree): match, thelambda = False, None - # for decorated lambdas, match any chain of one-argument calls. - d = is_decorated_lambda(tree, mode="any") and not has_deco(tree, "namelambda") + # For decorated lambdas, match any chain of one-argument calls. + # The `has_deco` check ignores any already named lambdas. + d = is_decorated_lambda(tree, mode="any") and not has_deco(["namelambda"], tree) c = iscurrywithfinallambda(tree) # this matches only during the second pass (after "with autocurry" has expanded) # so it can't have namelambda already applied From e4e231374920b617212a14e054e9da68414d1a70 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:06:10 +0300 Subject: [PATCH 246/832] namedlambda: auto-name with source location info if no name candidate --- CHANGELOG.md | 1 + unpythonic/syntax/lambdatools.py | 33 +++++++++++++++++++-- unpythonic/syntax/tests/test_lambdatools.py | 8 ++--- 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 63b663a5..6f926f5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Miscellaneous.** - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) + - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) - Python 3.8 and 3.9 support added. diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 09845789..4c6e23a9 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -263,10 +263,13 @@ def iscurrywithfinallambda(tree): currycall_name = "currycall" iscurryf = lambda name: name in ("curryf", "curry") # auto or manual curry in a "with autocurry" def isautocurrywithfinallambda(tree): + # "currycall(..., curryf(lambda ...: ...))" if not (type(tree) is Call and isx(tree.func, currycall_name) and tree.args and type(tree.args[-1]) is Call and isx(tree.args[-1].func, iscurryf)): return False - return type(tree.args[-1].args[-1]) is Lambda + curryf_callnode = tree.args[-1] + lastarg = curryf_callnode.args[-1] + return type(lastarg) is Lambda def iscallwithnamedargs(tree): return type(tree) is Call and tree.keywords @@ -279,11 +282,12 @@ def nameit(myname, tree): # The `has_deco` check ignores any already named lambdas. d = is_decorated_lambda(tree, mode="any") and not has_deco(["namelambda"], tree) c = iscurrywithfinallambda(tree) - # this matches only during the second pass (after "with autocurry" has expanded) + # This matches only during the second pass (after "with autocurry" has expanded) # so it can't have namelambda already applied if isautocurrywithfinallambda(tree): # "currycall(..., curryf(lambda ...: ...))" match = True thelambda = tree.args[-1].args[-1] + # --> "currycall(..., (namelambda(myname))(curryf(lambda ...: ...)))" tree.args[-1].args[-1] = q[h[namelambda](u[myname])(a[thelambda])] elif type(tree) is Lambda or d or c: match = True @@ -375,7 +379,30 @@ def transform(self, tree): newbody = dyn._macro_expander.visit(newbody) # inside out: transform in expanded autocurry - return NamedLambdaTransformer().visit(newbody) + newbody = NamedLambdaTransformer().visit(newbody) + + # v0.15.0+: Finally, auto-name any still anonymous `lambda` with source location info. + # We must perform this in a separate pass so that expanded autocurry invocations + # are transformed correctly first. + class NamedLambdaFinalizationTransformer(ASTTransformer): + def transform(self, tree): + # Recurse into the lambda body in already named lambdas. + if is_decorated_lambda(tree, mode="any") and has_deco(["namelambda"], tree): + decorator_list, thelambda = destructure_decorated_lambda(tree) + thelambda.body = self.visit(thelambda.body) + return tree + elif type(tree) is Lambda: + if hasattr(tree, "lineno"): + thename = f"" + tree, thelambda, match = nameit(thename, tree) + if match: + thelambda.body = self.visit(thelambda.body) + else: + tree = self.visit(tree) + return tree + return self.generic_visit(tree) + return NamedLambdaFinalizationTransformer().visit(newbody) + # The function `f` is adapted from the `f` macro in `macropy.quick_lambda`, # stripped into a bare syntax transformer., and then the `@Walker` inside diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index 0154b163..7349fd36 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -70,12 +70,12 @@ def foo(func1, func2): func2=lambda x: x**2) # function call with named arg: name as "func2" def bar(func1, func2): - test[func1.__name__ == ""] - test[func2.__name__ == ""] - bar(lambda x: x**2, lambda x: x**2) # no naming when passed positionally + test[func1.__name__.startswith(""] + test[func1.__name__.startswith(" Date: Mon, 10 May 2021 20:06:43 +0300 Subject: [PATCH 247/832] improve "what belongs in Python?" --- doc/design-notes.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 2790c160..77272724 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -82,20 +82,27 @@ If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.co Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). -In general, I like Python, and my hat's off to the devs. It's no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the above particular points, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. I think that with macros, Python can be so much more than just a beginner's language, and that language-level extensibility is the logical endpoint of that. +In general, I like Python, and my hat's off to the devs. It's no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the particular points above, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. -Of the particular points, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hanging when entering an accidentally infinite mutual recursion. But sometimes, I'd like to enable TCO selectively. If you ask for it, you know what to expect. Well, `unpythonic.syntax` has `with tco` precisely for that. +I think that with macros, Python can be so much more than just a beginner's language, and that language-level extensibility is the logical endpoint of that. I don't get the sentiment against metaprogramming, or toward some language-level features. For me, macros (and full-module transforms a.k.a. dialects) are just another tool for creating abstractions, at yet another level. We can already extract procedures, methods, and classes. Why limit that ability - namely, the ability to create abstractions - to what an [eager](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) language can express at run time? If the point is to keep code understandable, then it's a matter of education. It's perfectly possible to write unreadable code without macros, and in Python, no less. And it's perfectly possible to write readable code with macros. I'm willing to admit the technical objection that *macros don't compose*; but that doesn't make them useless. -I think multi-expression `lambda` is, on the surface, a good idea, but really the issue is that the `lambda` construct is broken. We would be much better off if `def` was an expression. Most of the time, anonymous functions aren't such a great idea, but defining closures inline is - and sometimes, the obvious solution is to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. +Of the particular points above, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hang, when entering an accidentally infinite mutual recursion. I do occasionally make such mistakes when developing complex algorithms. But sometimes, I'd like to enable TCO selectively. If you ask for it, you know what to expect. This is precisely why `unpythonic.syntax` has `with tco`. I'm not very happy with having a custom TCO layer on top of a language core that doesn't like the idea, because TCO support in the core (like Scheme and Racket have) would simplify the implementation of certain other language extensions; but then again, [this is exactly what Clojure did](https://clojuredocs.org/clojure.core/trampoline), too. -The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline. It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multi-expression lambdas (to define the actions inline), I don't know what is. +I think a multi-expression `lambda` is, on the surface, a good idea, but really the issue is that Python's `lambda` construct itself is broken. It's essentially a duplicate of `def`, but lacking some features. We would be much better off if `def` was an expression. Much of the time, anonymous functions aren't such a great idea, but defining closures inline is - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. (Also, why are lambdas strictly anonymous? In cases where it is useful to be able to omit a name (because sometimes there are many small helpers and [naming is hard](https://martinfowler.com/bliki/TwoHardThings.html)), why not include the source location information in the auto-generated name, instead of just `""`?) -True multi-shot continuations... `unpythonic.syntax` has `with continuations` precisely for that, but I'm not sure if I'll ever use it in production code. However, it's something that's great to have for teaching the concept in a programming course, when teaching in Python. For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. Python's generators) are often all that's needed to simplify certain patterns, especially those involving backtracking. I'm a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! +The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` statement into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline (see [Racket's pattern matcher](https://docs.racket-lang.org/reference/match.html) for a comparison). It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multi-expression lambdas (to define the actions inline), I don't know what is. -On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. +On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the parser mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I don't want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) +As for true multi-shot continuations... `unpythonic.syntax` has `with continuations` for that, but I'm not sure if I'll ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. However, the feature is great to have for teaching the concept of continuations in a programming course, when teaching in Python. For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. Python's generators) are often all that's needed to simplify certain patterns, especially those involving backtracking. I'm a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! + +Finally, how about subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/))? It is pretty much the point of language-level extensibility, to allow users to do that if they want. I wouldn't worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its users, and it is not very popular; it's hard to say what the programming community at large would do with an extensible language. + +What I can say is, `unpythonic` is not meant for the average Python project, either. But if used intelligently, it can make your code shorter, yet readable. Obviously, in a large project with a high developer turnover, the optimal solution looks different. + + ## Killer features of Common Lisp In my opinion, Common Lisp has three legendary killer features: From adec9c55988d12f3ae3071d4fd9d15f3a58cbd95 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:06:53 +0300 Subject: [PATCH 248/832] shorten dialects main level doc --- doc/dialects.md | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/doc/dialects.md b/doc/dialects.md index bb6159ca..90349ec2 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -23,21 +23,6 @@ # Examples of creating dialects using `mcpyrate` -What if Python had automatic tail-call optimization and an implicit return statement? Look no further: - -```python -from unpythonic.dialects import dialects, Lispython # noqa: F401 - -def factorial(n): - def f(k, acc): - if k == 1: - return acc - f(k - 1, k * acc) - f(n, acc=1) -assert factorial(4) == 24 -factorial(5000) # no crash -``` - The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). It provides the plumbing that allows to create, in Python, dialects that compile into Python at macro expansion time. It is geared toward creating languages that extend Python From 499acd99892dc623348927e62d83488aec33938f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:07:12 +0300 Subject: [PATCH 249/832] update explanation of helper macro operators --- doc/macros.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index b2bdbd1f..b67427cc 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -158,6 +158,8 @@ These syntaxes take no macro arguments; both the let-body and the bindings are p Note the bindings subform is always enclosed by brackets. +The `where` operator, if used, must be macro-imported. It may only appear at the top level of the let-where form, separating the body and the bindings subforms. In any invalid position, `where` is considered a syntax error at macro expansion time. +
Semantically, these do the exact same thing as the original lispy syntax: @@ -466,6 +468,8 @@ with let_syntax: After macro expansion completes, ``let_syntax`` has zero runtime overhead; it completely disappears in macro expansion. +The `expr` and `block` operators, if used, must be macro-imported. They may only appear in `with expr` and `with block` subforms at the top level of a `with let_syntax` or `with abbrev`. In any invalid position, `expr` and `block` are both considered a syntax error at macro expansion time. +
There are two kinds of substitutions: @@ -572,6 +576,8 @@ y = do[local[a << 17], Local variables are declared and initialized with ``local[var << value]``, where ``var`` is a bare name. To explicitly denote "no value", just use ``None``. ``delete[...]`` allows deleting a ``local[...]`` binding. This uses ``env.pop()`` internally, so a ``delete[...]`` returns the value the deleted local variable had at the time of deletion. (So if you manually use the ``do()`` function in some code without macros, feel free to ``env.pop()`` in a do-item if needed.) +The `local[]` and `delete[]` declarations may only appear at the top level of a `do[]`, `do0[]`, or implicit `do` (extra bracket syntax, e.g. for the body of a `let` form). In any invalid position, `local[]` and `delete[]` are considered a syntax error at macro expansion time. + A ``local`` declaration comes into effect in the expression following the one where it appears, capturing the declared name as a local variable for the **lexically** remaining part of the ``do``. In a ``local``, the RHS still sees the previous bindings, so this is valid (although maybe not readable): ```python @@ -1222,6 +1228,8 @@ If you need to place ``call_cc[]`` inside a loop, use ``@looped`` et al. from `` Multiple ``call_cc[]`` statements in the same function body are allowed. These essentially create nested closures. +In any invalid position, `call_cc[]` is considered a syntax error at macro expansion time. + **Syntax**: In ``unpythonic``, ``call_cc`` is a **statement**, with the following syntaxes: @@ -1500,6 +1508,8 @@ with prefix: assert (apply, g, "hi", "ho", lst) == (q, "hi" ,"ho", 1, 2, 3) ``` +If you use the `q`, `u` and `kw()` operators, they must be macro-imported. The `q`, `u` and `kw()` operators may only appear in a tuple inside a prefix block. In any invalid position, any of them is considered a syntax error at macro expansion time. + This comboes with ``autocurry`` for an authentic *Listhell* programming experience: ```python @@ -1641,7 +1651,7 @@ aif[2*21, print("it is falsey")] ``` -Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the ``then`` and ``otherwise`` parts of ``aif``, and anywhere else raises a syntax error at macro expansion time. +Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the ``then`` and ``otherwise`` parts of ``aif``, and anywhere else is considered a syntax error at macro expansion time. Any part of ``aif`` may have multiple expressions by surrounding it with brackets (implicit ``do[]``): From 3bf319deaf12fec24c644e1131dc9aca015b0f0d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:07:41 +0300 Subject: [PATCH 250/832] add multi-shot generators example --- unpythonic/syntax/tests/test_conts_gen.py | 57 ++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index 48a804f8..2b432a13 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -18,7 +18,7 @@ https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt """ -from ...syntax import macros, test # noqa: F401 +from ...syntax import macros, test, test_raises # noqa: F401 from ...test.fixtures import session, testset from ...syntax import macros, continuations, call_cc, dlet, abbrev, let_syntax, block # noqa: F401, F811 @@ -178,6 +178,61 @@ def result(loop, i=0): x = g2() # noqa: F821 test[out == list(range(10))] + with testset("multi-shot generators"): + with continuations: + with let_syntax: + with block[value] as my_yield: # noqa: F821 + call_cc[my_yieldf(value)] # noqa: F821 + with block[myname, body] as make_multishot_generator: # noqa: F821 + def myname(k=None): # "myname" is replaced by the user-supplied name + if k: # noqa: F821 + return k() # noqa: F821 + def my_yieldf(value=None, *, cc): + k = cc # noqa: F821 + cc = identity + if value is None: + return k + return k, value + body # noqa: F821 + # If we wanted a mechanism to `return` a final value, + # this would be the place to send it. + raise StopIteration + + # We must define the body as an abbrev block to give it a name, + # because template arguments must be expressions (and a name is, + # but a literal block of code isn't). + # + # This user-defined body gets spliced in after the make_generator + # template itself has expanded. + with block as mybody: + my_yield(1) + my_yield(2) + my_yield(3) + make_multishot_generator(g, mybody) + + # basic test + out = [] + k, x = g() + try: + while True: + out.append(x) + k, x = g(k) + except StopIteration: + pass + test[out == [1, 2, 3]] + + # multi-shot test + k1, x1 = g() # no argument: start from the beginning + k2, x2 = g(k1) # continue execution from k1 (after the first `my_yield`) + k3, x3 = g(k2) + k, x = g(k1) # multi-shot: continue *again* from k1 + test[x1 == 1] + test[x2 == x == 2] + test[x3 == 3] + test[k.__qualname__ == k2.__qualname__] # same bookmarked position... + test[k is not k2] # ...but different function object instance + test_raises[StopIteration, g(k3)] + # Unfortunately, this is as far as let_syntax[] gets us; if we wanted to # "librarify" this any further, we'd need to define a macro in `mcpyrate`. # From 34a8743eb782bd17f45c813fff20b783e7182f14 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:14:21 +0300 Subject: [PATCH 251/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f926f5e..c7c030ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -156,6 +156,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Fix docstring of `test`: multiple `the[]` marks were already supported in 0.14.3, as the macro documentation already said, but the docstring claimed otherwise. +- Fix bug in `with namedlambda`. Due to incorrect function arguments in the analyzer, already named lambdas were not detected correctly. + --- From d063bf9b00c508a157ee06c26ecf541875905f61 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:14:36 +0300 Subject: [PATCH 252/832] document v0.15.0 changes --- doc/features.md | 2 +- doc/macros.md | 10 ++++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/doc/features.md b/doc/features.md index f333a9e4..2c178a01 100644 --- a/doc/features.md +++ b/doc/features.md @@ -81,7 +81,7 @@ For many examples, see [the unit tests](unpythonic/tests/), the docstrings of th *This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out-of-date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* -**This document is up-to-date for v0.14.3.** +**This document is up-to-date for v0.15.0.** ## Bindings diff --git a/doc/macros.md b/doc/macros.md index b67427cc..5e0da68d 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -44,7 +44,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose - [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) [**Language features**](#language-features) -- [``curry``: automatic currying for Python](#curry-automatic-currying-for-python) +- [``autocurry``: automatic currying for Python](#autocurry-automatic-currying-for-python) - [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) - [``lazy[]`` and ``lazyrec[]`` macros](#lazy-and-lazyrec-macros) - [Forcing promises manually](#forcing-promises-manually) @@ -805,7 +805,9 @@ The ``with`` block adds a few elements, but if desired, it can be refactored int To boldly go where Python without macros just won't. Changing the rules by code-walking and making significant rewrites. -### ``curry``: automatic currying for Python +### ``autocurry``: automatic currying for Python + +**Changed in v0.15.0.** *The macro is now named `autocurry`, to avoid shadowing the `curry` function.* ```python from unpythonic.syntax import macros, autocurry @@ -903,6 +905,8 @@ Inspired by Haskell, Racket's ``(delay)`` and ``(force)``, and [lazy/racket](htt #### ``lazy[]`` and ``lazyrec[]`` macros +**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. Note that a lazy value now no longer has a `__call__` operator; instead, it has a `force()` method. The utility `unpythonic.lazyutil.force` (available in the top-level namespace of `unpythonic`) abstracts away this detail.* + We provide the macros ``unpythonic.syntax.lazy``, which explicitly lazifies a single expression, and ``unpythonic.syntax.lazyrec``, which can be used to lazify expressions inside container literals, recursively. Essentially, ``lazy[...]`` achieves the same result as ``memoize(lambda: ...)``, with the practical difference that a ``lazy[]`` promise ``p`` is evaluated by calling ``unpythonic.lazyutil.force(p)`` or ``p.force()``. In ``unpythonic``, the promise datatype (``unpythonic.lazyutil.Lazy``) does not have a ``__call__`` method, because the word ``force`` better conveys the intent. @@ -919,6 +923,8 @@ The `unpythonic` containers **must be from-imported** for ``lazyrec[]`` to recog #### Forcing promises manually +**Changed in v0.15.0.** *The functions `force1` and `force` now live in the top-level namespace of `unpythonic`, no longer in `unpythonic.syntax`.* + This is mainly useful if you ``lazy[]`` or ``lazyrec[]`` something explicitly, and want to compute its value outside a ``with lazify`` block. We provide the functions ``force1`` and ``force``. Using ``force1``, if ``x`` is a ``lazy[]`` promise, it will be forced, and the resulting value is returned. If ``x`` is not a promise, ``x`` itself is returned, à la Racket. The function ``force``, in addition, descends into containers (recursively). When an atom ``x`` (i.e. anything that is not a container) is encountered, it is processed using ``force1``. From bd1a65f07ce2c1f92db027753cbfe9928cfba15b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 10 May 2021 20:14:46 +0300 Subject: [PATCH 253/832] macros.md is now up to date --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 5e0da68d..aa0f9be9 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -22,7 +22,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose **Changed in v0.15.0.** *To run macro-enabled programs, use the [`macropython`](https://github.com/Technologicat/mcpyrate/blob/master/doc/repl.md#macropython-the-universal-bootstrapper) bootstrapper from [`mcpyrate`](https://github.com/Technologicat/mcpyrate).* -**This document is up-to-date for v0.14.3.** +**This document is up-to-date for v0.15.0.** ### Features From 5e7bf7a6df4ed945c2dfd83778ad22238164fa58 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 17:05:33 +0300 Subject: [PATCH 254/832] improve @generic docs --- doc/features.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index 2c178a01..1ed29878 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2998,9 +2998,9 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of **Changed in v0.14.3**. *The `@generic` and `@typed` decorators can now decorate also instance methods, class methods and static methods (beside regular functions, as previously in 0.14.2).* -**Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Added the `@generic_addmethod` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope.* +**Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Added the `@generic_addmethod` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. (Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.)* -The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. +The ``generic`` decorator allows creating multiple-dispatch generic functions (a.k.a. multimethods) with type annotation syntax. We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. @@ -3098,6 +3098,8 @@ When using both `@generic` or `@typed` and OOP: Based on my own initial experiments with this feature, the machinery itself works well enough, but to really shine - just like resumable exceptions - multiple dispatch needs to be used everywhere, throughout the language's ecosystem. Python obviously doesn't do that. +**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@generic_addmethod` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the method table is global state! + #### ``typed``: add run-time type checks with type annotation syntax From 3f00f5d86bf4098d597ff49c42fbe0c9036d2270 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 17:06:31 +0300 Subject: [PATCH 255/832] Split off `excutil` from `misc`; add `reraise`, `resignal` --- unpythonic/__init__.py | 1 + unpythonic/conditions.py | 101 ++++- unpythonic/excutil.py | 423 ++++++++++++++++++ unpythonic/misc.py | 325 +------------- unpythonic/syntax/tests/test_lazify.py | 3 +- .../syntax/tests/testing_testingtools.py | 2 +- unpythonic/test/fixtures.py | 2 +- unpythonic/tests/test_conditions.py | 42 +- unpythonic/tests/test_excutil.py | 161 +++++++ unpythonic/tests/test_misc.py | 126 +----- 10 files changed, 746 insertions(+), 440 deletions(-) create mode 100644 unpythonic/excutil.py create mode 100644 unpythonic/tests/test_excutil.py diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 7ac9df39..8e441fbb 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -17,6 +17,7 @@ from .dispatch import * # noqa: F401, F403 from .dynassign import * # noqa: F401, F403 from .ec import * # noqa: F401, F403 +from .excutil import * # noqa: F401, F403 from .fix import * # noqa: F401, F403 from .fold import * # noqa: F401, F403 from .fploop import * # noqa: F401, F403 diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 1605add7..44701716 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -57,7 +57,8 @@ "available_restarts", "available_handlers", "restarts", "with_restarts", "handlers", - "ControlError"] + "ControlError", + "resignal_in", "resignal"] import threading from collections import deque, namedtuple @@ -68,7 +69,8 @@ from .collections import box, unbox from .arity import arity_includes, UnknownArity -from .misc import namelambda, equip_with_traceback, safeissubclass +from .excutil import equip_with_traceback +from .misc import namelambda, safeissubclass _stacks = threading.local() def _ensure_stacks(): # per-thread init @@ -800,3 +802,98 @@ def __init__(self, value): muffle = invoker("muffle") muffle.__doc__ = "Invoke the 'muffle' restart. Restart function for use with `warn`." + +# Library to application signal type auto-conversion + +def _resignal(mapping, condition): + """Remap an signal instance to another signal type. + + `mapping`: dict-like, `{LibraryExc0: ApplicationExc0, ...}` + + Each `LibraryExc` must be a signal type. + + Each `ApplicationExc` can be a signal type or an instance. + If an instance, then that exact instance is signaled as the + converted signal. + + `libraryexc`: the signal instance to convert. It is + automatically chained into `ApplicationExc`. + + This function never returns normally. If no key in the mapping + matches, this delegates to the next outer handler. + """ + for LibraryExc, ApplicationExc in mapping.items(): + if isinstance(condition, LibraryExc): + # TODO: Would be nice to use the same protocol as the original. + # TODO: For this, we need to store that information in the signal instance. + signal(ApplicationExc, cause=condition) + # cancel and delegate to the next outer handler + +def resignal_in(body, mapping): + """Remap signal types in an expression. + + Like `unpythonic.excutil.reraise_in` (which see), but for conditions. + + Usage:: + + resignal_in(body, + {LibraryExc: ApplicationExc, + ...}) + + Whenever `body` signals an `exc` for which it holds that + `isinstance(exc, LibraryExc)`, that signal will be transparently + chained into an `ApplicationExc` signal. The automatic conversion + is in effect for the dynamic extent of `body`. + + ``body`` is a thunk (0-argument function). + + ``mapping`` is dict-like, ``{input0: output0, ...}``, where each + ``input`` is either an exception type, + or a tuple of exception types. + It will be matched using `isinstance`. + ``output`` is an exception type or an exception + instance. If an instance, then that exact + instance is signaled as the converted + signal. + + Conversions are tried in the order specified; hence, just like in + `with handlers`, place more specific types first. + + See also `resignal` for a block form. + """ + with handlers((BaseException, partial(_resignal, mapping))): + return body() + +@contextlib.contextmanager +def resignal(mapping): + """Remap signal types. Context manager. + + Like `unpythonic.excutil.reraise` (which see), but for conditions. + + Usage:: + + with resignal({LibraryExc: ApplicationExc, ...}): + body0 + ... + + Whenever the body signals an `exc` for which it holds that + `isinstance(exc, LibraryExc)`, that signal will be transparently + chained into an `ApplicationExc` signal. The automatic conversion + is in effect for the dynamic extent of the `with` block. + + ``mapping`` is dict-like, ``{input0: output0, ...}``, where each + ``input`` is either an exception type, + or a tuple of exception types. + It will be matched using `isinstance`. + ``output`` is an exception type or an exception + instance. If an instance, then that exact + instance is signaled as the converted + signal. + + Conversions are tried in the order specified; hence, just like in + `with handlers`, place more specific types first. + + See also `resignal_in` for an expression form. + """ + with handlers((BaseException, partial(_resignal, mapping))): + yield diff --git a/unpythonic/excutil.py b/unpythonic/excutil.py new file mode 100644 index 00000000..370fb5ab --- /dev/null +++ b/unpythonic/excutil.py @@ -0,0 +1,423 @@ +# -*- coding: utf-8 -*- +"""Exception-related utilities.""" + +__all__ = ["raisef", "tryf", + "equip_with_traceback", + "async_raise", + "reraise_in", "reraise"] + +from contextlib import contextmanager +import sys +import threading +from types import TracebackType + +# For async_raise only. Note `ctypes.pythonapi` is not an actual module; +# you'll get a `ModuleNotFoundError` if you try to import it. +# +# TODO: The "pycapi" PyPI package would allow us to regularly import the C API, +# but right now we don't want introduce dependencies, especially for a minor feature. +# https://github.com/brandtbucher/pycapi +if sys.implementation.name == "cpython": + import ctypes + PyThreadState_SetAsyncExc = ctypes.pythonapi.PyThreadState_SetAsyncExc +else: # pragma: no cover, coverage is measured on CPython. + ctypes = None + PyThreadState_SetAsyncExc = None + +from .arity import arity_includes, UnknownArity + + +def raisef(exc, *, cause=None): + """``raise`` as a function, to make it possible for lambdas to raise exceptions. + + Example:: + + raisef(ValueError("message")) + + is (almost) equivalent to:: + + raise ValueError("message") + + Parameters: + exc: exception instance, or exception class + The object to raise. This is whatever you would give as the argument to `raise`. + Both instances (e.g. `ValueError("oof")`) and classes (e.g. `StopIteration`) + can be used as `exc`. + + cause: exception instance, or `None` + If `exc` was triggered as a direct consequence of another exception, + and you would like to `raise ... from ...`, pass that other exception + instance as `cause`. The default `None` performs a plain `raise ...`. + """ + if cause: + raise exc from cause + else: + raise exc + +def tryf(body, *handlers, elsef=None, finallyf=None): + """``try``/``except``/``finally`` as a function. + + This allows lambdas to handle exceptions. + + ``body`` is a thunk (0-argument function) that represents + the body of the ``try`` block. + + ``handlers`` is ``(excspec, handler), ...``, where + ``excspec`` is either an exception type, + or a tuple of exception types. + ``handler`` is a 0-argument or 1-argument + function. If it takes an + argument, it gets the exception + instance. + + Handlers are tried in the order specified. + + ``elsef`` is a thunk that represents the ``else`` block. + + ``finallyf`` is a thunk that represents the ``finally`` block. + + Upon normal completion, the return value of ``tryf`` is + the return value of ``elsef`` if that was specified, otherwise + the return value of ``body``. + + If an exception was caught by one of the handlers, the return + value of ``tryf`` is the return value of the exception handler + that ran. + + If you need to share variables between ``body`` and ``finallyf`` + (which is likely, given what a ``finally`` block is intended + to do), consider wrapping the ``tryf`` in a ``let`` and storing + your variables there. If you want them to leak out of the ``tryf``, + you can also just create an ``env`` at an appropriate point, + and store them there. + """ + def accepts_arg(f): + try: + if arity_includes(f, 1): + return True + except UnknownArity: # pragma: no cover + return True # just assume it + return False + + def isexceptiontype(exc): + try: + if issubclass(exc, BaseException): + return True + except TypeError: # "issubclass() arg 1 must be a class" + pass + return False + + # validate handlers + for excspec, handler in handlers: + if isinstance(excspec, tuple): # tuple of exception types + if not all(isexceptiontype(t) for t in excspec): + raise TypeError(f"All elements of a tuple excspec must be exception types, got {excspec}") + elif not isexceptiontype(excspec): # single exception type + raise TypeError(f"excspec must be an exception type or tuple of exception types, got {excspec}") + + # run + try: + ret = body() + except BaseException as exception: + # Even if a class is raised, as in `raise StopIteration`, the `raise` statement + # converts it into an instance by instantiating with no args. So we need no + # special handling for the "class raised" case. + # https://docs.python.org/3/reference/simple_stmts.html#the-raise-statement + # https://stackoverflow.com/questions/19768515/is-there-a-difference-between-raising-exception-class-and-exception-instance/19768732 + exctype = type(exception) + for excspec, handler in handlers: + if isinstance(excspec, tuple): # tuple of exception types + # this is safe, exctype is always a class at this point. + if any(issubclass(exctype, t) for t in excspec): + if accepts_arg(handler): + return handler(exception) + else: + return handler() + else: # single exception type + if issubclass(exctype, excspec): + if accepts_arg(handler): + return handler(exception) + else: + return handler() + else: + if elsef is not None: + return elsef() + return ret + finally: + if finallyf is not None: + finallyf() + +def equip_with_traceback(exc, stacklevel=1): # Python 3.7+ + """Given an exception instance exc, equip it with a traceback. + + `stacklevel` is the starting depth below the top of the call stack, + to cull useless detail: + - `0` means the trace includes everything, also + `equip_with_traceback` itself, + - `1` means the trace includes everything up to the caller, + - And so on. + + So typically, for direct use of this function `stacklevel` should + be `1` (so it excludes `equip_with_traceback` itself, but shows + all stack levels from your code), and for use in a utility function + that itself is called from your code, it should be `2` (so it excludes + the utility function, too). + + The return value is `exc`, with its traceback set to the produced + traceback. + + Python 3.7 and later only. + + When not supported, raises `NotImplementedError`. + + This is useful mainly in special cases, where `raise` cannot be used for + some reason, and a manually created exception instance needs a traceback. + (The `signal` function in the conditions-and-restarts system uses this.) + + **CAUTION**: The `sys._getframe` function exists in CPython and in PyPy3, + but for another arbitrary Python implementation this is not guaranteed. + + Based on solution by StackOverflow user Zbyl: + https://stackoverflow.com/a/54653137 + + See also: + https://docs.python.org/3/library/types.html#types.TracebackType + https://docs.python.org/3/reference/datamodel.html#traceback-objects + https://docs.python.org/3/library/sys.html#sys._getframe + """ + if not isinstance(exc, BaseException): + raise TypeError(f"exc must be an exception instance; got {type(exc)} with value {repr(exc)}") + if not isinstance(stacklevel, int): + raise TypeError(f"stacklevel must be int, got {type(stacklevel)} with value {repr(stacklevel)}") + if stacklevel < 0: + raise ValueError(f"stacklevel must be >= 0, got {repr(stacklevel)}") + + try: + getframe = sys._getframe + except AttributeError as err: # pragma: no cover, both CPython and PyPy3 have sys._getframe. + raise NotImplementedError("Need a Python interpreter which has `sys._getframe`") from err + + frames = [] + depth = stacklevel + while True: + try: + frames.append(getframe(depth)) # 0 = top of call stack + depth += 1 + except ValueError: # beyond the root level + break + + # Python 3.7+ allows creating `types.TracebackType` objects in Python code. + try: + tracebacks = [] + nxt = None # tb_next should point toward the level where the exception occurred. + for frame in frames: # walk from top of call stack toward the root + tb = TracebackType(nxt, frame, frame.f_lasti, frame.f_lineno) + tracebacks.append(tb) + nxt = tb + if tracebacks: + tb = tracebacks[-1] # root level + else: + tb = None + except TypeError as err: # Python 3.6 or earlier + raise NotImplementedError("Need Python 3.7 or later to create traceback objects") from err + return exc.with_traceback(tb) # Python 3.7+ + +# TODO: To reduce the risk of spaghetti user code, we could require a non-main thread's entrypoint to declare +# via a decorator that it's willing to accept asynchronous exceptions, and check that mark here, making this +# mechanism strictly opt-in. The decorator could inject an `asyncexc_ok` attribute to the Thread object; +# that's enough to prevent accidental misuse. +# OTOH, having no such mechanism is the simpler design. +def async_raise(thread_obj, exception): + """Raise an exception in another thread. + + thread_obj: `threading.Thread` object + The target thread to inject the exception into. Must be running. + exception: ``Exception`` + The exception to be raised. As with regular `raise`, this may be + an exception instance or an exception class object. + + No return value. Normal return indicates success. + + If the specified `threading.Thread` is not active, or the thread's ident + was not accepted by the interpreter, raises `ValueError`. + + If the raise operation failed internally, raises `SystemError`. + + If not supported for the Python implementation we're currently running on, + raises `NotImplementedError`. + + **NOTE**: This currently works only in CPython, because there is no Python-level + API to achieve what this function needs to do, and PyPy3's C API emulation layer + `cpyext` doesn't currently (January 2020) implement the function required to do + this (and the C API functions in `cpyext` are not exposed to the Python level + anyway, unlike CPython's `ctypes.pythonapi`). + + **CAUTION**: This is **potentially dangerous**. If the async raise + operation fails, the interpreter may be left in an inconsistent state. + + **NOTE**: The term `async` here has nothing to do with `async`/`await`; + instead, it refers to an asynchronous exception such as `KeyboardInterrupt`. + https://en.wikipedia.org/wiki/Exception_handling#Exception_synchronicity + + In a nutshell, a *synchronous* exception (i.e. the usual kind of exception) + has an explicit `raise` somewhere in the code that the thread that + encountered the exception is running. In contrast, an *asynchronous* + exception **doesn't**, it just suddenly magically materializes from the outside. + As such, it can in principle happen *anywhere*, with absolutely no hint about + it in any obvious place in the code. + + **Hence, use this function very, very sparingly, if at all.** + + For example, `unpythonic` only uses this to support remotely injecting a + `KeyboardInterrupt` into a REPL session running in another thread. So this + may be interesting mainly if you're developing your own REPL server/client + pair. + + (Incidentally, that's **not** how `KeyboardInterrupt` usually works. + Rather, the OS sends a SIGINT, which is then trapped by an OS signal + handler that runs in the main thread. At that point the magic has already + happened: the control of the main thread is now inside the signal handler, + as if the signal handler was called from the otherwise currently innermost + point on the call stack. All the handler needs to do is to perform a regular + `raise`, and the exception will propagate correctly. + + REPL sessions running in other threads can't use the standard mechanism, + because in CPython, OS signal handlers only run in the main thread, and even + in PyPy3, there is no guarantee *which* thread gets the signal even if you + use `with __pypy__.thread.signals_enabled` to enable OS signal trapping in + some of your other threads. Only one thread (including the main thread, plus + any currently dynamically within a `signals_enabled`) will see the signal; + which one, is essentially random and not even reproducible.) + + See also: + https://vorpus.org/blog/control-c-handling-in-python-and-trio/ + + The function necessary to perform this magic is actually mentioned right + there in the official CPython C API docs, but it's not very well known: + https://docs.python.org/3/c-api/init.html#c.PyThreadState_SetAsyncExc + + Original detective work by Federico Ficarelli and LIU Wei: + https://gist.github.com/nazavode/84d1371e023bccd2301e + https://gist.github.com/liuw/2407154 + """ + if not ctypes or not PyThreadState_SetAsyncExc: + raise NotImplementedError("async_raise not supported on this Python interpreter.") # pragma: no cover + + if not hasattr(thread_obj, "ident"): + raise TypeError(f"Expected a thread object, got {type(thread_obj)} with value '{thread_obj}'") + + target_tid = thread_obj.ident + if target_tid not in {thread.ident for thread in threading.enumerate()}: + raise ValueError("Invalid thread object, cannot find its ident among currently active threads.") + + affected_count = PyThreadState_SetAsyncExc(ctypes.c_long(target_tid), ctypes.py_object(exception)) + if affected_count == 0: + raise ValueError("PyThreadState_SetAsyncExc did not accept the thread ident, even though it was among the currently active threads.") # pragma: no cover + + # TODO: check CPython source code if this case can actually ever happen. + # + # The API docs seem to hint that 0 or 1 are the only possible return values. + # If so, we can remove this `SystemError` case and the "potentially dangerous" caution. + elif affected_count > 1: # pragma: no cover + # Clear the async exception, targeting the same thread identity, and hope for the best. + PyThreadState_SetAsyncExc(ctypes.c_long(target_tid), ctypes.c_long(0)) + raise SystemError("PyThreadState_SetAsyncExc failed, broke the interpreter state.") + +def reraise_in(body, mapping): + """Remap exception types in an expression. + + This allows conveniently converting library exceptions to application + exceptions that are more relevant for the operation being implemented, + at the level of abstraction the operation represents. + + Usage:: + + reraise_in(body, + {LibraryExc: ApplicationExc, + ...}) + + Whenever `body` raises an exception `exc` for which it holds that + `isinstance(exc, LibraryExc)`, that exception will be transparently + chained into an `ApplicationExc`. The automatic conversion is in + effect for the dynamic extent of `body`. + + ``body`` is a thunk (0-argument function). + + ``mapping`` is dict-like, ``{input0: output0, ...}``, where each + ``input`` is either an exception type, + or a tuple of exception types. + It will be matched using `isinstance`. + ``output`` is an exception type or an exception + instance. If an instance, then that exact + instance is raised as the converted + exception. + + Conversions are tried in the order specified; hence, just like in + `except` clauses, place more specific types first. + + See also `reraise` for a block form. + """ + try: + return body() + except BaseException as libraryexc: + _reraise(mapping, libraryexc) + +@contextmanager +def reraise(mapping): + """Remap exception types. Context manager. + + This allows conveniently converting library exceptions to application + exceptions that are more relevant for the operation being implemented, + at the level of abstraction the operation represents. + + Usage:: + + with reraise({LibraryExc: ApplicationExc, ...}): + body0 + ... + + Whenever the body raises an exception `exc` for which it holds that + `isinstance(exc, LibraryExc)`, that exception will be transparently + chained into an `ApplicationExc`. The automatic conversion is in + effect for the dynamic extent of the `with` block. + + ``mapping`` is dict-like, ``{input0: output0, ...}``, where each + ``input`` is either an exception type, + or a tuple of exception types. + It will be matched using `isinstance`. + ``output`` is an exception type or an exception + instance. If an instance, then that exact + instance is raised as the converted + exception. + + Conversions are tried in the order specified; hence, just like in + `except` clauses, place more specific types first. + + See also `reraise_in` for an expression form. + """ + try: + yield + except BaseException as libraryexc: + _reraise(mapping, libraryexc) + +def _reraise(mapping, libraryexc): + """Remap an exception instance to another exception type. + + `mapping`: dict-like, `{LibraryExc0: ApplicationExc0, ...}` + + Each `LibraryExc` must be an exception type. + + Each `ApplicationExc` can be an exception type or an instance. + If an instance, then that exact instance is raised as the + converted exception. + + `libraryexc`: the exception instance to convert. It is + automatically chained into `ApplicationExc`. + + This function never returns normally. If no key in the mapping + matches, the original exception `libraryexc` is re-raised. + """ + for LibraryExc, ApplicationExc in mapping.items(): + if isinstance(libraryexc, LibraryExc): + raise ApplicationExc from libraryexc + raise diff --git a/unpythonic/misc.py b/unpythonic/misc.py index be0e5dbc..991cf3ce 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -1,11 +1,15 @@ # -*- coding: utf-8 -*- """Miscellaneous constructs.""" -__all__ = ["call", "callwith", "raisef", "tryf", "equip_with_traceback", - "pack", "namelambda", "timer", +__all__ = ["call", "callwith", + "pack", + "namelambda", + "timer", "getattrrec", "setattrrec", "Popper", "CountingIterator", - "slurp", "async_raise", "callsite_filename", "safeissubclass"] + "slurp", + "callsite_filename", + "safeissubclass"] from copy import copy from functools import partial @@ -13,27 +17,11 @@ import inspect from queue import Empty from sys import version_info -import threading from time import monotonic -from types import CodeType, FunctionType, LambdaType, TracebackType - -# For async_raise only. Note `ctypes.pythonapi` is not an actual module; -# you'll get a `ModuleNotFoundError` if you try to import it. -# -# TODO: The "pycapi" PyPI package would allow us to regularly import the C API, -# but right now we don't want introduce dependencies, especially for a minor feature. -# https://github.com/brandtbucher/pycapi -import sys -if sys.implementation.name == "cpython": - import ctypes - PyThreadState_SetAsyncExc = ctypes.pythonapi.PyThreadState_SetAsyncExc -else: # pragma: no cover, coverage is measured on CPython. - ctypes = None - PyThreadState_SetAsyncExc = None +from types import CodeType, FunctionType, LambdaType from .regutil import register_decorator from .lazyutil import passthrough_lazy_args, maybe_force_args, force -from .arity import arity_includes, UnknownArity # Only the single-argument form (just f) of the "call" decorator is supported by unpythonic.syntax.util.sort_lambda_decorators. # @@ -212,201 +200,6 @@ def applyfrozenargsto(f): return maybe_force_args(force(f), *args, **kwargs) return applyfrozenargsto -def raisef(exc, *, cause=None): - """``raise`` as a function, to make it possible for lambdas to raise exceptions. - - Example:: - - raisef(ValueError("message")) - - is (almost) equivalent to:: - - raise ValueError("message") - - Parameters: - exc: exception instance, or exception class - The object to raise. This is whatever you would give as the argument to `raise`. - Both instances (e.g. `ValueError("oof")`) and classes (e.g. `StopIteration`) - can be used as `exc`. - - cause: exception instance, or `None` - If `exc` was triggered as a direct consequence of another exception, - and you would like to `raise ... from ...`, pass that other exception - instance as `cause`. The default `None` performs a plain `raise ...`. - """ - if cause: - raise exc from cause - else: - raise exc - -def tryf(body, *handlers, elsef=None, finallyf=None): - """``try``/``except``/``finally`` as a function. - - This allows lambdas to handle exceptions. - - ``body`` is a thunk (0-argument function) that represents - the body of the ``try`` block. - - ``handlers`` is ``(excspec, handler), ...``, where - ``excspec`` is either an exception type, - or a tuple of exception types. - ``handler`` is a 0-argument or 1-argument - function. If it takes an - argument, it gets the exception - instance. - - Handlers are tried in the order specified. - - ``elsef`` is a thunk that represents the ``else`` block. - - ``finallyf`` is a thunk that represents the ``finally`` block. - - Upon normal completion, the return value of ``tryf`` is - the return value of ``elsef`` if that was specified, otherwise - the return value of ``body``. - - If an exception was caught by one of the handlers, the return - value of ``tryf`` is the return value of the exception handler - that ran. - - If you need to share variables between ``body`` and ``finallyf`` - (which is likely, given what a ``finally`` block is intended - to do), consider wrapping the ``tryf`` in a ``let`` and storing - your variables there. If you want them to leak out of the ``tryf``, - you can also just create an ``env`` at an appropriate point, - and store them there. - """ - def accepts_arg(f): - try: - if arity_includes(f, 1): - return True - except UnknownArity: # pragma: no cover - return True # just assume it - return False - - def isexceptiontype(exc): - try: - if issubclass(exc, BaseException): - return True - except TypeError: # "issubclass() arg 1 must be a class" - pass - return False - - # validate handlers - for excspec, handler in handlers: - if isinstance(excspec, tuple): # tuple of exception types - if not all(isexceptiontype(t) for t in excspec): - raise TypeError(f"All elements of a tuple excspec must be exception types, got {excspec}") - elif not isexceptiontype(excspec): # single exception type - raise TypeError(f"excspec must be an exception type or tuple of exception types, got {excspec}") - - # run - try: - ret = body() - except BaseException as exception: - # Even if a class is raised, as in `raise StopIteration`, the `raise` statement - # converts it into an instance by instantiating with no args. So we need no - # special handling for the "class raised" case. - # https://docs.python.org/3/reference/simple_stmts.html#the-raise-statement - # https://stackoverflow.com/questions/19768515/is-there-a-difference-between-raising-exception-class-and-exception-instance/19768732 - exctype = type(exception) - for excspec, handler in handlers: - if isinstance(excspec, tuple): # tuple of exception types - # this is safe, exctype is always a class at this point. - if any(issubclass(exctype, t) for t in excspec): - if accepts_arg(handler): - return handler(exception) - else: - return handler() - else: # single exception type - if issubclass(exctype, excspec): - if accepts_arg(handler): - return handler(exception) - else: - return handler() - else: - if elsef is not None: - return elsef() - return ret - finally: - if finallyf is not None: - finallyf() - -def equip_with_traceback(exc, stacklevel=1): # Python 3.7+ - """Given an exception instance exc, equip it with a traceback. - - `stacklevel` is the starting depth below the top of the call stack, - to cull useless detail: - - `0` means the trace includes everything, also - `equip_with_traceback` itself, - - `1` means the trace includes everything up to the caller, - - And so on. - - So typically, for direct use of this function `stacklevel` should - be `1` (so it excludes `equip_with_traceback` itself, but shows - all stack levels from your code), and for use in a utility function - that itself is called from your code, it should be `2` (so it excludes - the utility function, too). - - The return value is `exc`, with its traceback set to the produced - traceback. - - Python 3.7 and later only. - - When not supported, raises `NotImplementedError`. - - This is useful mainly in special cases, where `raise` cannot be used for - some reason, and a manually created exception instance needs a traceback. - (The `signal` function in the conditions-and-restarts system uses this.) - - **CAUTION**: The `sys._getframe` function exists in CPython and in PyPy3, - but for another arbitrary Python implementation this is not guaranteed. - - Based on solution by StackOverflow user Zbyl: - https://stackoverflow.com/a/54653137 - - See also: - https://docs.python.org/3/library/types.html#types.TracebackType - https://docs.python.org/3/reference/datamodel.html#traceback-objects - https://docs.python.org/3/library/sys.html#sys._getframe - """ - if not isinstance(exc, BaseException): - raise TypeError(f"exc must be an exception instance; got {type(exc)} with value {repr(exc)}") - if not isinstance(stacklevel, int): - raise TypeError(f"stacklevel must be int, got {type(stacklevel)} with value {repr(stacklevel)}") - if stacklevel < 0: - raise ValueError(f"stacklevel must be >= 0, got {repr(stacklevel)}") - - try: - getframe = sys._getframe - except AttributeError as err: # pragma: no cover, both CPython and PyPy3 have sys._getframe. - raise NotImplementedError("Need a Python interpreter which has `sys._getframe`") from err - - frames = [] - depth = stacklevel - while True: - try: - frames.append(getframe(depth)) # 0 = top of call stack - depth += 1 - except ValueError: # beyond the root level - break - - # Python 3.7+ allows creating `types.TracebackType` objects in Python code. - try: - tracebacks = [] - nxt = None # tb_next should point toward the level where the exception occurred. - for frame in frames: # walk from top of call stack toward the root - tb = TracebackType(nxt, frame, frame.f_lasti, frame.f_lineno) - tracebacks.append(tb) - nxt = tb - if tracebacks: - tb = tracebacks[-1] # root level - else: - tb = None - except TypeError as err: # Python 3.6 or earlier - raise NotImplementedError("Need Python 3.7 or later to create traceback objects") from err - return exc.with_traceback(tb) # Python 3.7+ - def pack(*args): """Multi-argument constructor for tuples. @@ -671,108 +464,6 @@ def slurp(queue): pass return out - -# TODO: To reduce the risk of spaghetti user code, we could require a non-main thread's entrypoint to declare -# via a decorator that it's willing to accept asynchronous exceptions, and check that mark here, making this -# mechanism strictly opt-in. The decorator could inject an `asyncexc_ok` attribute to the Thread object; -# that's enough to prevent accidental misuse. -# OTOH, having no such mechanism is the simpler design. -def async_raise(thread_obj, exception): - """Raise an exception in another thread. - - thread_obj: `threading.Thread` object - The target thread to inject the exception into. Must be running. - exception: ``Exception`` - The exception to be raised. As with regular `raise`, this may be - an exception instance or an exception class object. - - No return value. Normal return indicates success. - - If the specified `threading.Thread` is not active, or the thread's ident - was not accepted by the interpreter, raises `ValueError`. - - If the raise operation failed internally, raises `SystemError`. - - If not supported for the Python implementation we're currently running on, - raises `NotImplementedError`. - - **NOTE**: This currently works only in CPython, because there is no Python-level - API to achieve what this function needs to do, and PyPy3's C API emulation layer - `cpyext` doesn't currently (January 2020) implement the function required to do - this (and the C API functions in `cpyext` are not exposed to the Python level - anyway, unlike CPython's `ctypes.pythonapi`). - - **CAUTION**: This is **potentially dangerous**. If the async raise - operation fails, the interpreter may be left in an inconsistent state. - - **NOTE**: The term `async` here has nothing to do with `async`/`await`; - instead, it refers to an asynchronous exception such as `KeyboardInterrupt`. - https://en.wikipedia.org/wiki/Exception_handling#Exception_synchronicity - - In a nutshell, a *synchronous* exception (i.e. the usual kind of exception) - has an explicit `raise` somewhere in the code that the thread that - encountered the exception is running. In contrast, an *asynchronous* - exception **doesn't**, it just suddenly magically materializes from the outside. - As such, it can in principle happen *anywhere*, with absolutely no hint about - it in any obvious place in the code. - - **Hence, use this function very, very sparingly, if at all.** - - For example, `unpythonic` only uses this to support remotely injecting a - `KeyboardInterrupt` into a REPL session running in another thread. So this - may be interesting mainly if you're developing your own REPL server/client - pair. - - (Incidentally, that's **not** how `KeyboardInterrupt` usually works. - Rather, the OS sends a SIGINT, which is then trapped by an OS signal - handler that runs in the main thread. At that point the magic has already - happened: the control of the main thread is now inside the signal handler, - as if the signal handler was called from the otherwise currently innermost - point on the call stack. All the handler needs to do is to perform a regular - `raise`, and the exception will propagate correctly. - - REPL sessions running in other threads can't use the standard mechanism, - because in CPython, OS signal handlers only run in the main thread, and even - in PyPy3, there is no guarantee *which* thread gets the signal even if you - use `with __pypy__.thread.signals_enabled` to enable OS signal trapping in - some of your other threads. Only one thread (including the main thread, plus - any currently dynamically within a `signals_enabled`) will see the signal; - which one, is essentially random and not even reproducible.) - - See also: - https://vorpus.org/blog/control-c-handling-in-python-and-trio/ - - The function necessary to perform this magic is actually mentioned right - there in the official CPython C API docs, but it's not very well known: - https://docs.python.org/3/c-api/init.html#c.PyThreadState_SetAsyncExc - - Original detective work by Federico Ficarelli and LIU Wei: - https://gist.github.com/nazavode/84d1371e023bccd2301e - https://gist.github.com/liuw/2407154 - """ - if not ctypes or not PyThreadState_SetAsyncExc: - raise NotImplementedError("async_raise not supported on this Python interpreter.") # pragma: no cover - - if not hasattr(thread_obj, "ident"): - raise TypeError(f"Expected a thread object, got {type(thread_obj)} with value '{thread_obj}'") - - target_tid = thread_obj.ident - if target_tid not in {thread.ident for thread in threading.enumerate()}: - raise ValueError("Invalid thread object, cannot find its ident among currently active threads.") - - affected_count = PyThreadState_SetAsyncExc(ctypes.c_long(target_tid), ctypes.py_object(exception)) - if affected_count == 0: - raise ValueError("PyThreadState_SetAsyncExc did not accept the thread ident, even though it was among the currently active threads.") # pragma: no cover - - # TODO: check CPython source code if this case can actually ever happen. - # - # The API docs seem to hint that 0 or 1 are the only possible return values. - # If so, we can remove this `SystemError` case and the "potentially dangerous" caution. - elif affected_count > 1: # pragma: no cover - # Clear the async exception, targeting the same thread identity, and hope for the best. - PyThreadState_SetAsyncExc(ctypes.c_long(target_tid), ctypes.c_long(0)) - raise SystemError("PyThreadState_SetAsyncExc failed, broke the interpreter state.") - def callsite_filename(): """Return the filename of the call site, as a string. diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 6ab71ff9..a2e5b674 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -14,11 +14,12 @@ # and this one is a regular run-time function. from ...collections import frozendict from ...ec import call_ec +from ...excutil import raisef from ...fun import (curry, memoize, flip, rotate, apply, notf, andf, orf, tokth, withself) from ...it import flatten from ...llist import ll -from ...misc import raisef, call, callwith +from ...misc import call, callwith from ...tco import trampolined, jump from ...lazyutil import islazy, Lazy, force1, force # Lazy usually not needed in client code; for our tests only diff --git a/unpythonic/syntax/tests/testing_testingtools.py b/unpythonic/syntax/tests/testing_testingtools.py index 1187c560..78c799f7 100644 --- a/unpythonic/syntax/tests/testing_testingtools.py +++ b/unpythonic/syntax/tests/testing_testingtools.py @@ -23,7 +23,7 @@ TestFailure, TestError) from ...conditions import invoke, handlers, restarts, cerror # noqa: F401 -from ...misc import raisef +from ...excutil import raisef def runtests(): # Low-level machinery. diff --git a/unpythonic/test/fixtures.py b/unpythonic/test/fixtures.py index 486e8d66..76ea87c8 100644 --- a/unpythonic/test/fixtures.py +++ b/unpythonic/test/fixtures.py @@ -47,7 +47,7 @@ test[2 + 2 == 5] # Testsets can be named. The name is printed in the output. - from unpythonic.misc import raisef + from unpythonic.excutil import raisef from unpythonic.conditions import cerror with testset("my fancy tests"): test[2 + 2 == 4] diff --git a/unpythonic/tests/test_conditions.py b/unpythonic/tests/test_conditions.py index e543dded..ae9d64d0 100644 --- a/unpythonic/tests/test_conditions.py +++ b/unpythonic/tests/test_conditions.py @@ -22,8 +22,10 @@ available_restarts, available_handlers, error, cerror, proceed, warn, muffle, - ControlError) -from ..misc import raisef, slurp + ControlError, + resignal_in, resignal) +from ..excutil import raisef +from ..misc import slurp from ..collections import box, unbox from ..it import subset @@ -576,6 +578,42 @@ def worker(comm, tid): test[the[tuple(sorted(tag for tag, x in results)) == tuple(range(n))]] # de-spam: don't capture LHS multithreading() + with testset("resignal_in, resignal"): + def resignal_tests(): + class LibraryException(Exception): + pass + class MoreSophisticatedLibraryException(LibraryException): + pass + class UnrelatedException(Exception): + pass + class ApplicationException(Exception): + pass + test_signals[ApplicationException, resignal_in(lambda: signal(LibraryException), + {LibraryException: ApplicationException})] + # subclasses + test_signals[ApplicationException, resignal_in(lambda: signal(MoreSophisticatedLibraryException), + {LibraryException: ApplicationException})] + # tuple of types as input + test_signals[ApplicationException, resignal_in(lambda: signal(UnrelatedException), + {(LibraryException, UnrelatedException): + ApplicationException})] + test[returns_normally(resignal_in(lambda: 42, + {LibraryException: ApplicationException}))] + + with test_signals[ApplicationException]: + with resignal({LibraryException: ApplicationException}): + signal(LibraryException) + with test_signals[ApplicationException]: + with resignal({LibraryException: ApplicationException}): + signal(MoreSophisticatedLibraryException) + with test_signals[ApplicationException]: + with resignal({(LibraryException, UnrelatedException): ApplicationException}): + signal(LibraryException) + with test["should return normally"]: + with resignal({LibraryException: ApplicationException}): + 42 + resignal_tests() + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/tests/test_excutil.py b/unpythonic/tests/test_excutil.py new file mode 100644 index 00000000..858122cf --- /dev/null +++ b/unpythonic/tests/test_excutil.py @@ -0,0 +1,161 @@ +# -*- coding: utf-8 -*- + +from ..syntax import macros, test, test_raises, error, warn, the # noqa: F401 +from ..test.fixtures import session, testset, returns_normally + +import threading +from time import sleep +import sys + +from ..excutil import (raisef, tryf, + equip_with_traceback, + reraise_in, reraise, + async_raise) +from ..env import env + +def runtests(): + # raisef: raise an exception from an expression position + with testset("raisef (raise exception from an expression)"): + raise_instance = lambda: raisef(ValueError("all ok")) # the argument works the same as in `raise ...` + test_raises[ValueError, raise_instance()] + try: + raise_instance() + except ValueError as err: + test[err.__cause__ is None] # like plain `raise ...`, no cause set (default behavior) + + # using the `cause` parameter, raisef can also perform a `raise ... from ...` + exc = TypeError("oof") + raise_instance = lambda: raisef(ValueError("all ok"), cause=exc) + test_raises[ValueError, raise_instance()] + try: + raise_instance() + except ValueError as err: + test[err.__cause__ is exc] # cause specified, like `raise ... from ...` + + # can also raise an exception class (no instance) + test_raises[StopIteration, raisef(StopIteration)] + + # tryf: handle an exception in an expression position + with testset("tryf (try/except/finally in an expression)"): + raise_instance = lambda: raisef(ValueError("all ok")) + raise_class = lambda: raisef(ValueError) + + test[tryf(lambda: "hello") == "hello"] + test[tryf(lambda: "hello", + elsef=lambda: "there") == "there"] + test[tryf(lambda: raise_instance(), + (ValueError, lambda: "got a ValueError")) == "got a ValueError"] + test[tryf(lambda: raise_instance(), + (ValueError, lambda err: f"got a ValueError: '{err.args[0]}'")) == "got a ValueError: 'all ok'"] + test[tryf(lambda: raise_instance(), + ((RuntimeError, ValueError), lambda err: f"got a RuntimeError or ValueError: '{err.args[0]}'")) == "got a RuntimeError or ValueError: 'all ok'"] + test[tryf(lambda: "hello", + (ValueError, lambda: "got a ValueError"), + elsef=lambda: "there") == "there"] + test[tryf(lambda: raisef(ValueError("oof")), + (TypeError, lambda: "got a TypeError"), + ((TypeError, ValueError), lambda: "got a TypeError or a ValueError"), + (ValueError, lambda: "got a ValueError")) == "got a TypeError or a ValueError"] + + e = env(finally_ran=False) + test[e.finally_ran is False] + test[tryf(lambda: "hello", + elsef=lambda: "there", + finallyf=lambda: e << ("finally_ran", True)) == "there"] + test[e.finally_ran is True] + + test[tryf(lambda: raise_class(), + (ValueError, lambda: "ok")) == "ok"] + test[tryf(lambda: raise_class(), + ((RuntimeError, ValueError), lambda: "ok")) == "ok"] + + test_raises[TypeError, tryf(lambda: "hello", + (str, lambda: "got a string"))] # str is not an exception type + test_raises[TypeError, tryf(lambda: "hello", + ((ValueError, str), lambda: "got a string"))] # same, in the tuple case + test_raises[TypeError, tryf(lambda: "hello", + ("not a type at all!", lambda: "got a string"))] + + with testset("equip_with_traceback"): + e = Exception("just testing") + try: + e = equip_with_traceback(e) + except NotImplementedError: + warn["equip_with_traceback only supported on Python 3.7+, skipping test."] + else: + # Can't do meaningful testing on the result, so just check it's there. + test[e.__traceback__ is not None] + + test_raises[TypeError, equip_with_traceback("not an exception")] + + with testset("reraise_in, reraise"): + class LibraryException(Exception): + pass + class MoreSophisticatedLibraryException(LibraryException): + pass + class UnrelatedException(Exception): + pass + class ApplicationException(Exception): + pass + + test_raises[ApplicationException, reraise_in(lambda: raisef(LibraryException), + {LibraryException: ApplicationException})] + # subclasses + test_raises[ApplicationException, reraise_in(lambda: raisef(MoreSophisticatedLibraryException), + {LibraryException: ApplicationException})] + # tuple of types as input + test_raises[ApplicationException, reraise_in(lambda: raisef(UnrelatedException), + {(LibraryException, UnrelatedException): + ApplicationException})] + test[returns_normally(reraise_in(lambda: 42, + {LibraryException: ApplicationException}))] + + with test_raises[ApplicationException]: + with reraise({LibraryException: ApplicationException}): + raise LibraryException + with test_raises[ApplicationException]: + with reraise({LibraryException: ApplicationException}): + raise MoreSophisticatedLibraryException + with test_raises[ApplicationException]: + with reraise({(LibraryException, UnrelatedException): ApplicationException}): + raise LibraryException + with test["should return normally"]: + with reraise({LibraryException: ApplicationException}): + 42 + + # async_raise - evil ctypes hack to inject an asynchronous exception into another running thread + if sys.implementation.name != "cpython": + warn["async_raise only supported on CPython, skipping test."] # pragma: no cover + else: + with testset("async_raise (inject KeyboardInterrupt)"): + try: + # Test whether the Python we're running on provides ctypes. At least CPython and PyPy3 do. + # For PyPy3, the general rule is "if it imports, it should work", so let's go along with that. + import ctypes # noqa: F401 + out = [] # box, really, but let's not depend on unpythonic.collections in this unrelated unit test module + def test_async_raise_worker(): + try: + for j in range(10): + sleep(0.1) + except KeyboardInterrupt: # normally, KeyboardInterrupt is only raised in the main thread + pass + out.append(j) + t = threading.Thread(target=test_async_raise_worker) + t.start() + sleep(0.1) # make sure we're in the while loop + async_raise(t, KeyboardInterrupt) + t.join() + test[out[0] < 9] # terminated early due to the injected KeyboardInterrupt + except NotImplementedError: # pragma: no cover + error["async_raise not supported on this Python interpreter."] + + test_raises[TypeError, async_raise(42, KeyboardInterrupt)] # not a thread + + t = threading.Thread(target=lambda: None) + t.start() + t.join() + test_raises[ValueError, async_raise(t, KeyboardInterrupt)] # thread no longer running + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() diff --git a/unpythonic/tests/test_misc.py b/unpythonic/tests/test_misc.py index 75057a35..9b70bd39 100644 --- a/unpythonic/tests/test_misc.py +++ b/unpythonic/tests/test_misc.py @@ -7,16 +7,17 @@ from functools import partial from collections import deque from queue import Queue -from time import sleep -import threading -import sys - -from ..misc import (call, callwith, raisef, tryf, equip_with_traceback, - pack, namelambda, timer, - getattrrec, setattrrec, Popper, CountingIterator, slurp, - async_raise, callsite_filename, safeissubclass) + +from ..misc import (call, callwith, + pack, + namelambda, + timer, + getattrrec, setattrrec, + Popper, CountingIterator, + slurp, + callsite_filename, + safeissubclass) from ..fun import withself -from ..env import env def runtests(): with testset("@call (def as code block)"): @@ -99,80 +100,6 @@ def mul3(a, b, c): m = (f(3) for f in [lambda x: 2 * x, lambda x: x**2, lambda x: x**(1 / 2)]) test[tuple(m) == (6, 9, 3**(1 / 2))] - # raisef: raise an exception from an expression position - with testset("raisef (raise exception from an expression)"): - raise_instance = lambda: raisef(ValueError("all ok")) # the argument works the same as in `raise ...` - test_raises[ValueError, raise_instance()] - try: - raise_instance() - except ValueError as err: - test[err.__cause__ is None] # like plain `raise ...`, no cause set (default behavior) - - # using the `cause` parameter, raisef can also perform a `raise ... from ...` - exc = TypeError("oof") - raise_instance = lambda: raisef(ValueError("all ok"), cause=exc) - test_raises[ValueError, raise_instance()] - try: - raise_instance() - except ValueError as err: - test[err.__cause__ is exc] # cause specified, like `raise ... from ...` - - # can also raise an exception class (no instance) - test_raises[StopIteration, raisef(StopIteration)] - - # tryf: handle an exception in an expression position - with testset("tryf (try/except/finally in an expression)"): - raise_instance = lambda: raisef(ValueError("all ok")) - raise_class = lambda: raisef(ValueError) - - test[tryf(lambda: "hello") == "hello"] - test[tryf(lambda: "hello", - elsef=lambda: "there") == "there"] - test[tryf(lambda: raise_instance(), - (ValueError, lambda: "got a ValueError")) == "got a ValueError"] - test[tryf(lambda: raise_instance(), - (ValueError, lambda err: f"got a ValueError: '{err.args[0]}'")) == "got a ValueError: 'all ok'"] - test[tryf(lambda: raise_instance(), - ((RuntimeError, ValueError), lambda err: f"got a RuntimeError or ValueError: '{err.args[0]}'")) == "got a RuntimeError or ValueError: 'all ok'"] - test[tryf(lambda: "hello", - (ValueError, lambda: "got a ValueError"), - elsef=lambda: "there") == "there"] - test[tryf(lambda: raisef(ValueError("oof")), - (TypeError, lambda: "got a TypeError"), - ((TypeError, ValueError), lambda: "got a TypeError or a ValueError"), - (ValueError, lambda: "got a ValueError")) == "got a TypeError or a ValueError"] - - e = env(finally_ran=False) - test[e.finally_ran is False] - test[tryf(lambda: "hello", - elsef=lambda: "there", - finallyf=lambda: e << ("finally_ran", True)) == "there"] - test[e.finally_ran is True] - - test[tryf(lambda: raise_class(), - (ValueError, lambda: "ok")) == "ok"] - test[tryf(lambda: raise_class(), - ((RuntimeError, ValueError), lambda: "ok")) == "ok"] - - test_raises[TypeError, tryf(lambda: "hello", - (str, lambda: "got a string"))] # str is not an exception type - test_raises[TypeError, tryf(lambda: "hello", - ((ValueError, str), lambda: "got a string"))] # same, in the tuple case - test_raises[TypeError, tryf(lambda: "hello", - ("not a type at all!", lambda: "got a string"))] - - with testset("equip_with_traceback"): - e = Exception("just testing") - try: - e = equip_with_traceback(e) - except NotImplementedError: - warn["equip_with_traceback only supported on Python 3.7+, skipping test."] - else: - # Can't do meaningful testing on the result, so just check it's there. - test[e.__traceback__ is not None] - - test_raises[TypeError, equip_with_traceback("not an exception")] - with testset("pack"): myzip = lambda lol: map(pack, *lol) lol = ((1, 2), (3, 4), (5, 6)) @@ -271,39 +198,6 @@ def __init__(self, x): q.put(k) test[slurp(q) == list(range(10))] - # async_raise - evil ctypes hack to inject an asynchronous exception into another running thread - if sys.implementation.name != "cpython": - warn["async_raise only supported on CPython, skipping test."] # pragma: no cover - else: - with testset("async_raise (inject KeyboardInterrupt)"): - try: - # Test whether the Python we're running on provides ctypes. At least CPython and PyPy3 do. - # For PyPy3, the general rule is "if it imports, it should work", so let's go along with that. - import ctypes # noqa: F401 - out = [] # box, really, but let's not depend on unpythonic.collections in this unrelated unit test module - def test_async_raise_worker(): - try: - for j in range(10): - sleep(0.1) - except KeyboardInterrupt: # normally, KeyboardInterrupt is only raised in the main thread - pass - out.append(j) - t = threading.Thread(target=test_async_raise_worker) - t.start() - sleep(0.1) # make sure we're in the while loop - async_raise(t, KeyboardInterrupt) - t.join() - test[out[0] < 9] # terminated early due to the injected KeyboardInterrupt - except NotImplementedError: # pragma: no cover - error["async_raise not supported on this Python interpreter."] - - test_raises[TypeError, async_raise(42, KeyboardInterrupt)] # not a thread - - t = threading.Thread(target=lambda: None) - t.start() - t.join() - test_raises[ValueError, async_raise(t, KeyboardInterrupt)] # thread no longer running - with testset("callsite_filename"): test["test_misc.py" in the[callsite_filename()]] From e2f74cd00ad6bcf7d400986091db282033e71914 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 17:07:04 +0300 Subject: [PATCH 256/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c7c030ce..9e968331 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,6 +80,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. + - Add `unpythonic.excutil.remap_in`, `unpythonic.excutil.remap`, and the condition variants `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`: conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) - Python 3.8 and 3.9 support added. @@ -143,6 +144,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Miscellaneous.** - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + - The functions `raisef`, `tryf`, `equip_with_traceback`, and `async_raise` now live in `unpythonic.excutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - Remove the internal utility class `unpythonic.syntax.util.ASTMarker`. We now have `mcpyrate.markers.ASTMarker`, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time. - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From 7cff7a09e92333cb28aa2a6201556c195a2dd26a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 17:07:12 +0300 Subject: [PATCH 257/832] consistency: "canonize", not "normalize" --- unpythonic/syntax/letdoutil.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index b784d6ad..892d8ddc 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -30,7 +30,7 @@ def _set_subscript_slice(tree, newslice): # newslice: AST if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. tree.slice = newslice tree.slice.value = newslice -def _normalize_macroargs_node(macroargs): +def _canonize_macroargs_node(macroargs): # We do this like `mcpyrate.expander.destructure_candidate` does, # except that we also destructure a list. if type(macroargs) in (List, Tuple): # [a0, a1, ...] @@ -222,7 +222,7 @@ def maybeiscontentofletwhere(tree): try: # This could be a `let_syntax` or `abbrev` using the haskelly let-in syntax. # We don't want to care about that, so we always use `letsyntax_mode=True`. - _ = canonize_bindings(_normalize_macroargs_node(bindings), letsyntax_mode=True) + _ = canonize_bindings(_canonize_macroargs_node(bindings), letsyntax_mode=True) return "in_expr" except SyntaxError: pass @@ -449,17 +449,17 @@ def _getbindings(self): return canonize_bindings(thetree.args) # Subscript theargs = _get_subscript_slice(thetree) - return canonize_bindings(_normalize_macroargs_node(theargs)) + return canonize_bindings(_canonize_macroargs_node(theargs)) else: # haskelly let, `let[[...] in ...]`, `let[..., where[...]]` theexpr = self._theexpr_ref() # `[...] in ...`, `..., where[...]` if t == "in_expr": - return canonize_bindings(_normalize_macroargs_node(theexpr.left)) + return canonize_bindings(_canonize_macroargs_node(theexpr.left)) elif t == "where_expr": thewhere = theexpr.elts[1] if type(thewhere) is Call: return canonize_bindings(thewhere.args) else: # Subscript - return canonize_bindings(_normalize_macroargs_node(_get_subscript_slice(thewhere))) + return canonize_bindings(_canonize_macroargs_node(_get_subscript_slice(thewhere))) assert False def _setbindings(self, newbindings): t = self._type From 20418daa31033462a0226c4bbe1e9024042321b6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 18:15:02 +0300 Subject: [PATCH 258/832] improve resignal, signal - Resignal using the same protocol as the original signal. - The protocol information is now stored in the `__protocol__` attribute of the condition instance. It is automatically populated by the standard error-handling protocols. (If not given, it defaults to `signal` itself.) - `signal`, when the signal is not handled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. --- unpythonic/conditions.py | 110 ++++++++++++++++++++++++++++----------- unpythonic/excutil.py | 6 +-- 2 files changed, 82 insertions(+), 34 deletions(-) diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 44701716..7077fea1 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -88,7 +88,7 @@ class ControlError(Exception): when no handler handles the signal. """ -def signal(condition, *, cause=None): +def signal(condition, *, cause=None, protocol=None): """Signal a condition. Signaling a condition works similarly to raising an exception (pass an @@ -115,7 +115,18 @@ def signal(condition, *, cause=None): perform the restart and continue. If none of the matching handlers invokes a restart, `signal` returns - normally. There is no meaningful return value, it is always `None`. + normally. + + For most use cases, the return value is not needed. But for defining + custom error-handling protocols on top of `signal`, it can be very useful. + + The return value is the input `condition`, canonized to an instance + (even if originally, an exception *type* was passed to `signal`), + with its `__cause__` and `__protocol__` attributes filled in, + and with a traceback attached (on Python 3.7+). For example, the + `error` protocol uses the return value to chain the unhandled signal + properly into a `ControlError` exception; as a result, the error report + looks like a standard exception chain, with nice-looking tracebacks. If you want to error out on unhandled conditions, see `error`, which is otherwise the same as `signal`, except it raises if `signal` would have @@ -126,6 +137,16 @@ def signal(condition, *, cause=None): keyword, it essentially performs a `signal ... from ...`. The default `cause=None` performs a plain `signal ...`. + The optional `protocol` argument is a low-level detail, meant for use by + error-handling protocols (including custom ones). + + The `protocol` is stored into the `__protocol__` attribute of the condition + instance. It is the callable that was used to perform the signaling. If not + given, it defaults to the `signal` function itself. The main use case is for + `resignal` (for signal type conversion); using this information, it can + automatically emit the new signal using the same protocol that was used + for the original signal (so that e.g. an `error` remains an `error`). + **Notes** This condition system is implemented on top of exceptions. The magic trick @@ -151,6 +172,16 @@ def signal(condition, *, cause=None): # The unwinding, when it occurs, is performed when `invoke` is # called from inside the condition handler in the user code. + # stacklevel: in the traceback, omit equip_with_traceback(), _prepare_signal_instance(), and signal(). + # + # We can't have signal() there, because it would look like the call to `_prepare_signal_instance` + # was the cause of the signal, which is nonsense. + # + # Nicely, the resulting stack trace happens to be similar to how Python handles `raise` - the use site + # of `raise` (of an uncaught exception) is shown, but the internals of `raise` are not. + protocol = protocol or signal + condition = _prepare_signal_instance(condition, cause=cause, protocol=protocol, stacklevel=3) + def accepts_arg(f): try: if arity_includes(f, 1): @@ -159,6 +190,18 @@ def accepts_arg(f): return True # just assume it return False + for handler in _find_handlers(type(condition)): + if accepts_arg(handler): + handler(condition) + else: + handler() + + # When unhandled, return the condition instance. + # `error()` uses this return value; this allows us to provide a unified format for tracebacks. + return condition + +def _prepare_signal_instance(condition, *, cause, protocol, stacklevel): + """Canonize a condition, and populate its technical data.""" # Consistency with behavior of exceptions in Python: # Even if a class is raised, as in `raise StopIteration`, the `raise` statement # converts it into an instance by instantiating with no args. So we need no @@ -180,20 +223,16 @@ def canonize(exc, err_reason): condition = canonize(condition, "be signaled") cause = canonize(cause, "act as the cause of another signal") condition.__cause__ = cause + condition.__protocol__ = protocol # Embed a stack trace in the signal, like Python does for raised exceptions. # This only works on Python 3.7 and later, because we need to create a traceback object in pure Python code. try: - # In the result, omit equip_with_traceback() and signal(). - condition = equip_with_traceback(condition, stacklevel=2) + condition = equip_with_traceback(condition, stacklevel=stacklevel) except NotImplementedError: # pragma: no cover pass # well, we tried! - for handler in _find_handlers(type(condition)): - if accepts_arg(handler): - handler(condition) - else: - handler() + return condition def invoke(name_or_restart, *args, **kwargs): """Invoke a restart currently in scope. Known as `INVOKE-RESTART` in Common Lisp. @@ -695,15 +734,7 @@ def error(condition, *, cause=None): keyword, it essentially performs a `error ... from ...`. The default `cause=None` performs a plain `error ...`. """ - signal(condition, cause=cause) - # TODO: If we want to support the debugger at some point in the future, - # TODO: this is the appropriate point to ask the user what to do, - # TODO: before the call stack unwinds. - # - # TODO: Do we want to give one last chance to handle the ControlError? - # TODO: And do we want to raise ControlError, or the original condition? - condition.__cause__ = cause # chain the causes, since we'll add a new one next. - raise ControlError("Unhandled error condition") from condition + _error(condition, cause=cause, protocol=error) def cerror(condition, *, cause=None): """Like `error`, but allow a handler to instruct the caller to ignore the error. @@ -739,7 +770,21 @@ def __init__(self, value): """ with restarts(proceed=(lambda: None)): # just for control, no return value - error(condition, cause=cause) + _error(condition, cause=cause, protocol=cerror) + +def _error(condition, *, cause, protocol): + # The return value is canonized to an instance (even if `condition` was an exception *type*), + # and importantly, it has a nice-looking traceback that points to this line here. + # If the signal goes unhandled, Python's exception system will want to show that traceback + # when our `ControlError` *exception* goes uncaught. + condition = signal(condition, cause=cause, protocol=protocol) + # TODO: If we want to support the debugger at some point in the future, + # TODO: this is the appropriate point to ask the user what to do, + # TODO: before the call stack unwinds. + # + # TODO: Do we want to give one last chance to handle the ControlError? + # TODO: And do we want to raise ControlError, or the original condition? + raise ControlError("Unhandled error condition") from condition def warn(condition, *, cause=None): """Like `signal`, but emit a warning if the condition is not handled. @@ -789,7 +834,7 @@ def __init__(self, value): """ with restarts(muffle=(lambda: None)): # just for control, no return value with restarts(_proceed=(lambda: None)): # for internal use by unpythonic.test.fixtures - signal(condition, cause=cause) + signal(condition, cause=cause, protocol=warn) if isinstance(condition, Warning): warnings.warn(condition, stacklevel=2) # 2 to ignore our lispy `warn` wrapper. else: @@ -805,16 +850,16 @@ def __init__(self, value): # Library to application signal type auto-conversion -def _resignal(mapping, condition): - """Remap an signal instance to another signal type. +def _resignal_handler(mapping, condition): + """Remap a condition instance to another condition type. `mapping`: dict-like, `{LibraryExc0: ApplicationExc0, ...}` Each `LibraryExc` must be a signal type. - Each `ApplicationExc` can be a signal type or an instance. + Each `ApplicationExc` can be a condition type or an instance. If an instance, then that exact instance is signaled as the - converted signal. + converted condition. `libraryexc`: the signal instance to convert. It is automatically chained into `ApplicationExc`. @@ -824,13 +869,16 @@ def _resignal(mapping, condition): """ for LibraryExc, ApplicationExc in mapping.items(): if isinstance(condition, LibraryExc): - # TODO: Would be nice to use the same protocol as the original. - # TODO: For this, we need to store that information in the signal instance. - signal(ApplicationExc, cause=condition) + # Resignal using the same error-handling protocol as the original signal + # (so that e.g. an `error(...)` resignals into an `error(...)` of the new type). + if not hasattr(condition, "__protocol__"): + error(f"Cannot resignal: protocol information missing in condition instance {condition}") + resignaler = condition.__protocol__ + resignaler(ApplicationExc, cause=condition) # cancel and delegate to the next outer handler def resignal_in(body, mapping): - """Remap signal types in an expression. + """Remap condition types in an expression. Like `unpythonic.excutil.reraise_in` (which see), but for conditions. @@ -861,12 +909,12 @@ def resignal_in(body, mapping): See also `resignal` for a block form. """ - with handlers((BaseException, partial(_resignal, mapping))): + with handlers((BaseException, partial(_resignal_handler, mapping))): return body() @contextlib.contextmanager def resignal(mapping): - """Remap signal types. Context manager. + """Remap condition types. Context manager. Like `unpythonic.excutil.reraise` (which see), but for conditions. @@ -895,5 +943,5 @@ def resignal(mapping): See also `resignal_in` for an expression form. """ - with handlers((BaseException, partial(_resignal, mapping))): + with handlers((BaseException, partial(_resignal_handler, mapping))): yield diff --git a/unpythonic/excutil.py b/unpythonic/excutil.py index 370fb5ab..a09c1ec6 100644 --- a/unpythonic/excutil.py +++ b/unpythonic/excutil.py @@ -360,7 +360,7 @@ def reraise_in(body, mapping): try: return body() except BaseException as libraryexc: - _reraise(mapping, libraryexc) + _reraise_handler(mapping, libraryexc) @contextmanager def reraise(mapping): @@ -398,9 +398,9 @@ def reraise(mapping): try: yield except BaseException as libraryexc: - _reraise(mapping, libraryexc) + _reraise_handler(mapping, libraryexc) -def _reraise(mapping, libraryexc): +def _reraise_handler(mapping, libraryexc): """Remap an exception instance to another exception type. `mapping`: dict-like, `{LibraryExc0: ApplicationExc0, ...}` From b3d3c6941414085d5c269fb111ecedb1cb39d11e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 18:24:52 +0300 Subject: [PATCH 259/832] update changelog --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e968331..e9e48d98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,7 +80,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. - - Add `unpythonic.excutil.remap_in`, `unpythonic.excutil.remap`, and the condition variants `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`: conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). + - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). + - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) - Python 3.8 and 3.9 support added. @@ -97,6 +98,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. - **Miscellaneous.** + - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). - CI: Test coverage improved to 94%. From d1c114efce178aa925fd0357dc94952f7add00dc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 18:56:02 +0300 Subject: [PATCH 260/832] document excutil separately; document reraise, reraise_in --- doc/features.md | 291 +++++++++++++++++++++++++++++++----------------- 1 file changed, 190 insertions(+), 101 deletions(-) diff --git a/doc/features.md b/doc/features.md index 1ed29878..8bfeccde 100644 --- a/doc/features.md +++ b/doc/features.md @@ -61,11 +61,15 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``handlers``, ``restarts``: conditions and restarts](#handlers-restarts-conditions-and-restarts), a.k.a. **resumable exceptions**. - [``generic``, ``typed``, ``isoftype``: multiple dispatch](#generic-typed-isoftype-multiple-dispatch): create generic functions with type annotation syntax; also some friendly utilities. +[**Exception tools**](#exception-tools) +- [``raisef``, ``tryf``: ``raise`` and ``try`` as functions](#raisef-tryf-raise-and-try-as-functions), useful inside a lambda. +- [``equip_with_traceback``](#equip-with-traceback), equip a manually created exception instance with a traceback. +- [``async_raise``: inject an exception to another thread](#async_raise-inject-an-exception-to-another-thread) *(CPython only)* +- [`reraise_in`, `reraise`: automatically convert exception types](#reraise_in-reraise-automatically-convert-exception-types) + [**Other**](#other) - [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. - [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) -- [``raisef``, ``tryf``: ``raise`` and ``try`` as functions](#raisef-tryf-raise-and-try-as-functions), useful inside a lambda. -- [``equip_with_traceback``](#equip-with-traceback), equip a manually created exception instance with a traceback. - [``callsite_filename``](#callsite-filename) - [``safeissubclass``](#safeissubclass), convenience function. - [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) @@ -75,7 +79,6 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``arities``, ``kwargs``, ``resolve_bindings``: Function signature inspection utilities](#arities-kwargs-resolve_bindings-function-signature-inspection-utilities) - [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) - [``ulp``: unit in last place](#ulp-unit-in-last-place) -- [``async_raise``: inject an exception to another thread](#async_raise-inject-an-exception-to-another-thread) *(CPython only)* For many examples, see [the unit tests](unpythonic/tests/), the docstrings of the individual features, and this guide. @@ -2860,6 +2863,12 @@ The implementation is based on the List monad, and a bastardized variant of do-n *Signaling a class, as in `signal(SomeExceptionClass)`, now implicitly creates an instance with no arguments, just like the `raise` statement does. On Python 3.7+, `signal` now automatically equips the condition instance with a traceback, just like the `raise` statement does for an exception.* +**Changed in v0.15.0.** *Functions `resignal_in` and `resignal` added; these perform the same job for conditions as `reraise_in` and `reraise` do for exceptions, that is, they allow you to map library exception types to semantically appropriate application exception types, with minimum boilerplate.* + +*Upon an unhandled signal, `signal` now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report.* + +*The error-handling protocol that was used to send a signal is now available for inspection in the `__protocol__` attribute of the condition instance. It is the callable that sent the signal, such as `signal`, `error`, `cerror` or `warn`. It is the responsibility of each error-handling protocol (except the fundamental `signal` itself) to pass its own function to `signal` as the `protocol` argument; if not given, `protocol` defaults to `signal`. The protocol information is used by the `resignal` mechanism.* + One of the killer features of Common Lisp are *conditions*, which are essentially **resumable exceptions**. Following Peter Seibel ([Practical Common Lisp, chapter 19](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), we define *errors* as the consequences of [Murphy's Law](https://en.wikipedia.org/wiki/Murphy%27s_law), i.e. situations where circumstances cause interaction between the program and the outside world to fail. An error is no bug, but failing to handle an error certainly is. @@ -3205,6 +3214,184 @@ See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. If you need a run-time type checker for serious general use, consider the [`typeguard`](https://github.com/agronholm/typeguard) library, which focuses on that. +## Exception tools + +Utilities for dealing with exceptions. + +### ``raisef``, ``tryf``: ``raise`` and ``try`` as functions + +**Changed in v0.14.3**. *Now we have also `tryf`.* + +**Changed in v0.14.2**. *The parameters of `raisef` now more closely match what would be passed to `raise`. See examples below. Old-style parameters are now deprecated, and support for them will be dropped in v0.15.0.* + +Raise an exception from an expression position: + +```python +from unpythonic import raisef + +# plain `raise ...` +f = lambda x: raisef(RuntimeError("I'm in ur lambda raising exceptions")) + +# `raise ... from ...` +exc = TypeError("oof") +g = lambda x: raisef(RuntimeError("I'm in ur lambda raising exceptions"), cause=exc) +``` + +Catch an exception in an expression position: + +```python +from unpythonic import raisef, tryf + +raise_instance = lambda: raisef(ValueError("all ok")) +test[tryf(lambda: raise_instance(), + (ValueError, lambda err: f"got a ValueError: '{err.args[0]}'")) == "got a ValueError: 'all ok'"] +``` + +The exception handler is a function. It may optionally accept one argument, the exception instance. + +Functions can also be specified for the `else` and `finally` behavior; see the docstring of `unpythonic.misc.tryf` for details. + + +### ``equip_with_traceback`` + +**Added in v0.14.3**. + +In Python 3.7 and later, equip a manually created exception instance with a traceback. This is useful mainly in special cases, where `raise` cannot be used for some reason. (The `signal` function in the conditions-and-restarts system uses this.) + +```python +e = SomeException(...) +e = equip_with_traceback(e) +``` + +The traceback is automatically extracted from the call stack of the calling thread. + +Optionally, you can cull a number of the topmost frames by passing the optional argument `stacklevel=...`. Typically, for direct use of this function `stacklevel` should be the default `1` (so it excludes `equip_with_traceback` itself, but shows all stack levels from your code), and for use in a utility function that itself is called from your code, it should be `2` (so it excludes the utility function, too). + + +### ``async_raise``: inject an exception to another thread + +**Added in v0.14.2**. + +*Currently CPython only, because as of this writing (March 2020) PyPy3 does not expose the required functionality to the Python level, nor there seem to be any plans to do so.* + +Usually injecting an exception into an unsuspecting thread makes absolutely no sense. But there are special cases, such as a REPL server which needs to send a `KeyboardInterrupt` into a REPL session thread that's happily stuck waiting for input at [`InteractiveConsole.interact()`](https://docs.python.org/3/library/code.html#code.InteractiveConsole.interact) - while the client that receives the actual `Ctrl+C` is running in a separate process. This and similar awkward situations in network programming are pretty much the only legitimate use case for this feature. + +The name is `async_raise`, because it injects an *asynchronous exception*. This has nothing to do with `async`/`await`. Synchronous vs. asynchronous exceptions [mean something different](https://en.wikipedia.org/wiki/Exception_handling#Exception_synchronicity). + +In a nutshell, a *synchronous* exception (which is the usual kind of exception) has an explicit `raise` somewhere in the code that the thread that encountered the exception is running. In contrast, an *asynchronous* exception **doesn't**, it just suddenly magically materializes from the outside. As such, it can in principle happen *anywhere*, with absolutely no hint about it in any obvious place in the code. + +Needless to say this can be very confusing, so this feature should be used sparingly, if at all. **We only have it because the REPL server needs it.** + +```python +from unpythonic import async_raise, box + +out = box() +def worker(): + try: + for j in range(10): + sleep(0.1) + except KeyboardInterrupt: # normally, KeyboardInterrupt is only raised in the main thread + pass + out << j +t = threading.Thread(target=worker) +t.start() +sleep(0.1) # make sure the worker has entered the loop +async_raise(t, KeyboardInterrupt) +t.join() +assert unbox(out) < 9 # thread terminated early due to the injected KeyboardInterrupt +``` + +#### So this is how KeyboardInterrupt works under the hood? + +No, this is **not** how `KeyboardInterrupt` usually works. Rather, the OS sends a [SIGINT](https://en.wikipedia.org/wiki/Signal_(IPC)#SIGINT), which is then trapped by an [OS signal handler](https://docs.python.org/3/library/signal.html) that runs in the main thread. + +(Note OS signal, in the *nix sense; this is unrelated to the Lisp sense, as in conditions-and-restarts.) + +At that point the magic has already happened: the control of the main thread is now inside the signal handler, as if the signal handler was called from the otherwise currently innermost point on the call stack. All the handler needs to do is to perform a regular `raise`, and the exception will propagate correctly. + +#### History + +Original detective work by [Federico Ficarelli](https://gist.github.com/nazavode/84d1371e023bccd2301e) and [LIU Wei](https://gist.github.com/liuw/2407154). + +Raising async exceptions is a [documented feature of Python's public C API](https://docs.python.org/3/c-api/init.html#c.PyThreadState_SetAsyncExc), but it was never meant to be invoked from within pure Python code. But then the CPython devs gave us [ctypes.pythonapi](https://docs.python.org/3/library/ctypes.html#accessing-values-exported-from-dlls), which allows access to Python's C API from within Python. (If you think ctypes.pythonapi is too quirky, the [pycapi](https://pypi.org/project/pycapi/) PyPI package smooths over the rough edges.) Combining the two gives `async_raise` without the need to compile a C extension. + +Unfortunately PyPy doesn't currently (March 2020) implement this function in its CPython C API emulation layer, `cpyext`. See `unpythonic` issue [#58](https://github.com/Technologicat/unpythonic/issues/58). + + +### `reraise_in`, `reraise`: automatically convert exception types + +**Added in v0.15.0.** + +Sometimes it is useful to semantically convert exception types from one problem domain to another, particularly across the different levels of abstraction in an application. We provide `reraise_in` and `reraise` to do this with minimum boilerplate: + +```python +from unpythonic import reraise_in, reraise, raisef + +class LibraryException(Exception): + pass +class MoreSophisticatedLibraryException(LibraryException): + pass + +class UnrelatedException(Exception): + pass + +class ApplicationException(Exception): + pass + +# reraise_in: expr form +try: + # reraise_in(thunk, mapping) + reraise_in(lambda: raisef(LibraryException), + {LibraryException: ApplicationException}) +except ApplicationException: # note the type! + print("all ok!") + +try: + # subclasses are converted, too + reraise_in(lambda: raisef(MoreSophisticatedLibraryException), + {LibraryException: ApplicationException}) +except ApplicationException: + print("all ok!") + +try: + # tuples of types are accepted, like in `except` clauses + reraise_in(lambda: raisef(UnrelatedException), + {(LibraryException, UnrelatedException): + ApplicationException}) +except ApplicationException: + print("all ok!") + +# reraise: block form +try: + with reraise({LibraryException: ApplicationException}): + raise LibraryException +except ApplicationException: + print("all ok!") + +try: + with reraise({LibraryException: ApplicationException}): + raise MoreSophisticatedLibraryException +except ApplicationException: + print("all ok!") + +try: + with reraise({(LibraryException, UnrelatedException): + ApplicationException}): + raise LibraryException +except ApplicationException: + print("all ok!") + +``` + +If that's not much shorter than the hand-written `try`/`except`/`raise from`, consider that you can create the mapping once and then use it from a variable - this shortens it to just `with reraise(my_mapping)`. + +Any exceptions that don't match anything in the mapping are passed through. When no exception occurs, `reraise_in` passes the return value of `thunk` through, and `reraise` does nothing. + +Full details in docstrings. + +If you use the conditions-and-restarts system, see also `resignal_in`, `resignal`, which perform the same job for conditions. The new signal is sent using the same error handling protocol as the original signal, so e.g. an `error` will remain an `error` even if re-signaling changes its type. + + ## Other Stuff that didn't fit elsewhere. @@ -3398,56 +3585,6 @@ assert tuple(m) == (6, 9, 3**(1/2)) Inspired by *Function application with $* in [LYAH: Higher Order Functions](http://learnyouahaskell.com/higher-order-functions). -### ``raisef``, ``tryf``: ``raise`` and ``try`` as functions - -**Changed in v0.14.3**. *Now we have also `tryf`.* - -**Changed in v0.14.2**. *The parameters of `raisef` now more closely match what would be passed to `raise`. See examples below. Old-style parameters are now deprecated, and support for them will be dropped in v0.15.0.* - -Raise an exception from an expression position: - -```python -from unpythonic import raisef - -# plain `raise ...` -f = lambda x: raisef(RuntimeError("I'm in ur lambda raising exceptions")) - -# `raise ... from ...` -exc = TypeError("oof") -g = lambda x: raisef(RuntimeError("I'm in ur lambda raising exceptions"), cause=exc) -``` - -Catch an exception in an expression position: - -```python -from unpythonic import raisef, tryf - -raise_instance = lambda: raisef(ValueError("all ok")) -test[tryf(lambda: raise_instance(), - (ValueError, lambda err: f"got a ValueError: '{err.args[0]}'")) == "got a ValueError: 'all ok'"] -``` - -The exception handler is a function. It may optionally accept one argument, the exception instance. - -Functions can also be specified for the `else` and `finally` behavior; see the docstring of `unpythonic.misc.tryf` for details. - - -### ``equip_with_traceback`` - -**Added in v0.14.3**. - -In Python 3.7 and later, equip a manually created exception instance with a traceback. This is useful mainly in special cases, where `raise` cannot be used for some reason. (The `signal` function in the conditions-and-restarts system uses this.) - -```python -e = SomeException(...) -e = equip_with_traceback(e) -``` - -The traceback is automatically extracted from the call stack of the calling thread. - -Optionally, you can cull a number of the topmost frames by passing the optional argument `stacklevel=...`. Typically, for direct use of this function `stacklevel` should be the default `1` (so it excludes `equip_with_traceback` itself, but shows all stack levels from your code), and for use in a utility function that itself is called from your code, it should be `2` (so it excludes the utility function, too). - - ### ``callsite_filename`` **Added in v0.14.3**. @@ -3718,51 +3855,3 @@ print(ulp(2**52)) When `x` is a round number in base-10, the ULP is not, because the usual kind of floats use base-2. For more reading, see [David Goldberg (1991): What every computer scientist should know about floating-point arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), or for a [tl;dr](http://catplanet.org/tldr-cat-meme/) version, [the floating point guide](https://floating-point-gui.de/). - - -### ``async_raise``: inject an exception to another thread - -**Added in v0.14.2**. - -*Currently CPython only, because as of this writing (March 2020) PyPy3 does not expose the required functionality to the Python level, nor there seem to be any plans to do so.* - -Usually injecting an exception into an unsuspecting thread makes absolutely no sense. But there are special cases, such as a REPL server which needs to send a `KeyboardInterrupt` into a REPL session thread that's happily stuck waiting for input at [`InteractiveConsole.interact()`](https://docs.python.org/3/library/code.html#code.InteractiveConsole.interact) - while the client that receives the actual `Ctrl+C` is running in a separate process. This and similar awkward situations in network programming are pretty much the only legitimate use case for this feature. - -The name is `async_raise`, because it injects an *asynchronous exception*. This has nothing to do with `async`/`await`. Synchronous vs. asynchronous exceptions [mean something different](https://en.wikipedia.org/wiki/Exception_handling#Exception_synchronicity). - -In a nutshell, a *synchronous* exception (which is the usual kind of exception) has an explicit `raise` somewhere in the code that the thread that encountered the exception is running. In contrast, an *asynchronous* exception **doesn't**, it just suddenly magically materializes from the outside. As such, it can in principle happen *anywhere*, with absolutely no hint about it in any obvious place in the code. - -Needless to say this can be very confusing, so this feature should be used sparingly, if at all. **We only have it because the REPL server needs it.** - -```python -from unpythonic import async_raise, box - -out = box() -def worker(): - try: - for j in range(10): - sleep(0.1) - except KeyboardInterrupt: # normally, KeyboardInterrupt is only raised in the main thread - pass - out << j -t = threading.Thread(target=worker) -t.start() -sleep(0.1) # make sure the worker has entered the loop -async_raise(t, KeyboardInterrupt) -t.join() -assert unbox(out) < 9 # thread terminated early due to the injected KeyboardInterrupt -``` - -#### So this is how KeyboardInterrupt works under the hood? - -No, this is **not** how `KeyboardInterrupt` usually works. Rather, the OS sends a [SIGINT](https://en.wikipedia.org/wiki/Signal_(IPC)#SIGINT), which is then trapped by an [OS signal handler](https://docs.python.org/3/library/signal.html) that runs in the main thread. - -At that point the magic has already happened: the control of the main thread is now inside the signal handler, as if the signal handler was called from the otherwise currently innermost point on the call stack. All the handler needs to do is to perform a regular `raise`, and the exception will propagate correctly. - -#### History - -Original detective work by [Federico Ficarelli](https://gist.github.com/nazavode/84d1371e023bccd2301e) and [LIU Wei](https://gist.github.com/liuw/2407154). - -Raising async exceptions is a [documented feature of Python's public C API](https://docs.python.org/3/c-api/init.html#c.PyThreadState_SetAsyncExc), but it was never meant to be invoked from within pure Python code. But then the CPython devs gave us [ctypes.pythonapi](https://docs.python.org/3/library/ctypes.html#accessing-values-exported-from-dlls), which allows access to Python's C API from within Python. (If you think ctypes.pythonapi is too quirky, the [pycapi](https://pypi.org/project/pycapi/) PyPI package smooths over the rough edges.) Combining the two gives `async_raise` without the need to compile a C extension. - -Unfortunately PyPy doesn't currently (March 2020) implement this function in its CPython C API emulation layer, `cpyext`. See `unpythonic` issue [#58](https://github.com/Technologicat/unpythonic/issues/58). From 8a908802543c4459307d870fc543acfdef2d4664 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 19:31:45 +0300 Subject: [PATCH 261/832] fix borked example in docstring --- unpythonic/conditions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 7077fea1..080497ae 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -760,13 +760,13 @@ class OddNumberError(Exception): def __init__(self, value): self.value = value - with handlers=((OddNumberError, proceed)): + with handlers((OddNumberError, proceed)): out = [] for x in range(10): if x % 2 == 1: cerror(OddNumberError(x)) # if unhandled, raises ControlError out.append(x) - assert out == [0, 2, 4, 6, 8] + assert out == list(range(10)) """ with restarts(proceed=(lambda: None)): # just for control, no return value From 0fe3d357b60b75aff14962dd06e5bab372286c4d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 20:03:54 +0300 Subject: [PATCH 262/832] improve docstrings --- unpythonic/conditions.py | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 080497ae..56907434 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -282,23 +282,23 @@ def invoke(name_or_restart, *args, **kwargs): A handler that just invokes the `use_value` restart is such a common use case that it is useful to have an abbreviation for it. This:: - with handlers((OhNoes, lambda c: invoke("use_value", 42))): + with handlers((OhNoes, lambda: invoke("use_value", 42))): ... can be abbreviated to:: - with handlers((OhNoes, lambda c: use_value(42))): + with handlers((OhNoes, lambda: use_value(42))): ... -The `lambda c:` is still required, for consistency with Common Lisp, as well as -to allow the user code to access the condition instance if needed. - -(A common use case is to embed, in the condition instance, the data needed +A common use case is to embed, in the condition instance, the data needed for constructing the actual value to be sent to the `use_value` restart. In Seibel's log file parser example, when the parser sees a corrupt log entry, it embeds that data into the condition instance, and sends it to the handler, which then can in principle repair the log entry, and then invoke `use_value` -with the repaired log entry.) +with the repaired log entry. Then you can do something like:: + + with handlers((OhNoes, lambda c: use_value(produce_fixed_entry_from(c)))): + ... **Notes**: @@ -311,26 +311,27 @@ def invoke(name_or_restart, *args, **kwargs): This pattern can be useful for defining similar shorthands for your own restarts. -(Note that restarts are looked up by name, so a single module-level definition +Note that restarts are looked up by name, so a single module-level definition of a shorthand for each uniquely named restart is enough. You can re-use the same shorthand for any restart that has the same name - just like there is just one `use_value` function, even though the `use_value` restart itself is defined separately at each `with restarts` site that provides it (since only -each site itself knows how to "use a value").) +each site itself knows how to "use a value"). If you want a version for use cases where the condition instance argument is not needed, so you could in those cases omit the `lambda c:`, you can write that as:: - use_constant = partial(invoker, "use_value") - with handlers((OhNoes, use_constant(42))): + make_use_constant = partial(invoker, "use_value") + use_42 = make_use_constant(42) + with handlers((OhNoes, use_42)): ... Note `invoker`, not `invoke`, and we are still left with a factory (since `invoker` itself is a factory and `partial` defers the call until it gets more arguments). You then call the factory function with your desired constant args/kwargs, to instantiate a handler that sends that specific -set of args/kwargs. +set of constant args/kwargs. """ def invoker(restart_name, *args, **kwargs): @@ -390,8 +391,21 @@ def invoker(restart_name, *args, **kwargs): with handlers((OhNoes, lambda c: use_value(42))): ... # calling some code that may cerror(OhNoes("ouch")) - (The `use_value` function is convenient especially when the value being sent - is not a constant, but depends on data in the condition instance `c`.) + The `use_value` function is convenient especially when the value being sent + is not a constant, but depends on data in the condition instance `c`. + To do the same for your own restart, use this pattern (see `invoke`):: + + frobnicate = partial(invoke, "frobnicate") + with handlers((OhNoes, frobnicate)): + ... + + In this case, the `frobnicate` restart - if it accepts one positional + argument - will receive the condition instance. To send something else, + you can also do something like this:: + + frobnicate = partial(invoke, "frobnicate") + with handlers((OhNoes, lambda c: frobnicate(c.args[0] * 42))): + ... **Notes** From bc504200b556b6411067761abfbb57f52459688a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 11 May 2021 20:03:59 +0300 Subject: [PATCH 263/832] fix outdated comments --- unpythonic/tests/test_conditions.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/unpythonic/tests/test_conditions.py b/unpythonic/tests/test_conditions.py index ae9d64d0..e41bf1b4 100644 --- a/unpythonic/tests/test_conditions.py +++ b/unpythonic/tests/test_conditions.py @@ -68,7 +68,7 @@ def lowlevel(): return out # High-level logic. Choose here which action the low-level logic should take - # for each named signal. Here we only have one signal, named "odd_number". + # for each condition type. Here we only have one signal, `OddNumberError`. def highlevel(): # When using error() or cerror() to signal, not handling the condition # is a fatal error (like an uncaught exception). The `error` function @@ -278,8 +278,11 @@ def test_usevalue(): fail["This line should not be reached in the tests."] # pragma: no cover test[unbox(result) == 42] - # can be shortened using the predefined `use_value` function, which immediately + # This can be shortened using the predefined `use_value` function, which immediately # invokes the eponymous restart with the args and kwargs given. + # + # If you need to do the same for your own restart, use `functools.partial(invoke, restart_name)`. + # That will give you a function that you can use in a handler, and pass in args at that time. with handlers((JustTesting, lambda c: use_value(42))): with restarts(use_value=(lambda x: x)) as result: signal(JustTesting()) From a80647b776dca5e40228895423082b7fd88be87b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 12 May 2021 13:07:59 +0300 Subject: [PATCH 264/832] update macro docs --- doc/macros.md | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index aa0f9be9..263f58b6 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -905,7 +905,7 @@ Inspired by Haskell, Racket's ``(delay)`` and ``(force)``, and [lazy/racket](htt #### ``lazy[]`` and ``lazyrec[]`` macros -**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. Note that a lazy value now no longer has a `__call__` operator; instead, it has a `force()` method. The utility `unpythonic.lazyutil.force` (available in the top-level namespace of `unpythonic`) abstracts away this detail.* +**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. Note that a lazy value now no longer has a `__call__` operator; instead, it has a `force()` method. The utility `unpythonic.lazyutil.force` (previously exported in `unpythonic.syntax`; now moved to the top-level namespace of `unpythonic`) abstracts away this detail.* We provide the macros ``unpythonic.syntax.lazy``, which explicitly lazifies a single expression, and ``unpythonic.syntax.lazyrec``, which can be used to lazify expressions inside container literals, recursively. @@ -933,7 +933,7 @@ Mutable containers are updated in-place; for immutables, a new instance is creat #### Binding constructs and auto-lazification -Why do we auto-lazify in certain kinds of binding constructs, but not in others? Function calls and let-bindings have one feature in common: both are guaranteed to bind only new names. Auto-lazification of all assignments, on the other hand, in a language that allows mutation is dangerous, because then this superficially innocuous code will fail: +Why do we auto-lazify in certain kinds of binding constructs, but not in others? Function calls and let-bindings have one feature in common: both are guaranteed to bind only new names (even if that name is already in scope, they are distinct; the new binding will shadow the old one). Auto-lazification of all assignments, on the other hand, in a language that allows mutation is dangerous, because then this superficially innocuous code will fail: ```python a = 10 @@ -1103,7 +1103,7 @@ For various possible program topologies that continuations may introduce, see [t For full documentation, see the docstring of ``unpythonic.syntax.continuations``. The unit tests [[1]](../unpythonic/syntax/test/test_conts.py) [[2]](../unpythonic/syntax/test/test_conts_escape.py) [[3]](../unpythonic/syntax/test/test_conts_gen.py) [[4]](../unpythonic/syntax/test/test_conts_topo.py) may also be useful as usage examples. -**Note on debugging**: If a function containing a ``call_cc[]`` crashes below the ``call_cc[]``, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so you can pinpoint the source code line where the error occurred. (For a function ``f``, it is named ``f_cont``, ``f_cont1``, ...) But be aware that especially in complex macro combos (e.g. ``continuations, curry, lazify``), the other block macros may spit out many internal function calls *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. +**Note on debugging**: If a function containing a ``call_cc[]`` crashes below the ``call_cc[]``, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so you can pinpoint the source code line where the error occurred. (For a function ``f``, it is named ``f_cont_``) But be aware that especially in complex macro combos (e.g. ``continuations, curry, lazify``), the other block macros may spit out many internal function calls *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. **Note on exceptions**: Raising an exception, or [signaling and restarting](features.md#handlers-restarts-conditions-and-restarts), will partly unwind the call stack, so the continuation *from the level that raised the exception* will be cancelled. This is arguably exactly the expected behavior. @@ -2147,31 +2147,51 @@ The block macros are designed to run **in the following order (leftmost first)** ``` prefix > autoreturn, quicklambda > multilambda > continuations or tco > ... - ... > curry > namedlambda, autoref > lazify > envify + ... > autocurry > namedlambda, autoref > lazify > envify ``` The ``let_syntax`` (and ``abbrev``) block may be placed anywhere in the chain; just keep in mind what it does. -The ``dbg`` block can be run at any position after ``prefix`` and before ``tco`` (or ``continuations``). (It must be able to see regular function calls.) +The ``dbg`` block can be run at any position after ``prefix`` and before ``tco`` (or ``continuations``). (It must be able to see function calls in Python's standard format, for detecting calls to the print function.) For simplicity, **the block macros make no attempt to prevent invalid combos** (unless there is a specific technical reason to do that for some particular combination). Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. +The correct ordering for block macro invocations is somewhat complicated by the fact that some of the above are two-pass macros. Consider this artificial example, where `mac` is a two-pass macro: + +```python +with mac: + with cheese: + ... +``` + +The invocation `with mac` is *lexically on the outside*, thus the macro expander sees it first. The expansion order is then: + + 1. First pass (outside in) of `with mac`. + 2. Explicit recursion by `with mac`. This expands the `with cheese`. + 3. Second pass (inside out) of `with mac`. + +So for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. + Example combo in the single-line format: ```python -with autoreturn, tco, lazify: +with autoreturn, lazify, tco: ... ``` In the multiline format: ```python -with lazify: - with tco: - with autoreturn: +with autoreturn: + with lazify: + with tco: ... ``` +Of these, `autoreturn` expands outside-in, while `lazify` and `tco` are both two-pass macros. + +To see if something is a two-pass macro, for now, grep the codebase for `expander.visit`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). + See our [notes on macros](../doc/design-notes.md#detailed-notes-on-macros) for more information. ### Emacs syntax highlighting From a8bad1e9102fc270b4bce5965af8213591db1a4a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 12 May 2021 13:09:35 +0300 Subject: [PATCH 265/832] add note how to make a lazy identity function --- unpythonic/syntax/lazify.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index e89665ea..7d577a76 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -381,6 +381,17 @@ def f(a, b): - The default continuation ``identity`` is strict, so that return values from a continuation-enabled computation will be forced. + If you need a lazy ``identity`` (so that you can obtain those delicious + promises), use:: + + from unpythonic import identity + from unpythonic.lazyutil import passthrough_lazy_args + lazy_identity = passthrough_lazy_args(identity) + + and then explicitly set the kwarg `cc=lazy_identity` when invoking the + continuation-enabled computation (e.g. in the example below, we could + `ourpromises = doit(cc=lazy_identity)`). + Example:: with lazify, continuations: From 65bf3e876b35099663b3c71afd826fbdbb8bd286 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 12 May 2021 13:15:54 +0300 Subject: [PATCH 266/832] update Emacs syntax highlight list --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 263f58b6..8b62282c 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2212,7 +2212,7 @@ This Elisp snippet can be used to add syntax highlighting for keywords specific "where" "do" "local" "delete" "continuations" "call_cc" - "curry" "lazify" "envify" "tco" "prefix" "autoreturn" "forall" + "autocurry" "lazify" "envify" "tco" "prefix" "autoreturn" "forall" "multilambda" "namedlambda" "quicklambda" "cond" "aif" "autoref" "dbg" "nb" "macros" "dialects" "q" "u" "n" "a" "s" "t" "h")) ; mcpyrate From b500c93884c0ed275420ad712d287d2857bcc560 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 12 May 2021 13:23:13 +0300 Subject: [PATCH 267/832] improve comments --- unpythonic/syntax/lazify.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 7d577a76..e786b21c 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -761,11 +761,14 @@ def transform_starred(tree, dstarred=False): # NOTE: We must expand all inner macro invocations before we hit this, or we'll produce nonsense. # Hence it is easiest to have `lazify` expand inside-out. elif type(tree) is Subscript: # force only accessed part of obj[...] + # force the slice expression; it is needed to extract the relevant items. self.withstate(tree.slice, forcing_mode="full") tree.slice = self.visit(tree.slice) # resolve reference to the actual container without forcing its items. self.withstate(tree.value, forcing_mode="flat") tree.value = self.visit(tree.value) + # using the currently active forcing mode, force the value returned + # by the subscript expression. tree = f(tree) return tree @@ -787,10 +790,13 @@ def transform_starred(tree, dstarred=False): # in reality there is always an f() around the whole expr.) self.withstate(tree.value, forcing_mode="flat") tree.value = self.visit(tree.value) + # using the currently active forcing mode, force the value returned + # by the attribute expression. tree = f(tree) return tree elif type(tree) is Name and type(tree.ctx) is Load: + # using the currently active forcing mode, force the value. tree = f(tree) # must not recurse when a Name changes into a Call. return tree From e91317ff2a691e875ecd7e4ca7c8fe1050baea61 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 01:55:25 +0300 Subject: [PATCH 268/832] add wikipedia link for "action at a distance" as a technical term --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 8bfeccde..0418cc4c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3107,7 +3107,7 @@ When using both `@generic` or `@typed` and OOP: Based on my own initial experiments with this feature, the machinery itself works well enough, but to really shine - just like resumable exceptions - multiple dispatch needs to be used everywhere, throughout the language's ecosystem. Python obviously doesn't do that. -**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@generic_addmethod` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the method table is global state! +**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@generic_addmethod` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the method table is global state! #### ``typed``: add run-time type checks with type annotation syntax From c4291b0cc2897f0067acb2d5806db64c29d8ea80 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 02:23:48 +0300 Subject: [PATCH 269/832] name the testexpr lambda in _test_expr_signals_or_raises, too --- unpythonic/syntax/testingtools.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 4d125602..4d7edb9b 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -917,9 +917,12 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): # Same remark about outside-in source code capture as in `_test_expr`. sourcecode = unparse(tree) + # Name our lambda to make the stack trace more understandable. + # For consistency, the name matches that used by `_test_expr`. + func_tree = q[h[namelambda]("testexpr")(lambda: a[tree])] return q[(a[asserter])(a[exctype], u[sourcecode], - lambda: a[tree], + a[func_tree], filename=a[filename], lineno=a[ln], message=a[message])] From 2d821c4acb79fa736e93a3176123ebd62d1153f6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 02:29:06 +0300 Subject: [PATCH 270/832] eh, how many bugs can you fit on two lines? --- unpythonic/dispatch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index c6b8c060..7a40654c 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -317,8 +317,8 @@ def methods(): # # TODO: Compute closest candidates, like Julia does? (see methods, MethodError) a = [repr(a) for a in args] - sep = ", " if kwargs else "" - kw = [f"{k}={str(v)}" for k, v in kwargs] + sep = ", " if args and kwargs else "" + kw = [f"{k}={repr(v)}" for k, v in kwargs.items()] def format_method(method): # Taking a page from Julia and some artistic liberty here. thecallable, type_signature = method function, _ = getfunc(thecallable) From 4f2c0f649a7fed5787c05ddde9bb94c899811d8c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 02:59:44 +0300 Subject: [PATCH 271/832] clean up `resolve_bindings`; improve dispatch - We no longer need to implement the function parameter binding algorithm ourselves (for run-time analysis), since Python 3.5+ have `inspect.Signature.bind`. - Using this, improve the multiple-dispatch system so that we can also dispatch on a homogeneous type of contents collected by `**kwargs`. - Improve the error message upon no multiple-dispatch match for a call. --- CHANGELOG.md | 3 + doc/features.md | 12 +- unpythonic/arity.py | 192 ++++++++---------------------- unpythonic/dispatch.py | 116 +++++++++--------- unpythonic/tests/test_arity.py | 95 +++------------ unpythonic/tests/test_dispatch.py | 12 ++ 6 files changed, 155 insertions(+), 275 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9e48d98..38f008df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,6 +80,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. + - Add `unpythonic.dispatch.isgeneric` to detect whether a callable has been declared `@generic`. + - `@generic` et al.: it is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) @@ -140,6 +142,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) - Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. - This was an oversight when this function was added; most other functions in `unpythonic.it` have been curry-friendly from the beginning. + - Change output format of `resolve_bindings` to return an `inspect.BoundArguments` instead of the previous `OrderedDict` that had a custom format. Change the input format of `tuplify_bindings` to match. - Change parameter name from `l` to `length` in the functions `in_slice` and `index_in_slice` (in the `unpythonic.collections` module). - These are mostly used internally, but technically a part of the public API. - This change fixes a `flake8` [E741](https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes) warning, and the new name for the parameter is more descriptive. diff --git a/doc/features.md b/doc/features.md index 0418cc4c..206d87a8 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3007,7 +3007,11 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of **Changed in v0.14.3**. *The `@generic` and `@typed` decorators can now decorate also instance methods, class methods and static methods (beside regular functions, as previously in 0.14.2).* -**Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Added the `@generic_addmethod` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. (Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.)* +**Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Starting with this release, they receive the same semantic-versioning guarantees as the rest of `unpythonic`.* + +*Added the `@generic_addmethod` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* + +*It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* The ``generic`` decorator allows creating multiple-dispatch generic functions (a.k.a. multimethods) with type annotation syntax. @@ -3690,6 +3694,10 @@ assert getattrrec(w, "x") == 23 **Added in v0.14.2**: `resolve_bindings`. *Get the parameter bindings a given callable would establish if it was called with the given args and kwargs. This is mainly of interest for implementing memoizers, since this allows them to see (e.g.) `f(1)` and `f(a=1)` as the same thing for `def f(a): pass`.* +**Changed in v0.15.0.** *Now `resolve_bindings` is a thin wrapper on top of `inspect.Signature.bind`, which was added in Python 3.5. In `unpythonic` 0.14.2 and 0.14.3, we used to have our own implementation of the parameter binding algorithm (that ran also on Python 3.4), but it is no longer needed, since now we support only Python 3.6 and later. Now `resolve_bindings` returns an `inspect.BoundArguments` object.* + +*Now `tuplify_bindings` accepts an `inspect.BoundArguments` object instead of its previous input format. The function is only ever intended to be used to postprocess the output of `resolve_bindings`, so this change shouldn't affect your own code.* + Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by ``inspect``. Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit ``self`` and ``cls`` are ignored). @@ -3751,7 +3759,7 @@ We special-case the builtin functions that either fail to return any arity (are If the arity cannot be inspected, and the function is not one of the special-cased builtins, the ``UnknownArity`` exception is raised. -These functions are internally used in various places in unpythonic, particularly ``curry``. The ``let`` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. +These functions are internally used in various places in unpythonic, particularly ``curry``, ``fix``, and ``@generic``. The ``let`` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. Inspired by various Racket functions such as ``(arity-includes?)`` and ``(procedure-keywords)``. diff --git a/unpythonic/arity.py b/unpythonic/arity.py index f99adcba..3453dcd7 100644 --- a/unpythonic/arity.py +++ b/unpythonic/arity.py @@ -11,8 +11,8 @@ "resolve_bindings", "tuplify_bindings", "UnknownArity"] +import copy from inspect import signature, Parameter, ismethod -from collections import OrderedDict import operator class UnknownArity(ValueError): @@ -305,13 +305,25 @@ def arity_includes(f, n): lower, upper = arities(f) return lower <= n <= upper -# TODO: Can we replace this by `inspect.Signature.bind`, provided by Python 3.5+? def resolve_bindings(f, *args, **kwargs): """Resolve parameter bindings established by `f` when called with the given args and kwargs. This is an inspection tool, which does not actually call `f`. This is useful for memoizers and other similar decorators that need a canonical representation of `f`'s parameter bindings. - If you want a hashable result, postprocess the return value with `tuplify_bindings(result)`. + + **NOTE**: As of v0.15.0, this is a thin wrapper on top of `inspect.Signature.bind`, + which was added in Python 3.5. In `unpythonic` 0.14.2 and 0.14.3, we used to have + our own implementation of the parameter binding algorithm (that ran also on Python 3.4), + but it is no longer needed, since now we support only Python 3.6 and later. + + The only things we do beside call `inspect.Signature.bind` are: + + - If `f` is a method, we extract the raw function first, and analyze the bindings of that. + + - We apply default values (from the definition of `f`) automatically. + + The return value is an `inspect.BoundArguments`. If you want a hashable result, + postprocess the return value with `tuplify_bindings(result)`. For illustration, consider a simplistic memoizer:: @@ -358,152 +370,48 @@ def f(a): f(42) f(a=42) # now the cache hits + """ + f, _ = getfunc(f) + bound_arguments = signature(f).bind(*args, **kwargs) + bound_arguments.apply_defaults() + return bound_arguments - The return value of `resolve_bindings` is an `OrderedDict` with five keys: - args: `OrderedDict` of bindings made for regular parameters - (positional only, positional or keyword, keyword only). - vararg: `tuple` of arguments gathered by the vararg (`*args`) parameter - if the function definition has one; otherwise `None`. - vararg_name: `str`, the name of the vararg parameter; or `None`. - kwarg: `OrderedDict` of bindings gathered by `**kwargs` if the - function definition has one; otherwise `None`. - kwarg_name: `str`, the name of the kwarg parameter; or `None`. +def tuplify_bindings(bound_arguments): + """Convert the return value of `resolve_bindings` into a hashable form. - **NOTE**: + This is useful for memoizers and similar use cases, which need to use a + representation of the bindings as a dictionary key. - We attempt to implement the exact same algorithm Python itself uses for - resolving argument bindings. The process is explained in the language - reference, although not in a step-by-step algorithmic form. + `bound_arguments` is an `inspect.BoundArguments` object. - https://docs.python.org/3/reference/compound_stmts.html#function-definitions - https://docs.python.org/3/reference/expressions.html#calls + In our return value, `bound_arguments.arguments` itself, as well as the value of + the `**kwargs` parameter contained in it, if any, are converted from `OrderedDict` + to `tuple` using `tuple(od.items())`. - This function should report exactly those bindings that would actually be - established if `f` was actually called with the given `args` and `kwargs`. + The result is hashable, if all the passed arguments are. - If you encounter a case with any difference between what the result claims and - how Python itself assigns the bindings, that is a bug in our code. In such a - case, please report the issue, so it can be fixed, and then added to the unit - tests to ensure it won't come back. + See `resolve_bindings` for an example. """ - f, _ = getfunc(f) - params = signature(f).parameters - - # https://docs.python.org/3/library/inspect.html#inspect.Signature - # https://docs.python.org/3/library/inspect.html#inspect.Parameter - poskinds = set((Parameter.POSITIONAL_ONLY, - Parameter.POSITIONAL_OR_KEYWORD)) - kwkinds = set((Parameter.POSITIONAL_OR_KEYWORD, - Parameter.KEYWORD_ONLY)) - varkinds = set((Parameter.VAR_POSITIONAL, - Parameter.VAR_KEYWORD)) - - index = {} - nposparams = 0 - varpos = varkw = None - for slot, param in enumerate(params.values()): - if param.kind in poskinds: - nposparams += 1 - if param.kind in kwkinds: - index[param.name] = slot - if param.kind == Parameter.VAR_POSITIONAL: - varpos = slot - varpos_name = param.name - elif param.kind == Parameter.VAR_KEYWORD: - varkw = slot - varkw_name = param.name - - # https://docs.python.org/3/reference/compound_stmts.html#function-definitions - # https://docs.python.org/3/reference/expressions.html#calls - unassigned = object() # gensym("unassigned"), but object() is much faster, and we don't need a label, or pickle support. - slots = [unassigned for _ in range(len(params))] # yes, varparams too - - # fill from positional arguments - for slot, (param, value) in enumerate(zip(params.values(), args)): - if param.kind in varkinds: # these are always last in the function def + def tuplify(ordereddict): + return tuple(ordereddict.items()) + + # Tuplify the **kwargs dict. + # + # The information of which parameter it is, if any, is not contained in the + # `arguments` attribute of the `BoundArguments` instance; we need to scan + # the signature (stored in the `signature` attribute) against which the + # bindings were made. + for parameter in bound_arguments.signature.parameters.values(): + if parameter.kind == Parameter.VAR_KEYWORD: + kwargs_param = parameter.name break - slots[slot] = value - - if varpos is not None: - slots[varpos] = [] - if varkw is not None: - slots[varkw] = OrderedDict() - vkdict = slots[varkw] - - # gather excess positional arguments - if len(args) > nposparams: - if varpos is None: - raise TypeError(f"{f.__name__}() takes {nposparams} positional arguments but {len(args)} were given") - slots[varpos] = args[nposparams:] - - # fill from keyword arguments - for identifier, value in kwargs.items(): - if identifier in index: - slot = index[identifier] - if slots[slot] is unassigned: - slots[slot] = value - else: - raise TypeError(f"{f.__name__}() got multiple values for argument '{identifier}'") - elif varkw is not None: # gather excess keyword arguments - vkdict[identifier] = value - else: - raise TypeError(f"{f.__name__}() got an unexpected keyword argument '{identifier}'") - - # fill missing with defaults from function definition - failures = [] - for slot, param in enumerate(params.values()): - if slots[slot] is unassigned: - if param.default is Parameter.empty: - failures.append(param.name) - slots[slot] = param.default - # Python 3.6 goes so far to make this particular error message into proper - # English, that aping the standard error message takes the most effort here... - if failures: - if len(failures) == 1: - n1 = failures[0] - raise TypeError(f"{f.__name__}() missing required positional argument: '{n1}'") - if len(failures) == 2: - n1, n2 = failures - raise TypeError(f"{f.__name__}() missing 2 required positional arguments: '{n1}' and '{n2}'") - wrapped = [f"'{x}'" for x in failures] - others = ", ".join(wrapped[:-1]) - msg = f"{f.__name__}() missing {len(failures)} required positional arguments: {others}, and '{failures[-1]}'" - raise TypeError(msg) - - # build the result - regularargs = OrderedDict() - for param, value in zip(params.values(), slots): - if param.kind in varkinds: # skip varpos, varkw - continue - regularargs[param.name] = value - - # Naming of the fields matches `ast.arguments` - # https://greentreesnakes.readthedocs.io/en/latest/nodes.html#arguments - bindings = OrderedDict() - bindings["args"] = regularargs - bindings["vararg"] = slots[varpos] if varpos is not None else None - bindings["vararg_name"] = varpos_name if varpos is not None else None # for introspection - bindings["kwarg"] = slots[varkw] if varkw is not None else None - bindings["kwarg_name"] = varkw_name if varkw is not None else None # for introspection - - return bindings - -def tuplify_bindings(bindings): - """Convert the return value of `resolve_bindings` into a hashable form. + else: + kwargs_param = None - This is useful for memoizers and similar use cases, which need to use a - representation of the bindings as a dictionary key. + if kwargs_param: + thearguments = copy.copy(bound_arguments.arguments) # avoid mutating our input + thearguments[kwargs_param] = tuplify(thearguments[kwargs_param]) + else: + thearguments = bound_arguments.arguments - The values stored in the `"args"` and `"kwarg"` keys, as well as `bindings` - itself, are converted from `OrderedDict` to `tuple` using `tuple(od.items())`. - The result is hashable, if all the arguments passed in the bindings are. - """ - def tuplify(od): - return tuple(od.items()) - result = OrderedDict() - result["args"] = tuplify(bindings["args"]) - result["vararg"] = bindings["vararg"] - result["vararg_name"] = bindings["vararg_name"] - result["kwarg"] = tuplify(bindings["kwarg"]) if bindings["kwarg"] is not None else None - result["kwarg_name"] = bindings["kwarg_name"] - return tuplify(result) + return tuplify(thearguments) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 7a40654c..125a1cd3 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -6,7 +6,7 @@ https://docs.python.org/3/library/functools.html#functools.singledispatch """ -__all__ = ["generic", "generic_addmethod", "typed"] +__all__ = ["isgeneric", "generic", "generic_addmethod", "typed"] from functools import partial, wraps from itertools import chain @@ -36,36 +36,9 @@ # parameter, just append the names you use to this list. # """ -@register_decorator(priority=98) -def generic_addmethod(target): - """Parametric decorator. Add a method to function `target`. - - Like `@generic`, but the target function on which the method will be - registered is chosen separately, so that you can extend a generic - function previously defined in some other `.py` source file. - - Usage:: - - # example.py - from unpythonic import generic - - @generic - def f(x: int): - ... - - - # main.py - from unpythonic import generic_addmethod - import example - - @generic_addmethod(example.f) - def f(x: float): - ... - """ - # TODO: maybe needs some more official way to detect if `target` has been declared `@generic`. - if not hasattr(target, "_method_registry"): - raise TypeError(f"{target} is not a generic function, cannot add methods to it.") - return partial(_register_generic, _getfullname(target)) +def isgeneric(f): + """Return whether the callable `f` has been declared `@generic` (which see).""" + return hasattr(f, "_method_registry") @register_decorator(priority=98) def generic(f): @@ -191,9 +164,6 @@ def example(): annotate it as `typing.Any`; don't just omit the type annotation. Explicit is better than implicit; **this is a feature**. - Dispatching by the contents of the `**kwargs` dictionary is not (yet) - supported. - See the limitations in `unpythonic.typecheck` for which features of the `typing` module are supported and which are not. @@ -202,6 +172,51 @@ def example(): """ return _register_generic(_getfullname(f), f) +@register_decorator(priority=98) +def generic_addmethod(target): + """Parametric decorator. Add a method to function `target`. + + Like `@generic`, but the target function on which the method will be + registered is chosen separately, so that you can extend a generic + function previously defined in some other `.py` source file. + + Usage:: + + # example.py + from unpythonic import generic + + @generic + def f(x: int): + ... + + + # main.py + from unpythonic import generic_addmethod + import example + + class MyOwnType: + ... + + @generic_addmethod(example.f) + def f(x: MyOwnType): + ... + + **CAUTION**: Beware of type piracy when you use this. That is: + + 1. For arbitrary input types you don't own, extend only a function you own, OR + 2. Extend a function defined somewhere else only for input types you own. + + Satisfying **one** of these conditions is sufficient to avoid type piracy. + + See: + https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/ + https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming) + https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy + """ + if not isgeneric(target): + raise TypeError(f"{target} is not a generic function, cannot add methods to it.") + return partial(_register_generic, _getfullname(target)) + # Modeled after `mcpyrate.utils.format_macrofunction`, which does the same thing for macros. def _getfullname(f): function, _ = getfunc(f) @@ -240,17 +255,9 @@ def name_of_1st_positional_parameter(f): @wraps(f) def dispatcher(*args, **kwargs): # `signature` comes from typing.get_type_hints. - # `bindings` is populated in the surrounding scope below. + # `bound_arguments` is populated in the surrounding scope below. def match_argument_types(type_signature): - # TODO: handle **kwargs (bindings["kwarg"], bindings["kwarg_name"]) - args_items = bindings["args"].items() - if bindings["vararg_name"]: - vararg_item = (bindings["vararg_name"], bindings["vararg"]) # *args - all_items = tuple(chain(args_items, (vararg_item,))) - else: - all_items = args_items - - for parameter, value in all_items: + for parameter, value in bound_arguments.arguments.items(): assert parameter in type_signature # resolve_bindings should already TypeError when not. expected_type = type_signature[parameter] if not isoftype(value, expected_type): @@ -271,7 +278,7 @@ def methods(): # # See discussions on interaction between `@staticmethod` and `super` in Python: # https://bugs.python.org/issue31118 - # https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879 + # https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879 # # TODO/FIXME: Not possible to detect self/cls parameters correctly. # Here we're operating at the wrong abstraction level for that, @@ -294,17 +301,17 @@ def methods(): if hasattr(base, f.__name__): # does this particular super have f? base_oop_method = getattr(base, f.__name__) base_raw_function, _ = getfunc(base_oop_method) - if hasattr(base_raw_function, "_method_registry"): # it's @generic + if isgeneric(base_raw_function): # it's @generic base_registry = getattr(base_raw_function, "_method_registry") relevant_registries.append(reversed(base_registry)) return chain.from_iterable(relevant_registries) - for method, signature in methods(): + for method, type_signature in methods(): try: - bindings = resolve_bindings(method, *args, **kwargs) + bound_arguments = resolve_bindings(method, *args, **kwargs) except TypeError: # arity mismatch, so this method can't be the one the call is looking for. continue - if match_argument_types(signature): + if match_argument_types(type_signature): return method(*args, **kwargs) # No match, report error. @@ -321,17 +328,21 @@ def methods(): kw = [f"{k}={repr(v)}" for k, v in kwargs.items()] def format_method(method): # Taking a page from Julia and some artistic liberty here. thecallable, type_signature = method + # Our `type_signature` is based on `typing.get_type_hints`, + # but for the error message, we need something that formats + # like source code. Hence use `inspect.signature`. + thesignature = inspect.signature(thecallable) function, _ = getfunc(thecallable) filename = inspect.getsourcefile(function) source, firstlineno = inspect.getsourcelines(function) - return f"{type_signature} from {filename}:{firstlineno}" + return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" methods_str = [f" {format_method(x)}" for x in methods()] candidates = "\n".join(methods_str) function, _ = getfunc(f) args_str = ", ".join(a) kws_str = ", ".join(kw) - msg = (f"No method found matching {function.__qualname__}({args_str}{sep}{kws_str}).\n" - f"Candidate signatures (in order of match attempts):\n{candidates}") + msg = (f"No multiple-dispatch match for the call {function.__qualname__}({args_str}{sep}{kws_str}).\n" + f"Multimethods for {repr(function.__qualname__)} (most recent match attempt last):\n{candidates}") raise TypeError(msg) dispatcher._method_registry = [] @@ -400,7 +411,6 @@ def register(thecallable): return dispatcher._register(f) raise TypeError("@typed: cannot register additional methods.") - @register_decorator(priority=98) def typed(f): """Decorator. Restrict allowed argument types to one combination only. diff --git a/unpythonic/tests/test_arity.py b/unpythonic/tests/test_arity.py index ee1aa9a0..fccda24e 100644 --- a/unpythonic/tests/test_arity.py +++ b/unpythonic/tests/test_arity.py @@ -119,84 +119,33 @@ def f(a): def f(a=42): pass # pragma: no cover - test[r(f) == (("args", (("a", 42),)), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] - test[r(f, 17) == (("args", (("a", 17),)), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] - test[r(f, a=23) == (("args", (("a", 23),)), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] + test[r(f) == (("a", 42),)] + test[r(f, 17) == (("a", 17),)] + test[r(f, a=23) == (("a", 23),)] def f(a, b, c): pass # pragma: no cover - test[r(f, 1, 2, 3) == (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] - test[r(f, a=1, b=2, c=3) == (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] - test[r(f, 1, 2, c=3) == (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] - test[r(f, 1, c=3, b=2) == (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] - test[r(f, c=3, b=2, a=1) == (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", None), ("vararg_name", None), - ("kwarg", None), ("kwarg_name", None))] + test[r(f, 1, 2, 3) == (("a", 1), ("b", 2), ("c", 3))] + test[r(f, a=1, b=2, c=3) == (("a", 1), ("b", 2), ("c", 3))] + test[r(f, 1, 2, c=3) == (("a", 1), ("b", 2), ("c", 3))] + test[r(f, 1, c=3, b=2) == (("a", 1), ("b", 2), ("c", 3))] + test[r(f, c=3, b=2, a=1) == (("a", 1), ("b", 2), ("c", 3))] def f(a, b, c, *args): pass # pragma: no cover - test[r(f, 1, 2, 3, 4, 5) == (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", (4, 5)), ("vararg_name", "args"), - ("kwarg", None), ("kwarg_name", None))] - - # On Pythons < 3.6, there's no guarantee about the ordering of the kwargs. - # Our analysis machinery preserves the order it gets, but the *input* - # may already differ from how the invocation of `r` is written in the - # source code here. - # - # So we must allow for arbitrary ordering of the kwargs when checking - # the result. - # - def checkpre36(result, truth): - args_r, vararg_r, vararg_name_r, kwarg_r, kwarg_name_r = result - args_t, vararg_t, vararg_name_t, kwarg_t, kwarg_name_t = truth - couldbe = (args_r == args_t and vararg_r == vararg_t and - vararg_name_r == vararg_name_t and kwarg_name_r == kwarg_name_t) - if not couldbe: - return False # pragma: no cover, should only happen if the tests fail. - name_r, contents_r = kwarg_r - name_t, contents_t = kwarg_t - return name_r == name_t and set(contents_r) == set(contents_t) + test[r(f, 1, 2, 3, 4, 5) == (('a', 1), ('b', 2), ('c', 3), + ('args', (4, 5)))] def f(a, b, c, **kw): - pass # pragma: no cover - test[checkpre36(the[r(f, 1, 2, 3, d=4, e=5)], (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", None), ("vararg_name", None), - ("kwarg", (("d", 4), ("e", 5))), ("kwarg_name", "kw")))] + pass + test[r(f, 1, 2, 3, d=4, e=5) == (('a', 1), ('b', 2), ('c', 3), + ('kw', (('d', 4), ('e', 5))))] def f(a, b, c, *args, **kw): - pass # pragma: no cover - test[checkpre36(the[r(f, 1, 2, 3, 4, 5, d=6, e=7)], (("args", (("a", 1), ("b", 2), ("c", 3))), - ("vararg", (4, 5)), ("vararg_name", "args"), - ("kwarg", (("d", 6), ("e", 7))), ("kwarg_name", "kw")))] - - # TODO: On Python 3.6+, this becomes just: - # - # def f(a, b, c, **kw): - # pass - # test[r(f, 1, 2, 3, d=4, e=5) == (("args", (("a", 1), ("b", 2), ("c", 3))), - # ("vararg", None), ("vararg_name", None), - # ("kwarg", (("d", 4), ("e", 5))), ("kwarg_name", "kw"))] - # - # def f(a, b, c, *args, **kw): - # pass - # test[r(f, 1, 2, 3, 4, 5, d=6, e=7) == (("args", (("a", 1), ("b", 2), ("c", 3))), - # ("vararg", (4, 5)), ("vararg_name", "args"), - # ("kwarg", (("d", 6), ("e", 7))), ("kwarg_name", "kw"))] + pass + test[r(f, 1, 2, 3, 4, 5, d=6, e=7) == (('a', 1), ('b', 2), ('c', 3), + ('args', (4, 5)), + ('kw', (('d', 6), ('e', 7))))] with testset("resolve_bindings error cases"): def f(a): @@ -205,16 +154,6 @@ def f(a): test_raises[TypeError, resolve_bindings(f, 1, a=2)] # same arg assigned twice test_raises[TypeError, resolve_bindings(f, 1, b=2)] # unexpected kwarg - # The number of missing required positional args affects the error message - # à la Python 3.6, so let's exercise that part of the code, too. - test_raises[TypeError, resolve_bindings(f)] # missing 1 required positional arg - def g(a, b): - pass # pragma: no cover - test_raises[TypeError, resolve_bindings(g)] # missing 2 required positional args - def h(a, b, c): - pass # pragma: no cover - test_raises[TypeError, resolve_bindings(h)] # missing 3 required positional args - if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 7bcdc40a..3e89ad61 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -58,6 +58,14 @@ def gargle(*args: typing.Tuple[float, ...]): # any number of floats # noqa: F8 def gargle(*args: typing.Tuple[int, float, str]): # three args, matching the given types # noqa: F811 return "int, float, str" +# v0.15.0: dispatching on a homogeneous type inside **kwargs is also supported, via `typing.Dict` +@generic +def kittify(**kwargs: typing.Dict[str, int]): + return "int" +@generic +def kittify(**kwargs: typing.Dict[str, float]): # noqa: F811 + return "float" + # One-method pony, which automatically enforces argument types. # The type specification may use features from the `typing` stdlib module. @typed @@ -91,6 +99,10 @@ def runtests(): test[gargle(42, 6.022e23, "hello") == "int, float, str"] test[gargle(1, 2, 3) == "int"] # as many as in the [int, float, str] case + test[kittify(x=1, y=2) == "int"] + test[kittify(x=1.0, y=2.0) == "float"] + test_raises[TypeError, kittify(x=1, y=2.0)] + with testset("@generic_addmethod"): @generic def f1(x: typing.Any): From 3fcb6f56ae0d1ac5b4cb6a2be7144375a6b30ae8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 03:06:50 +0300 Subject: [PATCH 272/832] improve comment --- unpythonic/tests/test_dispatch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 3e89ad61..4c15b302 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -60,10 +60,10 @@ def gargle(*args: typing.Tuple[int, float, str]): # three args, matching the gi # v0.15.0: dispatching on a homogeneous type inside **kwargs is also supported, via `typing.Dict` @generic -def kittify(**kwargs: typing.Dict[str, int]): +def kittify(**kwargs: typing.Dict[str, int]): # all kwargs are ints return "int" @generic -def kittify(**kwargs: typing.Dict[str, float]): # noqa: F811 +def kittify(**kwargs: typing.Dict[str, float]): # all kwargs are floats # noqa: F811 return "float" # One-method pony, which automatically enforces argument types. From b14e17120bc541f59b82331708e8e29825fc36a5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 03:11:58 +0300 Subject: [PATCH 273/832] improve macro docs --- doc/macros.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 8b62282c..ecc50695 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2143,7 +2143,9 @@ Is this just a set of macros, a language extension, or a compiler for a new lang The macros in ``unpythonic.syntax`` are designed to work together, but some care needs to be taken regarding the order in which they expand. -The block macros are designed to run **in the following order (leftmost first)**: +For simplicity, **the block macros make no attempt to prevent invalid combos** (unless there is a specific technical reason to do that for some particular combination). Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. + +The **AST edits** performed by the block macros are designed to run **in the following order (leftmost first)**: ``` prefix > autoreturn, quicklambda > multilambda > continuations or tco > ... @@ -2154,9 +2156,7 @@ The ``let_syntax`` (and ``abbrev``) block may be placed anywhere in the chain; j The ``dbg`` block can be run at any position after ``prefix`` and before ``tco`` (or ``continuations``). (It must be able to see function calls in Python's standard format, for detecting calls to the print function.) -For simplicity, **the block macros make no attempt to prevent invalid combos** (unless there is a specific technical reason to do that for some particular combination). Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. - -The correct ordering for block macro invocations is somewhat complicated by the fact that some of the above are two-pass macros. Consider this artificial example, where `mac` is a two-pass macro: +The correct ordering for **block macro invocations** - which is the actual user-facing part - is somewhat complicated by the fact that some of the above are two-pass macros. Consider this artificial example, where `mac` is a two-pass macro: ```python with mac: @@ -2170,7 +2170,7 @@ The invocation `with mac` is *lexically on the outside*, thus the macro expander 2. Explicit recursion by `with mac`. This expands the `with cheese`. 3. Second pass (inside out) of `with mac`. -So for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. +So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. Example combo in the single-line format: @@ -2190,7 +2190,7 @@ with autoreturn: Of these, `autoreturn` expands outside-in, while `lazify` and `tco` are both two-pass macros. -To see if something is a two-pass macro, for now, grep the codebase for `expander.visit`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). +We aim to improve the macro docs in the future. For now, to see if something is a two-pass macro, grep the codebase for `expander.visit`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). See our [notes on macros](../doc/design-notes.md#detailed-notes-on-macros) for more information. From 831348c3eed70e33174f2733d2292cc2b4d0ee4d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 03:21:31 +0300 Subject: [PATCH 274/832] improve macro docs --- doc/macros.md | 4 ++-- unpythonic/syntax/tests/test_lazify.py | 10 ++++++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index ecc50695..59a23253 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2141,7 +2141,7 @@ Is this just a set of macros, a language extension, or a compiler for a new lang ### The xmas tree combo -The macros in ``unpythonic.syntax`` are designed to work together, but some care needs to be taken regarding the order in which they expand. +The macros in ``unpythonic.syntax`` are designed to work together, but some care needs to be taken regarding the order in which they expand. This complexity unfortunately comes with any pick-and-mix-your-own-language kit, because some features inevitably interact. For example, it is possible to lazify [continuation-enabled](https://en.wikipedia.org/wiki/Continuation-passing_style) code, but running the transformations the other way around produces nonsense. For simplicity, **the block macros make no attempt to prevent invalid combos** (unless there is a specific technical reason to do that for some particular combination). Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. @@ -2170,7 +2170,7 @@ The invocation `with mac` is *lexically on the outside*, thus the macro expander 2. Explicit recursion by `with mac`. This expands the `with cheese`. 3. Second pass (inside out) of `with mac`. -So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. +So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. Example combo in the single-line format: diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index a2e5b674..18e9e8ec 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -502,6 +502,16 @@ def f(a, b): # - cc built by chain_conts is treated as lazy, **itself**; then it's up to # the continuations chained by it to decide whether to force their args. # - the default cont ``identity`` is strict, so it will force return values + # - if you want a non-strict identity for use at the entry point to your + # continuation-enabled computation, do this: + # + # from unpythonic import identity + # from unpythonic.lazyutil import passthrough_lazy_args + # lazy_identity = passthrough_lazy_args(identity) + # + # and then explicitly set the kwarg `cc=lazy_identity` when invoking the + # continuation-enabled computation (e.g. in the example below, we could + # `ourpromises = doit(cc=lazy_identity)`). with testset("integration with continuations"): with lazify, continuations: k = None From a7c28021b45b4169555f2dcfb9d5f64f095049d4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 03:24:16 +0300 Subject: [PATCH 275/832] fix typo --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 59a23253..85b8cde9 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2170,7 +2170,7 @@ The invocation `with mac` is *lexically on the outside*, thus the macro expander 2. Explicit recursion by `with mac`. This expands the `with cheese`. 3. Second pass (inside out) of `with mac`. -So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. +So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, even though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. Example combo in the single-line format: From 94b3c0e146eea7fbb86ac4d0c5db959e39705cc1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 03:32:23 +0300 Subject: [PATCH 276/832] improve macro docs --- doc/macros.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/macros.md b/doc/macros.md index 85b8cde9..b555e9bc 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2194,6 +2194,10 @@ We aim to improve the macro docs in the future. For now, to see if something is See our [notes on macros](../doc/design-notes.md#detailed-notes-on-macros) for more information. +**NOTE**: In MacroPy, there sometimes were [differences](https://github.com/azazel75/macropy/issues/21) between the behavior of the single-line and multi-line invocation format, but in `mcpyrate`, they should behave the same. + +There is still [a minor difference](https://github.com/Technologicat/mcpyrate/issues/3) if there are at least three nested macro invocations, and a macro is scanning the tree for another macro invocation; then the tree looks different depending on whether the single-line or the multi-line format was used. The differences in that are as one would expect knowing [how `with` statements look like](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#With) in the Python AST. The reason the difference manifests only for three or more macro invocations is that `mcpyrate` pops the macro that is being expanded before it hands over the tree to the macro code; hence if there are only two, the inner tree will have only one "context manager" in its `with`. + ### Emacs syntax highlighting This Elisp snippet can be used to add syntax highlighting for keywords specific to `mcpyrate` and `unpythonic.syntax` to your Emacs setup: From b7b28c97109896c6515c95c6e798e1bf96aefd15 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 03:32:55 +0300 Subject: [PATCH 277/832] wording --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index b555e9bc..390c3acd 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2196,7 +2196,7 @@ See our [notes on macros](../doc/design-notes.md#detailed-notes-on-macros) for m **NOTE**: In MacroPy, there sometimes were [differences](https://github.com/azazel75/macropy/issues/21) between the behavior of the single-line and multi-line invocation format, but in `mcpyrate`, they should behave the same. -There is still [a minor difference](https://github.com/Technologicat/mcpyrate/issues/3) if there are at least three nested macro invocations, and a macro is scanning the tree for another macro invocation; then the tree looks different depending on whether the single-line or the multi-line format was used. The differences in that are as one would expect knowing [how `with` statements look like](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#With) in the Python AST. The reason the difference manifests only for three or more macro invocations is that `mcpyrate` pops the macro that is being expanded before it hands over the tree to the macro code; hence if there are only two, the inner tree will have only one "context manager" in its `with`. +With `mcpyrate`, there is still [a minor difference](https://github.com/Technologicat/mcpyrate/issues/3) if there are at least three nested macro invocations, and a macro is scanning the tree for another macro invocation; then the tree looks different depending on whether the single-line or the multi-line format was used. The differences in that are as one would expect knowing [how `with` statements look like](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#With) in the Python AST. The reason the difference manifests only for three or more macro invocations is that `mcpyrate` pops the macro that is being expanded before it hands over the tree to the macro code; hence if there are only two, the inner tree will have only one "context manager" in its `with`. ### Emacs syntax highlighting From d17d1b5b8aa9ef2ad92475325ce5c8952633c128 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 13 May 2021 12:20:58 +0300 Subject: [PATCH 278/832] isgeneric: return "generic", "typed", or False --- unpythonic/dispatch.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 125a1cd3..a52dfe76 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -37,8 +37,17 @@ # """ def isgeneric(f): - """Return whether the callable `f` has been declared `@generic` (which see).""" - return hasattr(f, "_method_registry") + """Return whether the callable `f` is a multiple-dispatch generic function. + + If `f` was declared `@generic`, return the string `"generic"`. + If `f` was declared `@typed`, return the string `"typed"`. + Otherwise, return `False`. + """ + if not hasattr(f, "_method_registry"): + return False + if hasattr(f, "_register"): + return "generic" + return "typed" @register_decorator(priority=98) def generic(f): @@ -407,9 +416,9 @@ def register(thecallable): dispatcher._register = register # save it for use by us later _dispatcher_registry[fullname] = dispatcher dispatcher = _dispatcher_registry[fullname] - if hasattr(dispatcher, "_register"): # co-operation with @typed, below - return dispatcher._register(f) - raise TypeError("@typed: cannot register additional methods.") + if isgeneric(dispatcher) == "typed": # co-operation with @typed, below + raise TypeError("@typed: cannot register additional methods.") + return dispatcher._register(f) @register_decorator(priority=98) def typed(f): From bdca0df7f0bc6a7778ce864d61869b5d5f0baf38 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 14 May 2021 00:26:12 +0300 Subject: [PATCH 279/832] rename generic_addmethod -> augment --- CHANGELOG.md | 2 +- doc/features.md | 4 ++-- unpythonic/dispatch.py | 14 +++++++------- unpythonic/tests/test_dispatch.py | 22 +++++++++++----------- 4 files changed, 21 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38f008df..077838a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,7 +79,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Miscellaneous.** - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - - Add `unpythonic.dispatch.generic_addmethod`: add methods to a generic function defined elsewhere. + - Add `unpythonic.dispatch.augment`: add methods to a generic function defined elsewhere. - Add `unpythonic.dispatch.isgeneric` to detect whether a callable has been declared `@generic`. - `@generic` et al.: it is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). diff --git a/doc/features.md b/doc/features.md index 206d87a8..7811dc31 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3009,7 +3009,7 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of **Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Starting with this release, they receive the same semantic-versioning guarantees as the rest of `unpythonic`.* -*Added the `@generic_addmethod` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* +*Added the `@augment` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* @@ -3111,7 +3111,7 @@ When using both `@generic` or `@typed` and OOP: Based on my own initial experiments with this feature, the machinery itself works well enough, but to really shine - just like resumable exceptions - multiple dispatch needs to be used everywhere, throughout the language's ecosystem. Python obviously doesn't do that. -**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@generic_addmethod` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the method table is global state! +**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the method table is global state! #### ``typed``: add run-time type checks with type annotation syntax diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index a52dfe76..76c8d51a 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -6,7 +6,7 @@ https://docs.python.org/3/library/functools.html#functools.singledispatch """ -__all__ = ["isgeneric", "generic", "generic_addmethod", "typed"] +__all__ = ["isgeneric", "generic", "augment", "typed"] from functools import partial, wraps from itertools import chain @@ -182,7 +182,7 @@ def example(): return _register_generic(_getfullname(f), f) @register_decorator(priority=98) -def generic_addmethod(target): +def augment(target): """Parametric decorator. Add a method to function `target`. Like `@generic`, but the target function on which the method will be @@ -200,20 +200,20 @@ def f(x: int): # main.py - from unpythonic import generic_addmethod + from unpythonic import augment import example class MyOwnType: ... - @generic_addmethod(example.f) + @augment(example.f) def f(x: MyOwnType): ... **CAUTION**: Beware of type piracy when you use this. That is: - 1. For arbitrary input types you don't own, extend only a function you own, OR - 2. Extend a function defined somewhere else only for input types you own. + 1. For arbitrary input types you don't own, augment only a function you own, OR + 2. Augment a function defined somewhere else only for input types you own. Satisfying **one** of these conditions is sufficient to avoid type piracy. @@ -236,7 +236,7 @@ def _getfullname(f): def _register_generic(fullname, f): """Register a method for a generic function. - This is a low-level function; you'll likely want `generic` or `generic_addmethod`. + This is a low-level function; you'll likely want `generic` or `augment`. fullname: str, fully qualified name of target function to register the method on, used as key in the dispatcher registry. diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 4c15b302..1fe708f6 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -5,7 +5,7 @@ import typing from ..fun import curry -from ..dispatch import generic, generic_addmethod, typed +from ..dispatch import generic, augment, typed @generic def zorblify(x: int, y: int): @@ -103,11 +103,11 @@ def runtests(): test[kittify(x=1.0, y=2.0) == "float"] test_raises[TypeError, kittify(x=1, y=2.0)] - with testset("@generic_addmethod"): + with testset("@augment"): @generic def f1(x: typing.Any): return False - @generic_addmethod(f1) + @augment(f1) def f2(x: int): return x test[f1("hello") is False] @@ -115,8 +115,8 @@ def f2(x: int): def f3(x: typing.Any): # not @generic! return False - with test_raises[TypeError, "should not be able to @generic_addmethod a non-generic function"]: - @generic_addmethod(f3) + with test_raises[TypeError, "should not be able to @augment a non-generic function"]: + @augment(f3) def f4(x: int): return x @@ -281,12 +281,12 @@ def flippable(x: typing.Any): # default # Since these are in the same lexical scope as the original definition of the # generic function `flippable`, we could do this using `@generic`, but # later extensions (which are the whole point of traits) will need to specify - # on which function the new methods are to be registered, using `@generic_addmethod`. + # on which function the new methods are to be registered, using `@augment`. # So let's do that to show how it's done. - @generic_addmethod(flippable) + @augment(flippable) def flippable(x: str): # noqa: F811 return IsFlippable() - @generic_addmethod(flippable) + @augment(flippable) def flippable(x: int): # noqa: F811 return IsNotFlippable() @@ -301,7 +301,7 @@ def flippable(x: int): # noqa: F811 def flip(x: typing.Any): return flip(flippable(x), x) - # Implementation of `flip`. Same comment about `@generic_addmethod` as above. + # Implementation of `flip`. Same comment about `@augment` as above. # # Here we provide one implementation for "flippable" objects and another one # for "nonflippable" objects. Note this dispatches regardless of the actual @@ -311,10 +311,10 @@ def flip(x: typing.Any): # We could also add methods for specific types if needed. Note this is not # Julia, so the first matching definition wins, instead of the most specific # one. - @generic_addmethod(flip) + @augment(flip) def flip(traitvalue: IsFlippable, x: typing.Any): # noqa: F811 return x[::-1] - @generic_addmethod(flip) + @augment(flip) def flip(traitvalue: IsNotFlippable, x: typing.Any): # noqa: F811 raise TypeError(f"{repr(x)} is IsNotFlippable") From 07a28a0b9f8508c075415d9aaf3096fff7535b7d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 14 May 2021 00:29:48 +0300 Subject: [PATCH 280/832] document return value in docstring --- unpythonic/dispatch.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 76c8d51a..a75af058 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -64,6 +64,9 @@ def generic(f): Julia). All of the definitions, including the first one, become registered as *methods* of the *generic function*. + The return value of `generic` is the multiple-dispatch dispatcher + for the generic function that was created or modified. + A generic function is identified by its *fullname*, defined as "{f.__module__}.{f.__qualname__}". The fullname is computed automatically when the `@generic` decorator runs. For example: @@ -189,6 +192,9 @@ def augment(target): registered is chosen separately, so that you can extend a generic function previously defined in some other `.py` source file. + The return value of `augment` is the multiple-dispatch dispatcher + for the generic function that was modified. + Usage:: # example.py From 8d73c7aac8aae9868725991ac340445539b41a8e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 14 May 2021 03:24:11 +0300 Subject: [PATCH 281/832] refactor multiple-dispatch system; add `methods` to list multimethods --- CHANGELOG.md | 8 +- doc/features.md | 2 + unpythonic/dispatch.py | 399 ++++++++++++++++++++++++++--------------- 3 files changed, 260 insertions(+), 149 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 077838a4..6eb91dd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,9 +79,11 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Miscellaneous.** - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - - Add `unpythonic.dispatch.augment`: add methods to a generic function defined elsewhere. - - Add `unpythonic.dispatch.isgeneric` to detect whether a callable has been declared `@generic`. - - `@generic` et al.: it is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. + - Multiple-dispatch system `unpythonic.dispatch`: + - Add decorator `@augment`: add methods to a generic function defined elsewhere. + - Add function `isgeneric` to detect whether a callable has been declared `@generic`. + - Add function `methods`: display a list of methods of a generic function. + - It is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) diff --git a/doc/features.md b/doc/features.md index 7811dc31..6fd91bc0 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3011,6 +3011,8 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of *Added the `@augment` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* +*Added the function `methods`, which displays a list of methods of a generic function.* + *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* The ``generic`` decorator allows creating multiple-dispatch generic functions (a.k.a. multimethods) with type annotation syntax. diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index a75af058..01957034 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -6,7 +6,8 @@ https://docs.python.org/3/library/functools.html#functools.singledispatch """ -__all__ = ["isgeneric", "generic", "augment", "typed"] +__all__ = ["isgeneric", "generic", "augment", "typed", + "methods", "format_methods", "list_methods"] from functools import partial, wraps from itertools import chain @@ -26,13 +27,13 @@ # TODO: meh, a list instance's __doc__ is not writable. Put this doc somewhere. # -# self_parameter_names.__doc__ = """self/cls parameter names for `@generic`. +# self_parameter_names.__doc__ = """`self`/`cls` parameter names for `@generic`. # # When one of these parameter names appears in the first positional parameter position # of a function decorated with `@generic` (or `@typed`), it is detected as being an # OOP-related `self` or `cls` parameter, triggering special handling. # -# If you use something other than the usual Python naming conventions for the self/cls +# If you use something other than the usual Python naming conventions for the `self`/`cls` # parameter, just append the names you use to this list. # """ @@ -49,16 +50,22 @@ def isgeneric(f): return "generic" return "typed" +# TODO: We essentially need the fullname because the second and further invocations +# TODO: of `@generic`, for the same generic function, receive an entirely different +# TODO: run-time object - the new method. There is no way to know which existing +# TODO: dispatcher to connect that to, other than having a registry that maps the +# TODO: fullname of each already-existing generic function to its dispatcher object. @register_decorator(priority=98) def generic(f): """Decorator. Make `f` a generic function (in the sense of CLOS or Julia). **How to use**: - Just make several function definitions, one for each call signature you - want to support, and decorate each of them with `@generic`. Here - *signature* refers to specific combinations of argument types and/or - different shapes for the argument list. + Just make several function definitions, with the same name in the same + lexical scope, one for each call signature you want to support, and + decorate each of them with `@generic`. Here *signature* refers to specific + combinations of argument types and/or different shapes for the argument + list. The first definition implicitly creates the generic function (like in Julia). All of the definitions, including the first one, become registered @@ -182,7 +189,7 @@ def example(): At the moment, `@generic` does not work with `curry`. Adding curry support needs changes to the dispatch logic in `curry`. """ - return _register_generic(_getfullname(f), f) + return _register_generic(_function_fullname(f), f) @register_decorator(priority=98) def augment(target): @@ -219,7 +226,8 @@ def f(x: MyOwnType): **CAUTION**: Beware of type piracy when you use this. That is: 1. For arbitrary input types you don't own, augment only a function you own, OR - 2. Augment a function defined somewhere else only for input types you own. + 2. Augment a function defined somewhere else only if at least one parameter + is of a type you own. Satisfying **one** of these conditions is sufficient to avoid type piracy. @@ -230,15 +238,208 @@ def f(x: MyOwnType): """ if not isgeneric(target): raise TypeError(f"{target} is not a generic function, cannot add methods to it.") - return partial(_register_generic, _getfullname(target)) + return partial(_register_generic, _function_fullname(target)) -# Modeled after `mcpyrate.utils.format_macrofunction`, which does the same thing for macros. -def _getfullname(f): +@register_decorator(priority=98) +def typed(f): + """Decorator. Restrict allowed argument types to one combination only. + + This can be used to eliminate `isinstance` boilerplate code in the + function body, by allowing the types (for dynamic, run-time checking) + to be specified with a very compact syntax - namely, type annotations. + + Also, unlike a basic `isinstance` check, this allows using features + from the `typing` stdlib module in the type specifications. + + After a `@typed` function has been created, no more methods can be + attached to it. + + `@typed` works with `curry`, because the function has only one call + signature, as usual. + + **CAUTION**: + + If used with `curry`, argument type errors will only be detected when + `curry` triggers the actual call. To fix this, `curry` would need to + perform some more introspection on the callable, and to actually know + about this dispatch system. It's not high on the priority list. + """ + # TODO: Fix the epic fail at fail-fast, and update the corresponding test. + s = generic(f) + del s._register # remove the ability to register more methods + return s + +def methods(f): + """Print, to stdout, a human-readable list of multimethods currently registered to `f`. + + This calls `list_methods`, which see. + + (This is like the `methods` function of Julia.) + """ + print(format_methods(f)) + +def format_methods(f): + """Format, as a string, a human-readable list of multimethods currently registered to `f`. + + This calls `list_methods`, which see. + """ + function, _ = getfunc(f) + methods_str = [f" {_format_method(x)}" for x in list_methods(f)] + methods = "\n".join(methods_str) + return f"Multimethods for @{isgeneric(f)} {repr(function.__qualname__)}:\n{methods}" + +def list_methods(f): + """Return a list of the multimethods currently registered to `f`. + + The methods are returned in the order they would be tried by the dispatcher. + + The return value is a list, where each item is `(callable, type_signature)`. + The type signature is in the format returned by `typing.get_type_hints`. + + `f`: a callable that has been declared `@generic` or `@typed`. + + Bound OOP methods are resolved to the underlying function automatically. + The `self`/`cls` argument is extracted from the `__self__` attribute of + the bound method, enabling linked dispatcher lookups in the MRO. + + **CAUTION**: + + Recall that in Python, instance methods when accessed through the *class* + are just raw functions; the method becomes bound, and thus `self` is set, + when accessed through *an instance* of that class. + + Let `Cat` be a class with an OOP instance method `meow`, and `cat` an + instance of that class. If you call `list_methods(cat.meow)`, you get the + MRO lookup for linked dispatchers, as expected. + + But if you call `list_methods(Cat.meow)` instead, it won't see the MRO, + because the value of the `self` argument isn't set for an unbound method + (which is really just a raw function). + + If `Cat` has a `@classmethod` `iscute`, calling `list_methods(Cat.iscute)` + performs the MRO lookup for linked dispatchers. This is because a class + method is already bound (to the class, so the `cls` argument already has + a value) when it is accessed through the class. + + Finally, note that while that is how `list_methods` works, it is not the + mechanism actually used to determine `self`/`cls` when *calling* the + generic function. There, the value is extracted from the first positional + argument of the call. This is because the dispatcher is actually installed + on the underlying raw function, so it has no access to the metadata of the + bound method (which, as seen from the dispatcher, is on the outside). + """ function, _ = getfunc(f) + if not isgeneric(function): + raise TypeError(f"{repr(function.__qualname__)} is not a generic function, it does not have multimethods.") + + # In case of a bound method (either `Foo.classmeth` or `foo.instmeth`), + # we can get the value for `self`/`cls` argument from its `__self__` attribute. + # + # Otherwise we have a regular function, an unbound method, or a `@staticmethod`; + # in those cases, there's no `self`/`cls`. (Technically, an unbound method has + # a parameter to receive it, but no value has been set yet.) + self_or_cls = f.__self__ if hasattr(f, "__self__") else None + return _list_multimethods(function, self_or_cls) + +# -------------------------------------------------------------------------------- + +# Modeled after `mcpyrate.utils.format_macrofunction`, which does the same thing for macros. +def _function_fullname(f): + function, _ = getfunc(f) # get the raw function also for OOP methods if not function.__module__: # At least macros defined in the REPL have `__module__=None`. return function.__qualname__ return f"{function.__module__}.{function.__qualname__}" +def _name_of_1st_positional_parameter(f): + function, _ = getfunc(f) # get the raw function also for OOP methods + parameters = inspect.signature(function).parameters + poskinds = set((inspect.Parameter.POSITIONAL_ONLY, + inspect.Parameter.POSITIONAL_OR_KEYWORD)) + for param in parameters.values(): + if param.kind in poskinds: + return param.name + return None + +def _list_multimethods(dispatcher, self_or_cls=None): + """List multimethods currently registered to a given dispatcher. + + `self_or_cls`: If `dispatcher` is installed on an OOP instance method + or on an OOP `@classmethod`, set this to perform MRO + lookups to find linked dispatchers. + """ + # For regular functions, ours is the only registry we need to look at: + relevant_registries = [reversed(dispatcher._method_registry)] + + # But if this dispatcher is installed on an OOP method, we must + # look up generic function methods also in the class's MRO. + # + # For *static methods* MRO is not supported. Basically, one of + # the roles of `cls` or `self` is to define the MRO; a static + # method doesn't have that. + # + # See discussions on interaction between `@staticmethod` and `super` in Python: + # https://bugs.python.org/issue31118 + # https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879 + if self_or_cls: + if isinstance(self_or_cls, type): + cls = self_or_cls + elif hasattr(self_or_cls, "__class__"): + cls = self_or_cls.__class__ + else: + assert False + + for base in cls.__mro__[1:]: # skip the class itself in the MRO + if hasattr(base, dispatcher.__name__): # does this particular super have f? + base_oop_method = getattr(base, dispatcher.__name__) + base_raw_function, _ = getfunc(base_oop_method) + if isgeneric(base_raw_function): # it's @generic + base_registry = getattr(base_raw_function, "_method_registry") + relevant_registries.append(reversed(base_registry)) + + return list(chain.from_iterable(relevant_registries)) + +# input format is an item returned by `_list_multimethods` +def _format_method(method): # Taking a page from Julia and some artistic liberty here. + thecallable, type_signature = method + # Our `type_signature` is based on `typing.get_type_hints`, + # but for the error message, we need something that formats + # like source code. Hence use `inspect.signature`. + thesignature = inspect.signature(thecallable) + function, _ = getfunc(thecallable) + filename = inspect.getsourcefile(function) + source, firstlineno = inspect.getsourcelines(function) + return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" + +# `type_signature`: in the format returned by `typing.get_type_hints`. +# `bound_arguments`: see `unpythonic.arity.resolve_bindings`. +def _match_argument_types(type_signature, bound_arguments): + for parameter, value in bound_arguments.arguments.items(): + assert parameter in type_signature # resolve_bindings should already TypeError when not. + expected_type = type_signature[parameter] + if not isoftype(value, expected_type): + return False + return True + +def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): + # TODO: It would be nice to show the type signature of the args actually given, + # TODO: but in the general case this is difficult. We can't just `type(x)`, since + # TODO: the signature may specify something like `Sequence[int]`. Knowing a `list` + # TODO: was passed doesn't help debug that it was `Sequence[str]` when a `Sequence[int]` + # TODO: was expected. The actual value at least implicitly contains the type information. + # + # TODO: Compute closest candidates, like Julia does? (see `methods`, `MethodError` in Julia) + # TODO: (If we do that, we must also modify the dispatch logic.) + args_list = [repr(x) for x in args] + args_str = ", ".join(args_list) + sep = ", " if args and kwargs else "" + kws_list = [f"{k}={repr(v)}" for k, v in kwargs.items()] + kws_str = ", ".join(kws_list) + methods_list = [f" {_format_method(x)}" for x in candidates] + methods_str = "\n".join(methods_list) + msg = (f"No multiple-dispatch match for the call {dispatcher.__qualname__}({args_str}{sep}{kws_str}).\n" + f"Multimethods for @{isgeneric(dispatcher)} {repr(dispatcher.__qualname__)} (most recent match attempt last):\n{methods_str}") + raise TypeError(msg) + def _register_generic(fullname, f): """Register a method for a generic function. @@ -254,111 +455,45 @@ def _register_generic(fullname, f): Return value is the dispatcher that replaces the original function. """ - # HACK for cls/self analysis - def name_of_1st_positional_parameter(f): - function, _ = getfunc(f) - params = inspect.signature(function).parameters - poskinds = set((inspect.Parameter.POSITIONAL_ONLY, - inspect.Parameter.POSITIONAL_OR_KEYWORD)) - for param in params.values(): - if param.kind in poskinds: - return param.name - return None - if fullname not in _dispatcher_registry: # Create the dispatcher. This will replace the original f. + # + # We want @wraps to preserve docstrings, so the decorator must be a function, not a class. + # https://stackoverflow.com/questions/6394511/python-functools-wraps-equivalent-for-classes + # https://stackoverflow.com/questions/25973376/functools-update-wrapper-doesnt-work-properly#25973438 + # + # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. + # + # Here we're operating at the wrong abstraction level for that, + # since we see just bare functions. In the OOP case, the dispatcher + # is installed on the raw function before it becomes a bound method. + # (That in itself is just as it should be.) + first_param_name = _name_of_1st_positional_parameter(f) + f_is_most_likely_an_oop_method = first_param_name in self_parameter_names @wraps(f) def dispatcher(*args, **kwargs): - # `signature` comes from typing.get_type_hints. - # `bound_arguments` is populated in the surrounding scope below. - def match_argument_types(type_signature): - for parameter, value in bound_arguments.arguments.items(): - assert parameter in type_signature # resolve_bindings should already TypeError when not. - expected_type = type_signature[parameter] - if not isoftype(value, expected_type): - return False - return True + # Let's see if we might have been passed a `self`/`cls` parameter, + # and if so, get its value. (Recall that in Python, it is always + # the first positional parameter.) + if f_is_most_likely_an_oop_method: + if len(args) < 1: # pragma: no cover, shouldn't happen. + raise TypeError(f"MRO lookup failed: no value provided for self-like parameter {repr(first_param_name)} when calling generic-function OOP method {fullname}") + self_or_cls = args[0] + else: + self_or_cls = None # Dispatch. - def methods(): - # For regular functions, ours is the only registry we need to look at: - relevant_registries = [reversed(dispatcher._method_registry)] - - # But if this dispatcher is installed on an OOP method, we must - # look up generic function methods also in the class's MRO. - # - # For *static methods* MRO is not supported. Basically, one of - # the roles of `cls` or `self` is to define the MRO; a static - # method doesn't have that. - # - # See discussions on interaction between `@staticmethod` and `super` in Python: - # https://bugs.python.org/issue31118 - # https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879 - # - # TODO/FIXME: Not possible to detect self/cls parameters correctly. - # Here we're operating at the wrong abstraction level for that, - # since we see just bare functions. - # - # Let's see if we might have a self/cls parameter, and if so, get its value. - first_param_name = name_of_1st_positional_parameter(f) - if first_param_name in self_parameter_names: - if len(args) < 1: # pragma: no cover, shouldn't happen. - raise TypeError(f"MRO lookup failed: no value provided for self-like parameter {repr(first_param_name)} when calling generic-function OOP method {fullname}") - first_arg_value = args[0] - dynamic_instance = first_arg_value # self/cls - theclass = None - if isinstance(dynamic_instance, type): # cls - theclass = dynamic_instance - elif hasattr(dynamic_instance, "__class__"): # self - theclass = dynamic_instance.__class__ - if theclass is not None: # ignore false positives when possible - for base in theclass.__mro__[1:]: # skip the class itself in the MRO - if hasattr(base, f.__name__): # does this particular super have f? - base_oop_method = getattr(base, f.__name__) - base_raw_function, _ = getfunc(base_oop_method) - if isgeneric(base_raw_function): # it's @generic - base_registry = getattr(base_raw_function, "_method_registry") - relevant_registries.append(reversed(base_registry)) - - return chain.from_iterable(relevant_registries) - for method, type_signature in methods(): + multimethods = _list_multimethods(dispatcher, self_or_cls) + for thecallable, type_signature in multimethods: try: - bound_arguments = resolve_bindings(method, *args, **kwargs) + bound_arguments = resolve_bindings(thecallable, *args, **kwargs) except TypeError: # arity mismatch, so this method can't be the one the call is looking for. continue - if match_argument_types(type_signature): - return method(*args, **kwargs) + if _match_argument_types(type_signature, bound_arguments): + return thecallable(*args, **kwargs) # No match, report error. - # - # TODO: It would be nice to show the type signature of the args actually given, - # TODO: but in the general case this is difficult. We can't just `type(x)`, since - # TODO: the signature may specify something like `Sequence[int]`. Knowing a `list` - # TODO: was passed doesn't help debug that it was `Sequence[str]` when a `Sequence[int]` - # TODO: was expected. The actual value at least implicitly contains the type information. - # - # TODO: Compute closest candidates, like Julia does? (see methods, MethodError) - a = [repr(a) for a in args] - sep = ", " if args and kwargs else "" - kw = [f"{k}={repr(v)}" for k, v in kwargs.items()] - def format_method(method): # Taking a page from Julia and some artistic liberty here. - thecallable, type_signature = method - # Our `type_signature` is based on `typing.get_type_hints`, - # but for the error message, we need something that formats - # like source code. Hence use `inspect.signature`. - thesignature = inspect.signature(thecallable) - function, _ = getfunc(thecallable) - filename = inspect.getsourcefile(function) - source, firstlineno = inspect.getsourcelines(function) - return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" - methods_str = [f" {format_method(x)}" for x in methods()] - candidates = "\n".join(methods_str) - function, _ = getfunc(f) - args_str = ", ".join(a) - kws_str = ", ".join(kw) - msg = (f"No multiple-dispatch match for the call {function.__qualname__}({args_str}{sep}{kws_str}).\n" - f"Multimethods for {repr(function.__qualname__)} (most recent match attempt last):\n{candidates}") - raise TypeError(msg) + _raise_multiple_dispatch_error(dispatcher, args, kwargs, candidates=multimethods) dispatcher._method_registry = [] def register(thecallable): @@ -375,20 +510,20 @@ def register(thecallable): # that have no type annotation, but that would likely be a footgun. # So we require a type annotation for each parameter. # - # One exception: the self/cls parameter of OOP instance methods and + # One exception: the `self`/`cls` parameter of OOP instance methods and # class methods is not meaningful for dispatching, and we don't # have a runtime value to auto-populate its expected type when the # definition runs. So we set it to `typing.Any` in the method's # expected type signature, which makes the dispatcher ignore it. - function, kind = getfunc(thecallable) - params = inspect.signature(function).parameters - params_names = [p.name for p in params.values()] + function, _ = getfunc(thecallable) + parameters = inspect.signature(function).parameters + parameter_names = [p.name for p in parameters.values()] type_signature = typing.get_type_hints(function) - # In the type signature, auto-`Any` the self/cls parameter, if any. + # In the type signature, auto-`Any` the `self`/`cls` parameter, if any. # - # TODO/FIXME: Not possible to detect self/cls parameters correctly. + # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. # # The `@generic` decorator runs while the class body is being # evaluated. In that context, an instance method looks just like a @@ -402,14 +537,14 @@ def register(thecallable): # when they appear the first position, though **Python itself # doesn't do that**. For any crazy person not following Python # naming conventions, our approach won't work. - if len(params_names) >= 1 and params_names[0] in self_parameter_names: + if len(parameter_names) >= 1 and parameter_names[0] in self_parameter_names: # In Python 3.6+, `dict` preserves insertion order. Make sure # the `self` parameter appears first, for clearer error messages # when no matching method is found. - type_signature = {params_names[0]: typing.Any, **type_signature} + type_signature = {parameter_names[0]: typing.Any, **type_signature} - if not all(name in type_signature for name in params_names): - failures = [name for name in params_names if name not in type_signature] + if not all(name in type_signature for name in parameter_names): + failures = [name for name in parameter_names if name not in type_signature] plural = "s" if len(failures) > 1 else "" wrapped_list = [f"'{x}'" for x in failures] wrapped_str = ", ".join(wrapped_list) @@ -419,38 +554,10 @@ def register(thecallable): dispatcher._method_registry.append((thecallable, type_signature)) return dispatcher # Replace the callable with the dispatcher for this generic function. - dispatcher._register = register # save it for use by us later + dispatcher._register = register # save it for use by us to register methods to this dispatcher _dispatcher_registry[fullname] = dispatcher + dispatcher = _dispatcher_registry[fullname] if isgeneric(dispatcher) == "typed": # co-operation with @typed, below raise TypeError("@typed: cannot register additional methods.") return dispatcher._register(f) - -@register_decorator(priority=98) -def typed(f): - """Decorator. Restrict allowed argument types to one combination only. - - This can be used to eliminate `isinstance` boilerplate code in the - function body, by allowing the types (for dynamic, run-time checking) - to be specified with a very compact syntax - namely, type annotations. - - Also, unlike a basic `isinstance` check, this allows using features - from the `typing` stdlib module in the type specifications. - - After a `@typed` function has been created, no more methods can be - attached to it. - - `@typed` works with `curry`, because the function has only one call - signature, as usual. - - **CAUTION**: - - If used with `curry`, argument type errors will only be detected when - `curry` triggers the actual call. To fix this, `curry` would need to - perform some more introspection on the callable, and to actually know - about this dispatch system. It's not high on the priority list. - """ - # TODO: Fix the epic fail at fail-fast, and update the corresponding test. - s = generic(f) - del s._register # remove the ability to register more methods - return s From f04a438a444bb106772f2504008d0e3ed2edc7fe Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 14 May 2021 12:10:09 +0300 Subject: [PATCH 282/832] format_methods: handle also case of empty multimethod list --- unpythonic/dispatch.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 01957034..87d18425 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -284,9 +284,13 @@ def format_methods(f): This calls `list_methods`, which see. """ function, _ = getfunc(f) - methods_str = [f" {_format_method(x)}" for x in list_methods(f)] - methods = "\n".join(methods_str) - return f"Multimethods for @{isgeneric(f)} {repr(function.__qualname__)}:\n{methods}" + multimethods = list_methods(f) + if multimethods: + methods_list = [f" {_format_method(x)}" for x in multimethods] + methods_str = "\n".join(methods_list) + else: # pragma: no cover, in practice should always have one method. + methods_str = " " + return f"Multimethods for @{isgeneric(f)} {repr(function.__qualname__)}:\n{methods_str}" def list_methods(f): """Return a list of the multimethods currently registered to `f`. From 0a2e6c081155ea25047ed0286fe65c1fd36b030b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 00:27:44 +0300 Subject: [PATCH 283/832] fix old todo --- unpythonic/tests/test_dispatch.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 1fe708f6..305a8328 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -17,8 +17,9 @@ def zorblify(x: str, y: int): # noqa: F811, registered as a method of the same @generic def zorblify(x: str, y: float): # noqa: F811 return f"{x[::-1]} {y}" - -# TODO: def zorblify(x: int, *args: typing.Sequence[str]): +@generic +def zorblify(x: int, *args: typing.Sequence[str]): # noqa: F811 + return f"{x}, {', '.join(args)}" # @generic can also be used to simplify argument handling code in functions # where the role of an argument in a particular position changes depending on @@ -83,6 +84,7 @@ def runtests(): test[zorblify(y=8, x=17) == 42] test[zorblify("tac", 1.0) == "cat 1.0"] test[zorblify(y=1.0, x="tac") == "cat 1.0"] + test[zorblify(23, "cat", "meow") == "23, cat, meow"] test_raises[TypeError, zorblify(1.0, 2.0)] # there's no zorblify(float, float) From 820f480450ff2987b7bb244c3766ec50dc041e24 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 00:27:54 +0300 Subject: [PATCH 284/832] fix test, now the comment is actually correct --- unpythonic/tests/test_dispatch.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 305a8328..794fc536 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -42,8 +42,11 @@ def _example_impl(start, step, stop): # no @generic! # shorter, same effect @generic -def example2(start: int, stop: int): - return example2(start, 1, stop) # just call the method that has the implementation +def example2(stop: int): + return example2(0, 1, stop) # just call the method that has the implementation +@generic +def example2(start: int, stop: int): # noqa: F811 + return example2(start, 1, stop) @generic def example2(start: int, step: int, stop: int): # noqa: F811 return start, step, stop @@ -92,6 +95,7 @@ def runtests(): test[example(2, 10) == (2, 1, 10)] test[example(2, 3, 10) == (2, 3, 10)] + test[example2(5) == (0, 1, 5)] test[example2(1, 5) == (1, 1, 5)] test[example2(1, 1, 5) == (1, 1, 5)] test[example2(1, 2, 5) == (1, 2, 5)] From 6536694a8ac6863921c10b54199a514f3d2cae29 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 00:28:07 +0300 Subject: [PATCH 285/832] add tests for format_methods --- unpythonic/tests/test_dispatch.py | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 794fc536..c427f911 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -5,7 +5,7 @@ import typing from ..fun import curry -from ..dispatch import generic, augment, typed +from ..dispatch import generic, augment, typed, format_methods @generic def zorblify(x: int, y: int): @@ -238,6 +238,23 @@ def instmeth(self, x: float): test[jack("foo") == "foo"] test_raises[TypeError, jack(3.14)] # jack only accepts int or str + with testset("list_methods"): + def check_formatted_multimethods(result, expected): + result_list = result.split("\n") + human_readable_header, *multimethod_descriptions = result_list + multimethod_descriptions = [x.strip() for x in multimethod_descriptions] + test[the[len(multimethod_descriptions)] == the[len(expected)]] + for r, e in zip(multimethod_descriptions, expected): + test[the[r].startswith(the[e])] + # @generic + check_formatted_multimethods(format_methods(example2), + ["example2(start: int, step: int, stop: int)", + "example2(start: int, stop: int)", + "example2(stop: int)"]) + # @typed + check_formatted_multimethods(format_methods(blubnify), + ["blubnify(x: int, y: float)"]) + with testset("error cases"): with test_raises[TypeError, "@typed should only accept a single method"]: @typed From dffc3dcf857e58a3c2cc143a1f012618f2bf44f6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 01:01:19 +0300 Subject: [PATCH 286/832] test the right thing --- unpythonic/tests/test_dispatch.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index c427f911..0e50a785 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -232,7 +232,10 @@ def instmeth(self, x: float): with testset("@typed"): test[blubnify(2, 21.0) == 42] test_raises[TypeError, blubnify(2, 3)] # blubnify only accepts (int, float) - test[not hasattr(blubnify, "register")] # and no more methods can be registered on it + with test_raises[TypeError, "should not be able to add more multimethods to a @typed function"]: + @augment(blubnify) + def blubnify2(x: float, y: float): + pass test[jack(42) == 42] test[jack("foo") == "foo"] From e94917dbc66740d8bd38c5d2dc46868bf531aade Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 01:01:26 +0300 Subject: [PATCH 287/832] terminology: multimethod --- unpythonic/tests/test_dispatch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 0e50a785..27d7cc2c 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -11,7 +11,7 @@ def zorblify(x: int, y: int): return 2 * x + y @generic -def zorblify(x: str, y: int): # noqa: F811, registered as a method of the same generic function. +def zorblify(x: str, y: int): # noqa: F811, registered as a multimethod of the same generic function. # Because dispatching occurs on both arguments, this method is not reached by the tests. fail["this method should not be reached by the tests"] # pragma: no cover @generic @@ -43,7 +43,7 @@ def _example_impl(start, step, stop): # no @generic! # shorter, same effect @generic def example2(stop: int): - return example2(0, 1, stop) # just call the method that has the implementation + return example2(0, 1, stop) # just call the multimethod that has the implementation @generic def example2(start: int, stop: int): # noqa: F811 return example2(start, 1, stop) From d166ff7b297a6ba202b92d6f46e5d18425a3e8d1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 01:04:00 +0300 Subject: [PATCH 288/832] fix test on Python 3.6 --- unpythonic/tests/test_dispatch.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 27d7cc2c..7f22ba9c 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -243,11 +243,15 @@ def blubnify2(x: float, y: float): with testset("list_methods"): def check_formatted_multimethods(result, expected): + def _remove_space_before_typehint(string): # Python 3.6 doesn't print a space there + return string.replace(": ", ":") result_list = result.split("\n") human_readable_header, *multimethod_descriptions = result_list multimethod_descriptions = [x.strip() for x in multimethod_descriptions] test[the[len(multimethod_descriptions)] == the[len(expected)]] for r, e in zip(multimethod_descriptions, expected): + r = _remove_space_before_typehint(r) + e = _remove_space_before_typehint(e) test[the[r].startswith(the[e])] # @generic check_formatted_multimethods(format_methods(example2), From 5dc451d553563892b11c96dcaa416c01fca2ffbc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:14:20 +0300 Subject: [PATCH 289/832] terminology: "multimethod", "generic function" --- CHANGELOG.md | 7 ++++-- doc/features.md | 60 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 45 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6eb91dd9..b7a693d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,9 +80,12 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `with namedlambda` now understands the walrus operator, too. In the construct `f := lambda ...: ...`, the lambda will get the name `f`. (Python 3.8 and later.) - `with namedlambda` now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined. - Multiple-dispatch system `unpythonic.dispatch`: - - Add decorator `@augment`: add methods to a generic function defined elsewhere. + - Use consistent terminology: + - The function that supports multiple call signatures is a *generic function*. + - Its individual implementations are *multimethods*. + - Add decorator `@augment`: add a multimethod to a generic function defined elsewhere. - Add function `isgeneric` to detect whether a callable has been declared `@generic`. - - Add function `methods`: display a list of methods of a generic function. + - Add function `methods`: display a list of multimethods of a generic function. - It is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. diff --git a/doc/features.md b/doc/features.md index 6fd91bc0..0fbdb377 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3003,29 +3003,38 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of **Added in v0.14.2**. -**Changed in v0.14.3**. *The multiple-dispatch decorator `@generic` no longer takes a master definition. Methods are registered directly with `@generic`; the first method definition implicitly creates the generic function.* +**Changed in v0.14.3**. *The multiple-dispatch decorator `@generic` no longer takes a master definition. Multimethods are registered directly with `@generic`; the first method definition implicitly creates the generic function.* **Changed in v0.14.3**. *The `@generic` and `@typed` decorators can now decorate also instance methods, class methods and static methods (beside regular functions, as previously in 0.14.2).* **Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Starting with this release, they receive the same semantic-versioning guarantees as the rest of `unpythonic`.* -*Added the `@augment` parametric decorator that can register a new method on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* +*Added the `@augment` parametric decorator that can register a new multimethod on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* -*Added the function `methods`, which displays a list of methods of a generic function.* +*Added the function `methods`, which displays a list of multimethods of a generic function.* *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* -The ``generic`` decorator allows creating multiple-dispatch generic functions (a.k.a. multimethods) with type annotation syntax. - -We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. +The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. For what kind of things can be done with this, see particularly the [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) example in [`unpythonic.tests.test_dispatch`](../unpythonic/tests/test_dispatch.py). +**NOTE**: This was inspired by the [multi-methods of CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) (the Common Lisp Object System), and the [generic functions of Julia](https://docs.julialang.org/en/v1/manual/methods/). + +In `unpythonic`, we define the terms as follows: + + - The function that supports multiple call signatures is a *generic function*. + - Its individual implementations are *multimethods*. + #### ``generic``: multiple dispatch with type annotation syntax -The ``generic`` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. +The ``generic`` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. This not only kills boilerplate, but makes the dispatch extensible, since the dispatcher lives outside the original function definition. There is no need to monkey-patch the original to add a new case. -If several methods of the same generic function match the arguments given, the most recently registered method wins. (**CAUTION**: This is different from Julia, where the most specific method wins. Doing that requires a more careful type analysis than what we have here.) +If several multimethods of the same generic function match the arguments given, the most recently registered multimethod wins. + +**CAUTION**: The winning multimethod is chosen differently from Julia, where the most specific multimethod wins. Doing that requires a more careful type analysis than what we have here. + +**CAUTION**: `@generic` does not currently work with `curry`. Adding support requires changes to the already complex logic in `curry`; it is not high on the priority list. The details are best explained by example: @@ -3033,10 +3042,10 @@ The details are best explained by example: import typing from unpythonic import generic -@generic # The first definition creates the generic function, and registers the first method. +@generic # The first definition creates the generic function, and registers the first multimethod. def zorblify(x: int, y: int): return "int, int" -@generic # noqa: F811, registered as a method of the same generic function. +@generic # noqa: F811, registered as a multimethod of the same generic function. def zorblify(x: str, y: int): return "str, int" @generic # noqa: F811 @@ -3044,7 +3053,7 @@ def zorblify(x: str, y: float): return "str, float" # Then we just call our function as usual. -# Note all arguments participate in dispatching (i.e. in choosing which method gets called). +# Note all arguments participate in dispatching (i.e. in choosing which multimethod gets called). assert zorblify(2, 3) == "int, int" assert zorblify("cat", 3) == "str, int" assert zorblify("cat", 3.14) == "str, float" @@ -3085,12 +3094,21 @@ assert gargle(1, 2, 3, 4, 5) == "int" assert gargle(2.71828, 3.14159) == "float" assert gargle(42, 6.022e23, "hello") == "int, float, str" assert gargle(1, 2, 3) == "int" # as many as in the [int, float, str] case. Still resolves correctly. + +# v0.15.0: dispatching on a homogeneous type inside **kwargs is also supported, via `typing.Dict` +@generic +def kittify(**kwargs: typing.Dict[str, int]): # all kwargs are ints + return "int" +@generic +def kittify(**kwargs: typing.Dict[str, float]): # all kwargs are floats # noqa: F811 + return "float" + +assert kittify(x=1, y=2) == "int" +assert kittify(x=1.0, y=2.0) == "float" ``` See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the ``typing`` stdlib module are supported, see ``isoftype`` below. -Inspired by the [multi-methods of CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) (the Common Lisp Object System), and the [generic functions of Julia](https://docs.julialang.org/en/v1/manual/methods/). - ##### ``@generic`` and OOP As of version 0.14.3, `@generic` and `@typed` can decorate instance methods, class methods and static methods (beside regular functions as in 0.14.2). @@ -3102,23 +3120,25 @@ When using both `@generic` or `@typed` and OOP: - Beside appearing as the first positional-or-keyword parameter, the self-like parameter **must be named** one of `self`, `this`, `cls`, or `klass` to be detected by the ignore mechanism. This limitation is due to implementation reasons; while a class body is being evaluated, the context needed to distinguish a method (OOP sense) from a regular function is not yet present. - **OOP inheritance**. - - When `@generic` is installed on an OOP method (instance method, or `@classmethod`), then at call time, classes are tried in [MRO](https://en.wikipedia.org/wiki/C3_linearization) order. All generic-function methods of the OOP method defined in the class currently being looked up are tested for matches first, before moving on to the next class in the MRO. (This has subtle consequences, related to in which class in the hierarchy the various generic-function methods for a particular OOP method are defined.) + - When `@generic` is installed on a method (instance method, or `@classmethod`), then at call time, classes are tried in [MRO](https://en.wikipedia.org/wiki/C3_linearization) order. All multimethods of the method defined in the class currently being looked up are tested for matches first, before moving on to the next class in the MRO. This has subtle consequences, related to in which class in the hierarchy the various multimethods for a particular method are defined. - To work with OOP inheritance, `@generic` must be the outermost decorator (except `@classmethod` or `@staticmethod`, which are essentially compiler annotations). - However, when installed on a `@staticmethod`, the `@generic` decorator does not support MRO lookup, because that would make no sense. See discussions on interaction between `@staticmethod` and `super` in Python: [[1]](https://bugs.python.org/issue31118) [[2]](https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879). ##### Notes -*Terminology*: in both CLOS and in Julia, *function* is the generic entity, while *method* refers to its specialization to a particular combination of argument types. Note that *no object instance or class is needed*. Contrast with the classical OOP sense of *method*, i.e. a function that is associated with an object instance or class, with single dispatch based on the class (or in exotic cases, such as monkey-patched instances, on the instance). +In both CLOS and in Julia, *function* is the generic entity, while *method* refers to its specialization to a particular combination of argument types. Note that *no object instance or class is needed*. Contrast with the classical OOP sense of *method*, i.e. a function that is associated with an object instance or class, with single dispatch based on the class (or in exotic cases, such as monkey-patched instances, on the instance). Based on my own initial experiments with this feature, the machinery itself works well enough, but to really shine - just like resumable exceptions - multiple dispatch needs to be used everywhere, throughout the language's ecosystem. Python obviously doesn't do that. -**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the method table is global state! +The machinery itself is also missing some advanced features, such as matching the most specific multimethod candidate instead of the most recently defined one; an `issubclass` equivalent that understands `typing` type specifications; and a mechanism to remove previously declared multimethods. + +**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the multiple-dispatch table is global state! #### ``typed``: add run-time type checks with type annotation syntax -The ``typed`` decorator creates a one-method pony, which automatically enforces its argument types. Just like with ``generic``, the type specification may use features from the `typing` stdlib module. +The ``typed`` decorator creates a one-multimethod pony, which automatically enforces its argument types. Just like with ``generic``, the type specification may use features from the `typing` stdlib module. ```python import typing @@ -3134,7 +3154,7 @@ def jack(x: typing.Union[int, str]): assert blubnify(2, 21.0) == 42 blubnify(2, 3) # TypeError -assert not hasattr(blubnify, "register") # no more methods can be registered on this function +assert not hasattr(blubnify, "register") # no more multimethods can be registered on this function assert jack(42) == 42 assert jack("foo") == "foo" @@ -3215,9 +3235,9 @@ See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. **CAUTION**: Callables are just checked for being callable; no further analysis is done. Type-checking callables properly requires a much more complex type checker. -**CAUTION**: The `isoftype` function is one big hack. As of Python 3.6, there is no consistent way to handle a type specification at run time. We must access some private attributes of the ``typing`` meta-utilities, because that seems to be the only way to get what we need to do this. +**CAUTION**: The `isoftype` function is one big hack. In Python 3.6 through 3.9, there is no consistent way to handle a type specification at run time. We must access some private attributes of the ``typing`` meta-utilities, because that seems to be the only way to get what we need to do this. -If you need a run-time type checker for serious general use, consider the [`typeguard`](https://github.com/agronholm/typeguard) library, which focuses on that. +For a similar tool for run-time type-checking, see also the [`typeguard`](https://github.com/agronholm/typeguard) library. ## Exception tools From 0e16e107189867d2ffd74d3ce418196c2686051b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:14:32 +0300 Subject: [PATCH 290/832] fix stray whitespace --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 0fbdb377..f2fbe8ba 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2865,7 +2865,7 @@ The implementation is based on the List monad, and a bastardized variant of do-n **Changed in v0.15.0.** *Functions `resignal_in` and `resignal` added; these perform the same job for conditions as `reraise_in` and `reraise` do for exceptions, that is, they allow you to map library exception types to semantically appropriate application exception types, with minimum boilerplate.* -*Upon an unhandled signal, `signal` now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report.* +*Upon an unhandled signal, `signal` now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report.* *The error-handling protocol that was used to send a signal is now available for inspection in the `__protocol__` attribute of the condition instance. It is the callable that sent the signal, such as `signal`, `error`, `cerror` or `warn`. It is the responsibility of each error-handling protocol (except the fundamental `signal` itself) to pass its own function to `signal` as the `protocol` argument; if not given, `protocol` defaults to `signal`. The protocol information is used by the `resignal` mechanism.* From 5062389dbefe1eae6b7061156cae0190175810f8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:14:37 +0300 Subject: [PATCH 291/832] improve comment in example --- doc/features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/features.md b/doc/features.md index f2fbe8ba..e79030c0 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3365,6 +3365,7 @@ class ApplicationException(Exception): pass # reraise_in: expr form +# The mapping is {in0: out0, ...} try: # reraise_in(thunk, mapping) reraise_in(lambda: raisef(LibraryException), @@ -3388,6 +3389,7 @@ except ApplicationException: print("all ok!") # reraise: block form +# The mapping is {in0: out0, ...} try: with reraise({LibraryException: ApplicationException}): raise LibraryException From 6c0937c349abb0225d1d1a21133c642c73492d12 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:14:52 +0300 Subject: [PATCH 292/832] Spelling: multimethod. Add related link to the expression problem. --- doc/readings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/readings.md b/doc/readings.md index b88e943c..80200ac0 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -36,7 +36,7 @@ The common denominator is programming. Some relate to language design, some to c - [William R. Cook, OOPSLA 2009: On Understanding Data Abstraction, Revisited](https://www.cs.utexas.edu/~wcook/Drafts/2009/essay.pdf). - This is a nice paper illustrating the difference between *abstract data types* and *objects*. - In section 4.3: *"In the 1970s [...] Reynolds noticed that abstract data types facilitate adding new operations, while 'procedural data values' (objects) facilitate adding new representations. Since then, this duality has been independently discovered at least three times [18, 14, 33]."* Then: *"The extensibility problem has been solved in numerous ways, and it still inspires new work on extensibility of data abstractions [48, 15]. Multi-methods are another approach to this problem [11]."* - - Multi-methods (as in multiple dispatch in CLOS or in Julia) seem nice, in that they don't enfore a particular way to slice the operation/representation matrix. Instead, one fills in individual cells as desired. + - Multimethods (as in multiple dispatch in CLOS or in Julia) seem nice, in that they don't enfore a particular way to slice the operation/representation matrix. Instead, one fills in individual cells as desired. It solves [the expression problem](https://en.wikipedia.org/wiki/Expression_problem). - In section 5.4, on Smalltalk: *"One conclusion you could draw from this analysis is that the untyped λ-calculus was the first object-oriented language."* - In section 6: *"Academic computer science has generally not accepted the fact that there is another form of data abstraction besides abstract data types. Hence the textbooks give the classic stack ADT and then say 'objects are another way to implement abstract data types'. [...] Some textbooks do better than others. Louden [38] and Mitchell [43] have the only books I found that describe the difference between objects and ADTs, although Mitchell does not go so far as to say that objects are a distinct kind of data abstraction."* From 5eb27d86826ce1ab11f59113f0854079f4b38f85 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:15:14 +0300 Subject: [PATCH 293/832] terminology: multimethod --- unpythonic/dispatch.py | 257 ++++++++++++++++++++++++----------------- 1 file changed, 151 insertions(+), 106 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 87d18425..4f459193 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -1,9 +1,22 @@ # -*- coding: utf-8; -*- -"""A multiple-dispatch decorator for Python. +"""A multiple-dispatch system (a.k.a. multimethods) for Python. + +Terminology: + + - The function that supports multiple call signatures is a *generic function*. + - Its individual implementations are *multimethods*. + +We use the term *multimethod* to distinguish them from the usual sense of *method* +in Python, and because this is multiple dispatch. Somewhat like `functools.singledispatch`, but for multiple dispatch. https://docs.python.org/3/library/functools.html#functools.singledispatch + +Somewhat like `typing.overload`, but for run-time use, not static type-checking. +Here the implementations are given in the multimethod bodies. + + https://docs.python.org/3/library/typing.html#typing.overload """ __all__ = ["isgeneric", "generic", "augment", "typed", @@ -38,38 +51,62 @@ # """ def isgeneric(f): - """Return whether the callable `f` is a multiple-dispatch generic function. + """Return whether the callable `f` is a generic function. - If `f` was declared `@generic`, return the string `"generic"`. - If `f` was declared `@typed`, return the string `"typed"`. - Otherwise, return `False`. + If `f` was declared `@generic` (which see), return the string `"generic"`. + If `f` was declared `@typed` (which see), return the string `"typed"`. + Else return `False`. """ - if not hasattr(f, "_method_registry"): - return False - if hasattr(f, "_register"): - return "generic" - return "typed" + if hasattr(f, "_method_registry"): + if hasattr(f, "_register"): + return "generic" + return "typed" + return False # TODO: We essentially need the fullname because the second and further invocations # TODO: of `@generic`, for the same generic function, receive an entirely different -# TODO: run-time object - the new method. There is no way to know which existing +# TODO: run-time object - the new multimethod. There is no way to know which existing # TODO: dispatcher to connect that to, other than having a registry that maps the # TODO: fullname of each already-existing generic function to its dispatcher object. @register_decorator(priority=98) def generic(f): """Decorator. Make `f` a generic function (in the sense of CLOS or Julia). + Multiple dispatch solves *the expression problem*: + https://en.wikipedia.org/wiki/Expression_problem + + Practical use cases: + + - Eliminate `if`/`elif`/`elif`... blocks that switch by `isinstance` on + function arguments, and then raise `TypeError` in the final `else`, + by having a central implementation for this machinery. + + This not only kills boilerplate, but makes the dispatch extensible, + since the dispatcher lives outside the original function definition. + There is no need to monkey-patch the original to add a new case. + + See `@augment`. + + - Dispatch on an extensible hierarchy of abstract features (called *traits*) + that is separate from the concrete type hierarchy, using the *holy traits* + pattern. For example, "behaves like a number" can be a trait. + + See `unpythonic/tests/test_dispatch.py` for an example. + + - Functions like the builtin `range`, where the *role* of an argument in a + particular position depends on the *number of* arguments passed in the call. + With `@generic`, each case can have its parameters named descriptively. + **How to use**: - Just make several function definitions, with the same name in the same - lexical scope, one for each call signature you want to support, and - decorate each of them with `@generic`. Here *signature* refers to specific - combinations of argument types and/or different shapes for the argument - list. + Make several function definitions, with the same name in the same lexical + scope, one for each call signature you want to support, and decorate each of + them with `@generic`. Here *signature* refers to specific combinations of + argument types and/or different shapes for the argument list. The first definition implicitly creates the generic function (like in Julia). All of the definitions, including the first one, become registered - as *methods* of the *generic function*. + as *multimethods* of the *generic function*. The return value of `generic` is the multiple-dispatch dispatcher for the generic function that was created or modified. @@ -87,7 +124,7 @@ def generic(f): @generic def example(x: int, y: int): ... # implementation here - @generic # noqa: F811, registered as a method of the same generic function. + @generic # noqa: F811, registered as a multimethod of the same generic function. def example(x: str, y: int): ... # implementation here @generic # noqa: F811 @@ -106,40 +143,36 @@ def example(): then, nested lambdas are not supported. Be careful that if you later rebind a variable that refers to a generic - function; that will not remove previously existing method definitions. + function, that will not remove previously existing method definitions. If you later rebind the same name again, pointing to a new generic function, it will suddenly gain all of the methods of the previous function that had the same fullname. - **Method lookup**: + As of v0.15.0, multimethods cannot be unregistered. - Each method definition must specify type hints **on all of its parameters** - except `**kwargs` (if it has one). Then, at call time, the types of **all** - arguments (except any bound to `**kwargs`), as well as the number of - arguments, are automatically used for *dispatching*, i.e. choosing which - method to call. In other words, multiple parameters participate in - dispatching, thus the term *multiple dispatch*. + **Multimethod lookup**: - **Varargs are supported**. To have the contents of `*args` participate in - dispatching, annotate the parameter as `*args: typing.Tuple[...]`. For the - `...` part, see the documentation of the `typing` module. Both homogeneous - and heterogeneous tuples are supported. + Each method definition must specify type hints **on all of its parameters**. + Then, at call time, the types of **all** arguments, as well as the number + of arguments, are automatically used for *dispatching*, i.e. choosing which + implementation to call. In other words, multiple parameters participate in + dispatching, hence *multiple dispatch*. - **The first method that matches wins, in most-recently-registered order.** - (This is unlike in Julia, which matches the most specific applicable method.) + **Varargs are supported**. Vararg type hint examples:: - In other words, later definitions override earlier ones. So specify the - implementation with the most generic types first, and then move on to the - more specific ones. The mnemonic is, "the function is generally defined - like this, except if the arguments match these particular types..." + - `*args: typing.Tuple[int, ...]` means "any number of `int`s" + - `*args: typing.Tuple[int, float, str]` means "exactly `(int, float, str)`, + in that order" + - `**kwargs: typing.Dict[str, int]` means "all **kwargs are of type `int`". + Note the key type for the `**kwargs` dict is always `str`. - The main point of this feature is to eliminate `if`/`elif`/`elif`... blocks - that switch by `isinstance` on arguments, and then raise `TypeError` - in the final `else`, by implementing this machinery centrally. + **The first multimethod that matches wins, in most-recently-registered order.** + (This is unlike in Julia, which matches the most specific applicable multimethod.) - Another use case of `@generic` are functions like the builtin `range`, where - the *role* of an argument in a particular position depends on the *number of* - arguments passed in the call. + In other words, later multimethod definitions override earlier ones. So specify + the implementation with the most generic types first, and then move on to the + more specific ones. The mnemonic is, "the function is generally defined like + this, except if the arguments match these particular types..." **Differences to tools in the standard library**: @@ -147,28 +180,28 @@ def example(): no public `register` attribute. Instead, generic functions are saved in a global registry. - Unlike `typing.overload`, the implementations are given in the method bodies. + Unlike `typing.overload`, the implementations are given in the multimethod + bodies. **Interaction with OOP**: Beside regular functions, `@generic` can be installed on instance, class - or static methods (in the OOP sense). `self` and `cls` parameters do not + or static *methods* (in the OOP sense). `self` and `cls` parameters do not participate in dispatching, and need no type annotation. On instance and class methods, the self-like parameter, beside appearing as the first positional-or-keyword parameter, **must be named** one of `self`, `this`, `cls`, or `klass` to be detected by the ignore mechanism. This limitation is due to implementation reasons; while a class body is being - evaluated, the context needed to distinguish a method (OOP sense) from a - regular function is not yet present. + evaluated, the context needed to distinguish a method from a regular function + is not yet present. When `@generic` is installed on an instance method or on a `@classmethod`, - then at call time, classes are tried in MRO order. **All** generic-function - methods of the OOP method defined in the class currently being looked up - are tested for matches first, **before** moving on to the next class in the - MRO. (This has subtle consequences, related to in which class in the - hierarchy the various generic-function methods for a particular OOP method - are defined.) + then at call time, classes are tried in MRO order. **All** multimethods + of the method defined in the class currently being looked up are tested + for matches first, **before** moving on to the next class in the MRO. + This has subtle consequences, related to in which class in the hierarchy + the various multimethods for a particular method are defined. For *static methods* MRO lookup is not supported. Basically, one of the roles of `cls` or `self` is to define the MRO; a `@staticmethod` doesn't @@ -179,7 +212,7 @@ def example(): **CAUTION**: - To declare a parameter of a method as dynamically typed, explicitly + To declare a parameter of a multimethod as dynamically typed, explicitly annotate it as `typing.Any`; don't just omit the type annotation. Explicit is better than implicit; **this is a feature**. @@ -188,15 +221,16 @@ def example(): At the moment, `@generic` does not work with `curry`. Adding curry support needs changes to the dispatch logic in `curry`. + """ return _register_generic(_function_fullname(f), f) @register_decorator(priority=98) def augment(target): - """Parametric decorator. Add a method to function `target`. + """Parametric decorator. Add a multimethod to generic function `target`. - Like `@generic`, but the target function on which the method will be - registered is chosen separately, so that you can extend a generic + Like `@generic`, but the generic function on which the method will be + registered is chosen separately, so that you can augment a generic function previously defined in some other `.py` source file. The return value of `augment` is the multiple-dispatch dispatcher @@ -223,11 +257,11 @@ class MyOwnType: def f(x: MyOwnType): ... - **CAUTION**: Beware of type piracy when you use this. That is: + **CAUTION**: Beware of type piracy when you use `@augment`. That is: 1. For arbitrary input types you don't own, augment only a function you own, OR 2. Augment a function defined somewhere else only if at least one parameter - is of a type you own. + (in the call signature you are adding) is of a type you own. Satisfying **one** of these conditions is sufficient to avoid type piracy. @@ -237,7 +271,7 @@ def f(x: MyOwnType): https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy """ if not isgeneric(target): - raise TypeError(f"{target} is not a generic function, cannot add methods to it.") + raise TypeError(f"{target} is not a generic function, cannot add multimethods to it.") return partial(_register_generic, _function_fullname(target)) @register_decorator(priority=98) @@ -251,7 +285,7 @@ def typed(f): Also, unlike a basic `isinstance` check, this allows using features from the `typing` stdlib module in the type specifications. - After a `@typed` function has been created, no more methods can be + Once a `@typed` function has been created, no more multimethods can be attached to it. `@typed` works with `curry`, because the function has only one call @@ -272,16 +306,19 @@ def typed(f): def methods(f): """Print, to stdout, a human-readable list of multimethods currently registered to `f`. - This calls `list_methods`, which see. + For introspection in the REPL. This works by calling `list_methods`, which see. - (This is like the `methods` function of Julia.) + This is like the `methods` function of Julia. """ print(format_methods(f)) def format_methods(f): """Format, as a string, a human-readable list of multimethods currently registered to `f`. - This calls `list_methods`, which see. + One level lower than `methods`; format a human-readable message, but return it + instead of printing it. + + This works by calling `list_methods`, which see. """ function, _ = getfunc(f) multimethods = list_methods(f) @@ -295,14 +332,17 @@ def format_methods(f): def list_methods(f): """Return a list of the multimethods currently registered to `f`. - The methods are returned in the order they would be tried by the dispatcher. + The multimethods are returned in the order they would be tested by the dispatcher + when the generic function is called. The return value is a list, where each item is `(callable, type_signature)`. - The type signature is in the format returned by `typing.get_type_hints`. + Each type signature is in the format returned by `typing.get_type_hints`. `f`: a callable that has been declared `@generic` or `@typed`. - Bound OOP methods are resolved to the underlying function automatically. + **Interaction with OOP**: + + Bound methods are resolved to the underlying function automatically. The `self`/`cls` argument is extracted from the `__self__` attribute of the bound method, enabling linked dispatcher lookups in the MRO. @@ -327,10 +367,11 @@ def list_methods(f): Finally, note that while that is how `list_methods` works, it is not the mechanism actually used to determine `self`/`cls` when *calling* the - generic function. There, the value is extracted from the first positional - argument of the call. This is because the dispatcher is actually installed - on the underlying raw function, so it has no access to the metadata of the - bound method (which, as seen from the dispatcher, is on the outside). + generic function. There, the value of `self`/`cls` is extracted from the + first positional argument of the call. This is because the dispatcher is + actually installed on the underlying raw function, so it has no access to + the metadata of the bound method (which, as seen from the dispatcher, is + on the outside). """ function, _ = getfunc(f) if not isgeneric(function): @@ -367,14 +408,18 @@ def _name_of_1st_positional_parameter(f): def _list_multimethods(dispatcher, self_or_cls=None): """List multimethods currently registered to a given dispatcher. - `self_or_cls`: If `dispatcher` is installed on an OOP instance method - or on an OOP `@classmethod`, set this to perform MRO + `self_or_cls`: If `dispatcher` is installed on an instance method + or on a `@classmethod`, set this to perform MRO lookups to find linked dispatchers. """ + # TODO: Compute closest candidates, like Julia does? (see `methods`, `MethodError` in Julia) + # TODO: (If we do that, we need to look at the attempted call. When no call (just listing + # TODO: multimethods in the REPL), the current ordering is probably fine.) + # For regular functions, ours is the only registry we need to look at: relevant_registries = [reversed(dispatcher._method_registry)] - # But if this dispatcher is installed on an OOP method, we must + # But if this dispatcher is installed on a method, we must # look up generic function methods also in the class's MRO. # # For *static methods* MRO is not supported. Basically, one of @@ -396,7 +441,7 @@ def _list_multimethods(dispatcher, self_or_cls=None): if hasattr(base, dispatcher.__name__): # does this particular super have f? base_oop_method = getattr(base, dispatcher.__name__) base_raw_function, _ = getfunc(base_oop_method) - if isgeneric(base_raw_function): # it's @generic + if isgeneric(base_raw_function): # it's @generic or @typed base_registry = getattr(base_raw_function, "_method_registry") relevant_registries.append(reversed(base_registry)) @@ -407,7 +452,7 @@ def _format_method(method): # Taking a page from Julia and some artistic libert thecallable, type_signature = method # Our `type_signature` is based on `typing.get_type_hints`, # but for the error message, we need something that formats - # like source code. Hence use `inspect.signature`. + # like source code. Hence we use `inspect.signature`. thesignature = inspect.signature(thecallable) function, _ = getfunc(thecallable) filename = inspect.getsourcefile(function) @@ -430,9 +475,6 @@ def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): # TODO: the signature may specify something like `Sequence[int]`. Knowing a `list` # TODO: was passed doesn't help debug that it was `Sequence[str]` when a `Sequence[int]` # TODO: was expected. The actual value at least implicitly contains the type information. - # - # TODO: Compute closest candidates, like Julia does? (see `methods`, `MethodError` in Julia) - # TODO: (If we do that, we must also modify the dispatch logic.) args_list = [repr(x) for x in args] args_str = ", ".join(args_list) sep = ", " if args and kwargs else "" @@ -444,23 +486,26 @@ def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): f"Multimethods for @{isgeneric(dispatcher)} {repr(dispatcher.__qualname__)} (most recent match attempt last):\n{methods_str}") raise TypeError(msg) -def _register_generic(fullname, f): - """Register a method for a generic function. +def _register_generic(fullname, multimethod): + """Register a multimethod for a generic function. - This is a low-level function; you'll likely want `generic` or `augment`. + This is a low-level function; you'll likely want `@generic` or `@augment`. - fullname: str, fully qualified name of target function to register - the method on, used as key in the dispatcher registry. + fullname: str, fully qualified name of function to register the multimethod + on, used as key in the dispatcher registry. - Registering the first method on a given `fullname` makes + Registering the first multimethod on a given `fullname` makes that function generic, and creates the dispatcher for it. - f: callable, the new method to register. + Second and further registrations using the same `fullname` add + the new multimethod to the existing dispatcher. + + multimethod: callable, the new multimethod to register. - Return value is the dispatcher that replaces the original function. + Return value is the dispatcher. """ if fullname not in _dispatcher_registry: - # Create the dispatcher. This will replace the original f. + # Create the dispatcher. This will replace the original function. # # We want @wraps to preserve docstrings, so the decorator must be a function, not a class. # https://stackoverflow.com/questions/6394511/python-functools-wraps-equivalent-for-classes @@ -472,16 +517,16 @@ def _register_generic(fullname, f): # since we see just bare functions. In the OOP case, the dispatcher # is installed on the raw function before it becomes a bound method. # (That in itself is just as it should be.) - first_param_name = _name_of_1st_positional_parameter(f) - f_is_most_likely_an_oop_method = first_param_name in self_parameter_names - @wraps(f) + first_param_name = _name_of_1st_positional_parameter(multimethod) + most_likely_an_oop_method = first_param_name in self_parameter_names + @wraps(multimethod) def dispatcher(*args, **kwargs): # Let's see if we might have been passed a `self`/`cls` parameter, # and if so, get its value. (Recall that in Python, it is always # the first positional parameter.) - if f_is_most_likely_an_oop_method: + if most_likely_an_oop_method: if len(args) < 1: # pragma: no cover, shouldn't happen. - raise TypeError(f"MRO lookup failed: no value provided for self-like parameter {repr(first_param_name)} when calling generic-function OOP method {fullname}") + raise TypeError(f"MRO lookup failed: no value provided for self-like parameter {repr(first_param_name)} when calling OOP method-like generic function {fullname}") self_or_cls = args[0] else: self_or_cls = None @@ -500,10 +545,10 @@ def dispatcher(*args, **kwargs): _raise_multiple_dispatch_error(dispatcher, args, kwargs, candidates=multimethods) dispatcher._method_registry = [] - def register(thecallable): - """Decorator. Register a new method for this generic function. + def register(multimethod): + """Decorator. Register a new multimethod for this generic function. - The method must have type annotations for all of its parameters; + The multimethod must have type annotations for all of its parameters; these are used for dispatching. An exception is the `self` or `cls` parameter of an OOP instance @@ -517,10 +562,10 @@ def register(thecallable): # One exception: the `self`/`cls` parameter of OOP instance methods and # class methods is not meaningful for dispatching, and we don't # have a runtime value to auto-populate its expected type when the - # definition runs. So we set it to `typing.Any` in the method's + # definition runs. So we set it to `typing.Any` in the multimethod's # expected type signature, which makes the dispatcher ignore it. - function, _ = getfunc(thecallable) + function, _ = getfunc(multimethod) parameters = inspect.signature(function).parameters parameter_names = [p.name for p in parameters.values()] type_signature = typing.get_type_hints(function) @@ -550,18 +595,18 @@ def register(thecallable): if not all(name in type_signature for name in parameter_names): failures = [name for name in parameter_names if name not in type_signature] plural = "s" if len(failures) > 1 else "" - wrapped_list = [f"'{x}'" for x in failures] - wrapped_str = ", ".join(wrapped_list) - msg = f"Method definition missing type annotation for parameter{plural}: {wrapped_str}" + repr_list = [repr(x) for x in failures] + repr_str = ", ".join(repr_list) + msg = f"Multimethod definition missing type annotation for parameter{plural}: {repr_str}" raise TypeError(msg) - dispatcher._method_registry.append((thecallable, type_signature)) - return dispatcher # Replace the callable with the dispatcher for this generic function. + dispatcher._method_registry.append((multimethod, type_signature)) + return dispatcher # Replace the callable with this generic function's dispatcher. - dispatcher._register = register # save it for use by us to register methods to this dispatcher + dispatcher._register = register _dispatcher_registry[fullname] = dispatcher dispatcher = _dispatcher_registry[fullname] - if isgeneric(dispatcher) == "typed": # co-operation with @typed, below - raise TypeError("@typed: cannot register additional methods.") - return dispatcher._register(f) + if isgeneric(dispatcher) == "typed": + raise TypeError("@typed: cannot register additional multimethods.") + return dispatcher._register(multimethod) From 58a8b242feb0ff07024fb50ff5e9f4ea83a1e456 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:25:41 +0300 Subject: [PATCH 294/832] fix stray whitespace --- unpythonic/tests/test_dispatch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 7f22ba9c..36f1f811 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -219,7 +219,7 @@ def instmeth(self, x: float): # # See discussions on interaction between `@staticmethod` and `super` in Python: # https://bugs.python.org/issue31118 - # https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879 + # https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879 test[tt2.staticmeth(3.14) == "float 6.28"] # this is available on `tt2` test_raises[TypeError, tt2.staticmeth("hi")] # but this is not (no MRO) test_raises[TypeError, tt2.staticmeth(21)] From adcfb50b36ae049eb0eaf0c7e5d253a8ca26bd87 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:34:59 +0300 Subject: [PATCH 295/832] advertise multiple dispatch in README --- README.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/README.md b/README.md index 8967b41d..4f8e42e4 100644 --- a/README.md +++ b/README.md @@ -314,6 +314,63 @@ assert x == 85 The point is usability: in a function composition using pipe syntax, data flows from left to right.
+
Multiple-dispatch generic functions, like in CLOS or Julia. + +[[docs](doc/features.md#generic-typed-isoftype-multiple-dispatch)] + +```python +from unpythonic import generic + +@generic +def my_range(stop: int): # first registration creates the generic function and the first multimethod + return my_range(0, 1, stop) +@generic +def my_range(start: int, stop: int): # further registrations add more multimethods + return my_range(start, 1, stop) +@generic +def my_range(start: int, step: int, stop: int): + return start, step, stop +``` + +This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. *Holy traits* are also a possibility: + +```python +import typing +from unpythonic import generic, augment + +class FunninessTrait: + pass +class IsFunny(FunninessTrait): + pass +class IsNotFunny(FunninessTrait): + pass + +@generic +def funny(x: typing.Any): # default + raise NotImplementedError(f"`funny` trait not registered for any type specification matching {type(x)}") + +@augment(funny) +def funny(x: str): # noqa: F811 + return IsFunny() +@augment(funny) +def funny(x: int): # noqa: F811 + return IsNotFunny() + +@generic +def laugh(x: typing.Any): + return laugh(funny(x), x) + +@augment(laugh) +def laugh(traitvalue: IsFunny, x: typing.Any): + return f"Ha ha ha, {x} is funny!" +@augment(laugh) +def laugh(traitvalue: IsNotFunny, x: typing.Any): + return f"{x} is not funny." + +assert laugh("that") == "Ha ha ha, that is funny!" +assert laugh(42) == "42 is not funny." +``` +
Conditions: resumable, modular error handling, like in Common Lisp. [[docs](doc/features.md#handlers-restarts-conditions-and-restarts)] From 1cb558f3215a0cb1a18468c4c757c337ff1f2d90 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:36:08 +0300 Subject: [PATCH 296/832] formatting --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 4f8e42e4..805b083d 100644 --- a/README.md +++ b/README.md @@ -322,7 +322,7 @@ The point is usability: in a function composition using pipe syntax, data flows from unpythonic import generic @generic -def my_range(stop: int): # first registration creates the generic function and the first multimethod +def my_range(stop: int): # create the generic function and the first multimethod return my_range(0, 1, stop) @generic def my_range(start: int, stop: int): # further registrations add more multimethods @@ -332,7 +332,9 @@ def my_range(start: int, step: int, stop: int): return start, step, stop ``` -This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. *Holy traits* are also a possibility: +This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. + +*Holy traits* are also a possibility: ```python import typing @@ -347,7 +349,7 @@ class IsNotFunny(FunninessTrait): @generic def funny(x: typing.Any): # default - raise NotImplementedError(f"`funny` trait not registered for any type specification matching {type(x)}") + raise NotImplementedError(f"`funny` trait not registered for anything matching {type(x)}") @augment(funny) def funny(x: str): # noqa: F811 From d2785a5cd60a682a9cf691f01368ab41d033ce96 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 02:49:59 +0300 Subject: [PATCH 297/832] add multimethod docstring to generic function __doc__ on register --- unpythonic/dispatch.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 4f459193..961588a0 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -601,6 +601,18 @@ def register(multimethod): raise TypeError(msg) dispatcher._method_registry.append((multimethod, type_signature)) + + # Update entry point docstring to include docs for the new multimethod, + # and its call signature. + our_doc = _format_method((multimethod, type_signature)) + if multimethod.__doc__: + our_doc += "\n" + multimethod.__doc__ + if dispatcher.__doc__: + dispatcher.__doc__ += "\n\n" + ("-" * 80) + "\n" + dispatcher.__doc__ += our_doc + else: + dispatcher.__doc__ = our_doc + return dispatcher # Replace the callable with this generic function's dispatcher. dispatcher._register = register From ecbc21b25602e05b9688746af02c59ad5fa5cba7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 03:01:11 +0300 Subject: [PATCH 298/832] handle docstring correctly for the first multimethod, too --- unpythonic/dispatch.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 961588a0..b302c294 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -604,16 +604,22 @@ def register(multimethod): # Update entry point docstring to include docs for the new multimethod, # and its call signature. - our_doc = _format_method((multimethod, type_signature)) + call_signature_desc = _format_method((multimethod, type_signature)) + our_doc = call_signature_desc if multimethod.__doc__: our_doc += "\n" + multimethod.__doc__ - if dispatcher.__doc__: + + isfirstmultimethod = len(dispatcher._method_registry) == 1 + if isfirstmultimethod or not dispatcher.__doc__: + # Override the original doc of the function that was converted + # into the dispatcher; this adds the call signature to the top. + dispatcher.__doc__ = our_doc + else: + # Add the call signature and doc for the new multimethod. dispatcher.__doc__ += "\n\n" + ("-" * 80) + "\n" dispatcher.__doc__ += our_doc - else: - dispatcher.__doc__ = our_doc - return dispatcher # Replace the callable with this generic function's dispatcher. + return dispatcher # Replace the multimethod callable with this generic function's dispatcher. dispatcher._register = register _dispatcher_registry[fullname] = dispatcher From e1e880d1f75b2949308548c2a32541c1b74f83f9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 03:04:06 +0300 Subject: [PATCH 299/832] update comment It's not critical to preserve the docstring at this point, because we will in any case overwrite it in `register`. But the preservation of the other fields done by `functools.wraps` sounds like it's the right thing to do here. --- unpythonic/dispatch.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index b302c294..dc4746c4 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -507,10 +507,6 @@ def _register_generic(fullname, multimethod): if fullname not in _dispatcher_registry: # Create the dispatcher. This will replace the original function. # - # We want @wraps to preserve docstrings, so the decorator must be a function, not a class. - # https://stackoverflow.com/questions/6394511/python-functools-wraps-equivalent-for-classes - # https://stackoverflow.com/questions/25973376/functools-update-wrapper-doesnt-work-properly#25973438 - # # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. # # Here we're operating at the wrong abstraction level for that, From 6180eb5fcf9c645a1acec1eaddb354f2fda66277 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 03:12:19 +0300 Subject: [PATCH 300/832] mention docstring auto-concatenation for multimethods --- doc/features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/features.md b/doc/features.md index e79030c0..7c0c4cb6 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3013,6 +3013,8 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of *Added the function `methods`, which displays a list of multimethods of a generic function.* +*Docstrings of the multimethods are now automatically concatenated to make up the docstring of the generic function, so you can document each multimethod separately.* + *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. From 93842772d5bae98e6ccdd542f7849741b44d0ac5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 12:45:07 +0300 Subject: [PATCH 301/832] update comment --- unpythonic/dispatch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index dc4746c4..f9154d2a 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -413,8 +413,8 @@ def _list_multimethods(dispatcher, self_or_cls=None): lookups to find linked dispatchers. """ # TODO: Compute closest candidates, like Julia does? (see `methods`, `MethodError` in Julia) - # TODO: (If we do that, we need to look at the attempted call. When no call (just listing - # TODO: multimethods in the REPL), the current ordering is probably fine.) + # TODO: (If we do that, we need to look at the bound arguments. When just listing multimethods + # TODO: in the REPL, the current ordering is probably fine.) # For regular functions, ours is the only registry we need to look at: relevant_registries = [reversed(dispatcher._method_registry)] From 9eaa4c76c934687511d919c43e934bd44eb035e3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 12:47:02 +0300 Subject: [PATCH 302/832] terminology and wording fixes --- doc/features.md | 2 +- unpythonic/dispatch.py | 2 +- unpythonic/tests/test_dispatch.py | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 7c0c4cb6..68b3d36f 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3135,7 +3135,7 @@ Based on my own initial experiments with this feature, the machinery itself work The machinery itself is also missing some advanced features, such as matching the most specific multimethod candidate instead of the most recently defined one; an `issubclass` equivalent that understands `typing` type specifications; and a mechanism to remove previously declared multimethods. -**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If methods are added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the multiple-dispatch table is global state! +**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If a new multimethod is added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the multiple-dispatch table is global state! #### ``typed``: add run-time type checks with type annotation syntax diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index f9154d2a..30609b4b 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -420,7 +420,7 @@ def _list_multimethods(dispatcher, self_or_cls=None): relevant_registries = [reversed(dispatcher._method_registry)] # But if this dispatcher is installed on a method, we must - # look up generic function methods also in the class's MRO. + # look up multimethods also in the class's MRO. # # For *static methods* MRO is not supported. Basically, one of # the roles of `cls` or `self` is to define the MRO; a static diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 36f1f811..62e5ebba 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -201,9 +201,9 @@ def instmeth(self, x: float): return f"floating with {self.a * x}" tt2 = BabyTestTarget(3) - # the new generic-function methods become available, installed on the OOP method + # the new multimethods become available, installed on the OOP method test[tt2.instmeth(3.14) == "floating with 9.42"] - # old generic-function methods registered by the ancestor remain available + # old multimethods registered by the ancestor remain available test[tt2.instmeth("hi") == "hi hi hi"] test[tt2.instmeth(21) == 63] test[tt2.clsmeth(3.14) == "Test target floats: 6.28"] From cef82ca4f06786bb3a189d9f2ca3c50e888548d9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 14:33:25 +0300 Subject: [PATCH 303/832] add `@generic` support to `curry` and arity utilities `curry` and the utilities `arities`, `required_kwargs`, and `optional_kwargs` now support `@generic` functions. --- CHANGELOG.md | 2 + doc/features.md | 8 +- unpythonic/arity.py | 44 +++++- unpythonic/dispatch.py | 289 +++++++++++++++++++---------------- unpythonic/fun.py | 31 ++-- unpythonic/tests/test_fun.py | 16 ++ 6 files changed, 238 insertions(+), 152 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b7a693d8..4d8c764d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,6 +87,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Add function `isgeneric` to detect whether a callable has been declared `@generic`. - Add function `methods`: display a list of multimethods of a generic function. - It is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. + - `curry` now supports `@generic` functions. + - The utilities `arities`, `required_kwargs`, and `optional_kwargs` now support `@generic` functions. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) diff --git a/doc/features.md b/doc/features.md index 68b3d36f..cb546124 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3015,6 +3015,10 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of *Docstrings of the multimethods are now automatically concatenated to make up the docstring of the generic function, so you can document each multimethod separately.* +*`curry` now supports `@generic`. In the case where the **number** of positional arguments supplied so far matches at least one multimethod, but there is no match for the given combination of argument **types**, `curry` waits for more arguments (returning the curried function).* + +**CAUTION**: *Determining whether there **could** be a match for a `@generic` is the only type checking performed by ``curry``. When using ``curry`` with ``@generic`` or ``@typed``, argument type errors are only detected when the actual call triggers - just like in code using ``curry`` and traditional run-time ``isinstance`` checks. This may make it hard to debug.* + *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. @@ -3036,8 +3040,6 @@ If several multimethods of the same generic function match the arguments given, **CAUTION**: The winning multimethod is chosen differently from Julia, where the most specific multimethod wins. Doing that requires a more careful type analysis than what we have here. -**CAUTION**: `@generic` does not currently work with `curry`. Adding support requires changes to the already complex logic in `curry`; it is not high on the priority list. - The details are best explained by example: ```python @@ -3165,8 +3167,6 @@ jack(3.14) # TypeError For which features of the ``typing`` stdlib module are supported, see ``isoftype`` below. -**CAUTION**: When using ``typed`` with ``curry``, the type checking (and hence ``TypeError``, if any) only occurs when the actual call triggers. Code using that combination may be hard to debug. - #### ``isoftype``: the big sister of ``isinstance`` diff --git a/unpythonic/arity.py b/unpythonic/arity.py index 3453dcd7..6265ce21 100644 --- a/unpythonic/arity.py +++ b/unpythonic/arity.py @@ -217,6 +217,9 @@ def arities(f): does not implicitly provide a `self`, because there is none to be had. This behavior is reflected in the return value of `arities`.) + If `f` is `@generic` (see `unpythonic.dispatch`), we scan its multimethods, + and return the smallest `min_arity` and the largest `max_arity`. + Parameters: `f`: function The function to inspect. @@ -238,6 +241,20 @@ def arities(f): return _builtin_arities[f] except TypeError: # f is of an unhashable type pass + + # Integration with the multiple-dispatch system (multimethods). + from .dispatch import isgeneric, list_methods # circular import + if isgeneric(f): + min_lower = _infty + max_upper = 0 + for (thecallable, type_signature) in list_methods(f): + lower, upper = arities(thecallable) # let UnknownArity propagate + if lower < min_lower: + min_lower = lower + if upper > max_upper: + max_upper = upper + return min_lower, max_upper + try: lower = 0 upper = 0 @@ -258,25 +275,40 @@ def arities(f): raise UnknownArity(*e.args) def required_kwargs(f): - """Return a set containing the names of required name-only arguments of f. + """Return a set containing the names of required name-only arguments of `f`. - "Required": has no default. + *Required* means the parameter has no default. - Raises UnknownArity if inspection failed. + If `f` is `@generic` (see `unpythonic.dispatch`), we scan its multimethods, + and return the names of required kwargs accepted by *any* of its multimethods. + + Raises `UnknownArity` if inspection failed. """ return _kwargs(f, optionals=False) def optional_kwargs(f): - """Return a set containing the names of optional name-only arguments of f. + """Return a set containing the names of optional name-only arguments of `f`. - "Optional": has a default. + *Optional* means the parameter has a default. - Raises UnknownArity if inspection failed. + If `f` is `@generic` (see `unpythonic.dispatch`), we scan its multimethods, + and return the names of optional kwargs accepted by *any* of its multimethods. + + Raises `UnknownArity` if inspection failed. """ return _kwargs(f, optionals=True) def _kwargs(f, optionals=True): f, _ = getfunc(f) + + # Integration with the multiple-dispatch system (multimethods). + from .dispatch import isgeneric, list_methods # circular import + if isgeneric(f): + thekwargs = {} + for (thecallable, type_signature) in list_methods(f): + thekwargs.update(_kwargs(thecallable, optionals=optionals)) + return thekwargs + try: if optionals: pred = lambda v: v.default is not Parameter.empty # optionals diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 30609b4b..d8eb678d 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -210,6 +210,13 @@ def example(): To work with OOP inheritance, in the decorator list, `@generic` must be on inside of (i.e. run before) `@classmethod` or `@staticmethod`. + **Interaction with `curry`**: + + Starting with v0.15.0, `curry` supports `@generic`. In the case where the + *number* of positional arguments supplied so far matches at least one + multimethod, but there is no match for the given combination of argument + *types*, `curry` waits for more arguments (returning the curried function). + **CAUTION**: To declare a parameter of a multimethod as dynamically typed, explicitly @@ -218,12 +225,8 @@ def example(): See the limitations in `unpythonic.typecheck` for which features of the `typing` module are supported and which are not. - - At the moment, `@generic` does not work with `curry`. Adding curry support - needs changes to the dispatch logic in `curry`. - """ - return _register_generic(_function_fullname(f), f) + return _setup(_function_fullname(f), f) @register_decorator(priority=98) def augment(target): @@ -271,8 +274,8 @@ def f(x: MyOwnType): https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy """ if not isgeneric(target): - raise TypeError(f"{target} is not a generic function, cannot add multimethods to it.") - return partial(_register_generic, _function_fullname(target)) + raise TypeError(f"{_function_fullname(target)} is not a generic function, cannot add multimethods to it.") + return partial(_setup, _function_fullname(target)) @register_decorator(priority=98) def typed(f): @@ -287,16 +290,6 @@ def typed(f): Once a `@typed` function has been created, no more multimethods can be attached to it. - - `@typed` works with `curry`, because the function has only one call - signature, as usual. - - **CAUTION**: - - If used with `curry`, argument type errors will only be detected when - `curry` triggers the actual call. To fix this, `curry` would need to - perform some more introspection on the callable, and to actually know - about this dispatch system. It's not high on the priority list. """ # TODO: Fix the epic fail at fail-fast, and update the corresponding test. s = generic(f) @@ -308,6 +301,26 @@ def methods(f): For introspection in the REPL. This works by calling `list_methods`, which see. + Example - entering this in an IPython session:: + + from unpythonic import generic, methods + + @generic + def f(x: int): + return "int" + + @generic + def f(x: float): + return "float" + + methods(f) + + the result is: + + Multimethods for @generic __main__.f: + f(x: float) from :1 + f(x: int) from :1 + This is like the `methods` function of Julia. """ print(format_methods(f)) @@ -325,9 +338,9 @@ def format_methods(f): if multimethods: methods_list = [f" {_format_method(x)}" for x in multimethods] methods_str = "\n".join(methods_list) - else: # pragma: no cover, in practice should always have one method. + else: # pragma: no cover, in practice a generic should always have at least one method. methods_str = " " - return f"Multimethods for @{isgeneric(f)} {repr(function.__qualname__)}:\n{methods_str}" + return f"Multimethods for @{isgeneric(f)} {_function_fullname(function)}:\n{methods_str}" def list_methods(f): """Return a list of the multimethods currently registered to `f`. @@ -375,7 +388,7 @@ def list_methods(f): """ function, _ = getfunc(f) if not isgeneric(function): - raise TypeError(f"{repr(function.__qualname__)} is not a generic function, it does not have multimethods.") + raise TypeError(f"{_function_fullname(function)} is not a generic function, it does not have multimethods.") # In case of a bound method (either `Foo.classmeth` or `foo.instmeth`), # we can get the value for `self`/`cls` argument from its `__self__` attribute. @@ -459,6 +472,17 @@ def _format_method(method): # Taking a page from Julia and some artistic libert source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" +def _find_matching_multimethod(dispatcher, args, kwargs): + multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) + for thecallable, type_signature in multimethods: + try: + bound_arguments = resolve_bindings(thecallable, *args, **kwargs) + except TypeError: # arity mismatch, so this method can't be the one the call is looking for. + continue + if _match_argument_types(type_signature, bound_arguments): + return thecallable + return None + # `type_signature`: in the format returned by `typing.get_type_hints`. # `bound_arguments`: see `unpythonic.arity.resolve_bindings`. def _match_argument_types(type_signature, bound_arguments): @@ -469,6 +493,28 @@ def _match_argument_types(type_signature, bound_arguments): return False return True +# Given a callable and a tuple of positional arguments, extract the value of `self`/`cls` argument, if any. +def _extract_self_or_cls(thecallable, args): + # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. + # + # Here we're operating at the wrong abstraction level for that, + # since we see just bare functions. In the OOP case, the dispatcher + # is installed on the raw function before it becomes a bound method. + # (That in itself is just as it should be.) + first_param_name = _name_of_1st_positional_parameter(thecallable) + most_likely_an_oop_method = first_param_name in self_parameter_names + + # Let's see if we might have been passed a `self`/`cls` parameter, + # and if so, get its value. (Recall that in Python, it is always + # the first positional parameter.) + if most_likely_an_oop_method: + if len(args) < 1: # pragma: no cover, shouldn't happen. + raise TypeError(f"MRO lookup failed: no value provided for self-like parameter {repr(first_param_name)} for OOP method-like generic function {_function_fullname(thecallable)}") + self_or_cls = args[0] + else: + self_or_cls = None + return self_or_cls + def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): # TODO: It would be nice to show the type signature of the args actually given, # TODO: but in the general case this is difficult. We can't just `type(x)`, since @@ -483,11 +529,11 @@ def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): methods_list = [f" {_format_method(x)}" for x in candidates] methods_str = "\n".join(methods_list) msg = (f"No multiple-dispatch match for the call {dispatcher.__qualname__}({args_str}{sep}{kws_str}).\n" - f"Multimethods for @{isgeneric(dispatcher)} {repr(dispatcher.__qualname__)} (most recent match attempt last):\n{methods_str}") + f"Multimethods for @{isgeneric(dispatcher)} {_function_fullname(dispatcher)} (most recent match attempt last):\n{methods_str}") raise TypeError(msg) -def _register_generic(fullname, multimethod): - """Register a multimethod for a generic function. +def _setup(fullname, multimethod): + """Register a multimethod for a generic function, creating the generic function if necessary. This is a low-level function; you'll likely want `@generic` or `@augment`. @@ -506,121 +552,100 @@ def _register_generic(fullname, multimethod): """ if fullname not in _dispatcher_registry: # Create the dispatcher. This will replace the original function. - # - # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. - # - # Here we're operating at the wrong abstraction level for that, - # since we see just bare functions. In the OOP case, the dispatcher - # is installed on the raw function before it becomes a bound method. - # (That in itself is just as it should be.) - first_param_name = _name_of_1st_positional_parameter(multimethod) - most_likely_an_oop_method = first_param_name in self_parameter_names @wraps(multimethod) def dispatcher(*args, **kwargs): - # Let's see if we might have been passed a `self`/`cls` parameter, - # and if so, get its value. (Recall that in Python, it is always - # the first positional parameter.) - if most_likely_an_oop_method: - if len(args) < 1: # pragma: no cover, shouldn't happen. - raise TypeError(f"MRO lookup failed: no value provided for self-like parameter {repr(first_param_name)} when calling OOP method-like generic function {fullname}") - self_or_cls = args[0] - else: - self_or_cls = None - - # Dispatch. - multimethods = _list_multimethods(dispatcher, self_or_cls) - for thecallable, type_signature in multimethods: - try: - bound_arguments = resolve_bindings(thecallable, *args, **kwargs) - except TypeError: # arity mismatch, so this method can't be the one the call is looking for. - continue - if _match_argument_types(type_signature, bound_arguments): - return thecallable(*args, **kwargs) - - # No match, report error. - _raise_multiple_dispatch_error(dispatcher, args, kwargs, candidates=multimethods) + thecallable = _find_matching_multimethod(dispatcher, args, kwargs) + if thecallable: + return thecallable(*args, **kwargs) + _raise_multiple_dispatch_error(dispatcher, args, kwargs, + candidates=_list_multimethods(dispatcher, + _extract_self_or_cls(dispatcher, args))) dispatcher._method_registry = [] - def register(multimethod): - """Decorator. Register a new multimethod for this generic function. - - The multimethod must have type annotations for all of its parameters; - these are used for dispatching. - - An exception is the `self` or `cls` parameter of an OOP instance - method or class method; that does not participate in dispatching, - and does not need a type annotation. - """ - # Using `inspect.signature` et al., we could auto-`Any` parameters - # that have no type annotation, but that would likely be a footgun. - # So we require a type annotation for each parameter. - # - # One exception: the `self`/`cls` parameter of OOP instance methods and - # class methods is not meaningful for dispatching, and we don't - # have a runtime value to auto-populate its expected type when the - # definition runs. So we set it to `typing.Any` in the multimethod's - # expected type signature, which makes the dispatcher ignore it. - - function, _ = getfunc(multimethod) - parameters = inspect.signature(function).parameters - parameter_names = [p.name for p in parameters.values()] - type_signature = typing.get_type_hints(function) - - # In the type signature, auto-`Any` the `self`/`cls` parameter, if any. - # - # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. - # - # The `@generic` decorator runs while the class body is being - # evaluated. In that context, an instance method looks just like a - # regular function. - # - # Also if `@generic` runs before `@classmethod` (to place Python's - # implicit `cls` handling outermost), also a class method looks - # just like a regular function to us. - # - # So we HACK, and special-case some suggestive *parameter names* - # when they appear the first position, though **Python itself - # doesn't do that**. For any crazy person not following Python - # naming conventions, our approach won't work. - if len(parameter_names) >= 1 and parameter_names[0] in self_parameter_names: - # In Python 3.6+, `dict` preserves insertion order. Make sure - # the `self` parameter appears first, for clearer error messages - # when no matching method is found. - type_signature = {parameter_names[0]: typing.Any, **type_signature} - - if not all(name in type_signature for name in parameter_names): - failures = [name for name in parameter_names if name not in type_signature] - plural = "s" if len(failures) > 1 else "" - repr_list = [repr(x) for x in failures] - repr_str = ", ".join(repr_list) - msg = f"Multimethod definition missing type annotation for parameter{plural}: {repr_str}" - raise TypeError(msg) - - dispatcher._method_registry.append((multimethod, type_signature)) - - # Update entry point docstring to include docs for the new multimethod, - # and its call signature. - call_signature_desc = _format_method((multimethod, type_signature)) - our_doc = call_signature_desc - if multimethod.__doc__: - our_doc += "\n" + multimethod.__doc__ - - isfirstmultimethod = len(dispatcher._method_registry) == 1 - if isfirstmultimethod or not dispatcher.__doc__: - # Override the original doc of the function that was converted - # into the dispatcher; this adds the call signature to the top. - dispatcher.__doc__ = our_doc - else: - # Add the call signature and doc for the new multimethod. - dispatcher.__doc__ += "\n\n" + ("-" * 80) + "\n" - dispatcher.__doc__ += our_doc - - return dispatcher # Replace the multimethod callable with this generic function's dispatcher. - - dispatcher._register = register + dispatcher._register = partial(_register_to, dispatcher) _dispatcher_registry[fullname] = dispatcher dispatcher = _dispatcher_registry[fullname] if isgeneric(dispatcher) == "typed": raise TypeError("@typed: cannot register additional multimethods.") - return dispatcher._register(multimethod) + return dispatcher._register(multimethod) # this returns the *dispatcher* + +def _register_to(dispatcher, multimethod): + """Decorator. Register a new `multimethod` to `dispatcher`. + + This is a low-level function used by `_setup`. + + The multimethod must have type annotations for all of its parameters; + these are used for dispatching. + + An exception is the `self` or `cls` parameter of an OOP instance + method or class method; that does not participate in dispatching, + and does not need a type annotation. + + After registering, this returns `dispatcher`. + """ + # Using `inspect.signature` et al., we could auto-`Any` parameters + # that have no type annotation, but that would likely be a footgun. + # So we require a type annotation for each parameter. + # + # One exception: the `self`/`cls` parameter of OOP instance methods and + # class methods is not meaningful for dispatching, and we don't + # have a runtime value to auto-populate its expected type when the + # definition runs. So we set it to `typing.Any` in the multimethod's + # expected type signature, which makes the dispatcher ignore it. + + function, _ = getfunc(multimethod) + parameters = inspect.signature(function).parameters + parameter_names = [p.name for p in parameters.values()] + type_signature = typing.get_type_hints(function) + + # In the type signature, auto-`Any` the `self`/`cls` parameter, if any. + # + # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. + # + # The `@generic` decorator runs while the class body is being + # evaluated. In that context, an instance method looks just like a + # regular function. + # + # Also if `@generic` runs before `@classmethod` (to place Python's + # implicit `cls` handling outermost), also a class method looks + # just like a regular function to us. + # + # So we HACK, and special-case some suggestive *parameter names* + # when they appear the first position, though **Python itself + # doesn't do that**. For any crazy person not following Python + # naming conventions, our approach won't work. + if len(parameter_names) >= 1 and parameter_names[0] in self_parameter_names: + # In Python 3.6+, `dict` preserves insertion order. Make sure + # the `self` parameter appears first, for clearer error messages + # when no matching method is found. + type_signature = {parameter_names[0]: typing.Any, **type_signature} + + if not all(name in type_signature for name in parameter_names): + failures = [name for name in parameter_names if name not in type_signature] + plural = "s" if len(failures) > 1 else "" + repr_list = [repr(x) for x in failures] + repr_str = ", ".join(repr_list) + msg = f"Multimethod definition missing type annotation for parameter{plural}: {repr_str}" + raise TypeError(msg) + + dispatcher._method_registry.append((multimethod, type_signature)) + + # Update entry point docstring to include docs for the new multimethod, + # and its call signature. + call_signature_desc = _format_method((multimethod, type_signature)) + our_doc = call_signature_desc + if multimethod.__doc__: + our_doc += "\n" + multimethod.__doc__ + + isfirstmultimethod = len(dispatcher._method_registry) == 1 + if isfirstmultimethod or not dispatcher.__doc__: + # Override the original doc of the function that was converted + # into the dispatcher; this adds the call signature to the top. + dispatcher.__doc__ = our_doc + else: + # Add the call signature and doc for the new multimethod. + dispatcher.__doc__ += "\n\n" + ("-" * 80) + "\n" + dispatcher.__doc__ += our_doc + + return dispatcher # Replace the multimethod callable with this generic function's dispatcher. diff --git a/unpythonic/fun.py b/unpythonic/fun.py index a0b9c47d..d1da10d0 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -21,6 +21,7 @@ from .arity import arities, resolve_bindings, tuplify_bindings, UnknownArity from .fold import reducel +from .dispatch import isgeneric, _find_matching_multimethod from .dynassign import dyn, make_dynvar from .regutil import register_decorator from .symbol import sym @@ -76,12 +77,12 @@ def memoized(*args, **kwargs): def _currycall(f, *args, **kwargs): """Co-operate with unpythonic.syntax.curry. - In a ``with autocurry`` block, need to call also when ``f()`` has transformed - to ``curry(f)``, but definitions can be curried as usual. + In a ``with autocurry`` block, we need to call `f` also when ``f()`` has + transformed to ``curry(f)``, but definitions can be curried as usual. Hence we provide this separate mode to curry-and-call even if no args. - This mode also no-ops when ``f`` is not inspectable, instead of raising + This mode no-ops when ``f`` is not inspectable, instead of raising an ``unpythonic.arity.UnknownArity`` exception. """ return curry(f, *args, _curry_force_call=True, _curry_allow_uninspectable=True, **kwargs) @@ -212,13 +213,23 @@ def foo(a, b, *, c, d): def curried(*args, **kwargs): outerctx = dyn.curry_context with dyn.let(curry_context=(outerctx + [f])): - if len(args) < min_arity: + # If `f` is `@generic` (see `unpythonic.dispatch`), and `min_arity <= len(args) <= max_arity`, + # we need to check for a multimethod match. If there is no match, it means that the arguments + # provided so far don't satisfy any registered multimethod call signature, but more arguments + # can still be accepted by the other call signatures. In that case, there are effectively + # too few arguments. + # + # For `@typed`, there is only one call signature, so the arity is the only important factor. + nargs = len(args) + if (nargs < min_arity or + (isgeneric(f) == "generic" and min_arity <= nargs <= max_arity and + not _find_matching_multimethod(f, args, kwargs))): p = partial(f, *args, **kwargs) if islazy(f): p = passthrough_lazy_args(p) return curry(p) # passthrough on right, like https://github.com/Technologicat/spicy - if len(args) > max_arity: + if nargs > max_arity: now_args, later_args = args[:max_arity], args[max_arity:] now_result = maybe_force_args(f, *now_args, **kwargs) # use up all kwargs now now_result = force(now_result) if not isinstance(now_result, tuple) else force1(now_result) @@ -259,7 +270,7 @@ def iscurried(f): def flip(f): """Decorator: flip (reverse) the positional arguments of f.""" - @wraps(f) + @ wraps(f) def flipped(*args, **kwargs): return maybe_force_args(f, *reversed(args), **kwargs) if islazy(f): @@ -282,7 +293,7 @@ def rotate(k): assert (rotate(1)(identity))(1, 2, 3) == (2, 3, 1) """ def rotate_k(f): - @wraps(f) + @ wraps(f) def rotated(*args, **kwargs): n = len(args) if not n: @@ -297,7 +308,7 @@ def rotated(*args, **kwargs): return rotated return rotate_k -@passthrough_lazy_args +@ passthrough_lazy_args def apply(f, arg0, *more, **kwargs): """Scheme/Racket-like apply. @@ -646,7 +657,7 @@ def to(*specs): """ return composeli(tokth(k, f) for k, f in specs) -@register_decorator(priority=80) +@ register_decorator(priority=80) def withself(f): """Decorator. Allow a lambda to refer to itself. @@ -671,7 +682,7 @@ def withself(f): assert fact(5) == 120 fact(5000) # no crash """ - @wraps(f) + @ wraps(f) def fwithself(*args, **kwargs): #return f(fwithself, *args, **kwargs) return maybe_force_args(f, fwithself, *args, **kwargs) # support unpythonic.syntax.lazify diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 798da7f3..91e56f94 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -6,6 +6,7 @@ from collections import Counter import sys +from ..dispatch import generic from ..fun import (memoize, curry, apply, identity, const, andf, orf, notf, @@ -210,6 +211,21 @@ def double(x): m2 = curry(m1, _curry_allow_uninspectable=True) test[m2 is m1] + with testset("curry integration with @generic"): # v0.15.0+ + @generic + def f(x: int): + return "int" + @generic + def f(x: float, y: str): # noqa: F811, new multimethod for the same generic function. + return "float, str" + test[callable(curry(f))] + test[curry(f, 42) == "int"] + # Although `f` has a multimethod that takes one argument, if that argument is a float, + # the call signature does not match, so in that case `curry` waits for more arguments + # (because it knows `f` has also a multimethod that accepts two arguments). + test[callable(curry(f, 3.14))] + test[curry(f, 3.14, "cat") == "float, str"] + with testset("compose"): double = lambda x: 2 * x inc = lambda x: x + 1 From 4b204a47c16cb41332cedf5b2d502dd55e6ad072 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 14:39:50 +0300 Subject: [PATCH 304/832] wording; mention `augment` --- doc/features.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/features.md b/doc/features.md index cb546124..f9c86a1e 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3021,16 +3021,18 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* -The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first two) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. +The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``augment`` adds a new multimethod to an existing generic function, ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first three) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. For what kind of things can be done with this, see particularly the [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) example in [`unpythonic.tests.test_dispatch`](../unpythonic/tests/test_dispatch.py). **NOTE**: This was inspired by the [multi-methods of CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) (the Common Lisp Object System), and the [generic functions of Julia](https://docs.julialang.org/en/v1/manual/methods/). -In `unpythonic`, we define the terms as follows: +In `unpythonic`, the terminology is as follows: - The function that supports multiple call signatures is a *generic function*. - - Its individual implementations are *multimethods*. + - Each of its individual implementations is a *multimethod*. + +The term *multimethod* distinguishes them from the OOP sense of *method*, already established in Python, as well as reminds that multiple arguments participate in dispatching. #### ``generic``: multiple dispatch with type annotation syntax From b670957e3772f97697d98e9c524cf647cd8e2e8c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 22:04:21 +0300 Subject: [PATCH 305/832] naming: _find_matching_multimethod -> _resolve_multimethod --- unpythonic/dispatch.py | 5 +++-- unpythonic/fun.py | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index d8eb678d..e33bf15e 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -472,7 +472,8 @@ def _format_method(method): # Taking a page from Julia and some artistic libert source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" -def _find_matching_multimethod(dispatcher, args, kwargs): +# Find the first matching multimethod that is registered on `dispatcher`. +def _resolve_multimethod(dispatcher, args, kwargs): multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) for thecallable, type_signature in multimethods: try: @@ -554,7 +555,7 @@ def _setup(fullname, multimethod): # Create the dispatcher. This will replace the original function. @wraps(multimethod) def dispatcher(*args, **kwargs): - thecallable = _find_matching_multimethod(dispatcher, args, kwargs) + thecallable = _resolve_multimethod(dispatcher, args, kwargs) if thecallable: return thecallable(*args, **kwargs) _raise_multiple_dispatch_error(dispatcher, args, kwargs, diff --git a/unpythonic/fun.py b/unpythonic/fun.py index d1da10d0..925574b3 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -21,7 +21,7 @@ from .arity import arities, resolve_bindings, tuplify_bindings, UnknownArity from .fold import reducel -from .dispatch import isgeneric, _find_matching_multimethod +from .dispatch import isgeneric, _resolve_multimethod from .dynassign import dyn, make_dynvar from .regutil import register_decorator from .symbol import sym @@ -223,7 +223,7 @@ def curried(*args, **kwargs): nargs = len(args) if (nargs < min_arity or (isgeneric(f) == "generic" and min_arity <= nargs <= max_arity and - not _find_matching_multimethod(f, args, kwargs))): + not _resolve_multimethod(f, args, kwargs))): p = partial(f, *args, **kwargs) if islazy(f): p = passthrough_lazy_args(p) From 9b4c3f9c875243a420d145e49fcc798ff2ea03ae Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 15 May 2021 22:13:22 +0300 Subject: [PATCH 306/832] improve docstrings --- unpythonic/dispatch.py | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index e33bf15e..387a5264 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -403,12 +403,14 @@ def list_methods(f): # Modeled after `mcpyrate.utils.format_macrofunction`, which does the same thing for macros. def _function_fullname(f): + """Return the full name of the callable `f`, including also its module name.""" function, _ = getfunc(f) # get the raw function also for OOP methods if not function.__module__: # At least macros defined in the REPL have `__module__=None`. return function.__qualname__ return f"{function.__module__}.{function.__qualname__}" def _name_of_1st_positional_parameter(f): + """Return the name, as a string, of the first positional parameter of the callable `f`.""" function, _ = getfunc(f) # get the raw function also for OOP methods parameters = inspect.signature(function).parameters poskinds = set((inspect.Parameter.POSITIONAL_ONLY, @@ -460,9 +462,16 @@ def _list_multimethods(dispatcher, self_or_cls=None): return list(chain.from_iterable(relevant_registries)) -# input format is an item returned by `_list_multimethods` -def _format_method(method): # Taking a page from Julia and some artistic liberty here. - thecallable, type_signature = method +def _format_method(multimethod): + """Format, as a string, a human-readable description of a multimethod. + + Input format is an item returned by `_list_multimethods`. + + The returned string includes the call signature, and the source filename + and starting line number. This output format takes a page from Julia, + with some artistic liberty. + """ + thecallable, type_signature = multimethod # Our `type_signature` is based on `typing.get_type_hints`, # but for the error message, we need something that formats # like source code. Hence we use `inspect.signature`. @@ -472,21 +481,26 @@ def _format_method(method): # Taking a page from Julia and some artistic libert source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" -# Find the first matching multimethod that is registered on `dispatcher`. def _resolve_multimethod(dispatcher, args, kwargs): + """Find the first matching multimethod on `dispatcher` for the given `args` and `kwargs`.""" multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) for thecallable, type_signature in multimethods: try: bound_arguments = resolve_bindings(thecallable, *args, **kwargs) - except TypeError: # arity mismatch, so this method can't be the one the call is looking for. + except TypeError: # arity mismatch, so this multimethod is not acceptable for the given args/kwargs. continue if _match_argument_types(type_signature, bound_arguments): return thecallable return None -# `type_signature`: in the format returned by `typing.get_type_hints`. -# `bound_arguments`: see `unpythonic.arity.resolve_bindings`. def _match_argument_types(type_signature, bound_arguments): + """Match bound arguments against the given type signature. + + Return whether the arguments match the signature. + + type_signature`: in the format returned by `typing.get_type_hints`. + `bound_arguments`: see `unpythonic.arity.resolve_bindings`. + """ for parameter, value in bound_arguments.arguments.items(): assert parameter in type_signature # resolve_bindings should already TypeError when not. expected_type = type_signature[parameter] @@ -494,8 +508,12 @@ def _match_argument_types(type_signature, bound_arguments): return False return True -# Given a callable and a tuple of positional arguments, extract the value of `self`/`cls` argument, if any. def _extract_self_or_cls(thecallable, args): + """From `thecallable` and positional arguments `args`, extract the value of `self`/`cls`, if any. + + Return value is either the value bound that would be bound to `self`/`cls` + (the first positional parameter), or `None`. + """ # TODO/FIXME: Not possible to detect `self`/`cls` parameters correctly. # # Here we're operating at the wrong abstraction level for that, @@ -517,6 +535,10 @@ def _extract_self_or_cls(thecallable, args): return self_or_cls def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): + """Raise a `TypeError` regarding a failed multiple dispatch (no matching multimethod). + + `candidates`: a list of multimethods that were attempted, but did not match. + """ # TODO: It would be nice to show the type signature of the args actually given, # TODO: but in the general case this is difficult. We can't just `type(x)`, since # TODO: the signature may specify something like `Sequence[int]`. Knowing a `list` From 41cca4625773016e46eebacd9d604461c53231bc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 17 May 2021 16:44:41 +0300 Subject: [PATCH 307/832] add TODO comment --- unpythonic/dispatch.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 387a5264..f2ad204b 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -19,6 +19,10 @@ https://docs.python.org/3/library/typing.html#typing.overload """ +# TODO: Reimplement in the same spirit as `functools.singledispatch`? +# TODO: The complication is, we support `typing` type specifications, not only concrete types. +# TODO: OTOH, `singledispatch` does handle the specific case of ABCs, via the subtype hooks. + __all__ = ["isgeneric", "generic", "augment", "typed", "methods", "format_methods", "list_methods"] From ae586e833333867f479f50a7fa4c6ba23b3b5e76 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 17 May 2021 17:33:20 +0300 Subject: [PATCH 308/832] add `partial` with type check; error in `curry` if no partial match --- doc/features.md | 3 +- unpythonic/arity.py | 18 ++++- unpythonic/dispatch.py | 130 ++++++++++++++++++++++-------- unpythonic/fun.py | 90 ++++++++++++++++++--- unpythonic/tests/test_dispatch.py | 31 ++++++- unpythonic/tests/test_fun.py | 28 +++++-- 6 files changed, 243 insertions(+), 57 deletions(-) diff --git a/doc/features.md b/doc/features.md index f9c86a1e..19202201 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1004,6 +1004,7 @@ Things missing from the standard library. - Can be used both as a decorator and as a regular function. - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. - **Caution**: If the positional arities of ``f`` cannot be inspected, currying fails, raising ``UnknownArity``. This may happen with builtins such as ``list.append``. + - `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. Type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions. Our `curry` uses this type-checking `partial` instead of the standard one, so currying supports fail-fast, too. **Added in v0.15.0.** - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values packed into a tuple are unpacked to the argument list of the next function in the chain. - `composelc`, `composerc`: curry each function before composing them. Useful with passthrough. @@ -3017,8 +3018,6 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of *`curry` now supports `@generic`. In the case where the **number** of positional arguments supplied so far matches at least one multimethod, but there is no match for the given combination of argument **types**, `curry` waits for more arguments (returning the curried function).* -**CAUTION**: *Determining whether there **could** be a match for a `@generic` is the only type checking performed by ``curry``. When using ``curry`` with ``@generic`` or ``@typed``, argument type errors are only detected when the actual call triggers - just like in code using ``curry`` and traditional run-time ``isinstance`` checks. This may make it hard to debug.* - *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``augment`` adds a new multimethod to an existing generic function, ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first three) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. diff --git a/unpythonic/arity.py b/unpythonic/arity.py index 6265ce21..0d21b369 100644 --- a/unpythonic/arity.py +++ b/unpythonic/arity.py @@ -8,7 +8,7 @@ __all__ = ["getfunc", "arities", "arity_includes", "required_kwargs", "optional_kwargs", "kwargs", - "resolve_bindings", "tuplify_bindings", + "resolve_bindings", "resolve_bindings_partial", "tuplify_bindings", "UnknownArity"] import copy @@ -337,6 +337,13 @@ def arity_includes(f, n): lower, upper = arities(f) return lower <= n <= upper +def resolve_bindings_partial(f, *args, **kwargs): + """Like `resolve_bindings`, but use `inspect.Signature.bind_partial`. + + That is, it is acceptable for some parameters of `f` not to have a binding. + """ + return _resolve_bindings(f, args, kwargs, _partial=True) + def resolve_bindings(f, *args, **kwargs): """Resolve parameter bindings established by `f` when called with the given args and kwargs. @@ -403,8 +410,15 @@ def f(a): f(42) f(a=42) # now the cache hits """ + return _resolve_bindings(f, args, kwargs, _partial=False) + +def _resolve_bindings(f, args, kwargs, *, _partial): f, _ = getfunc(f) - bound_arguments = signature(f).bind(*args, **kwargs) + thesignature = signature(f) + if _partial: + bound_arguments = thesignature.bind_partial(*args, **kwargs) + else: + bound_arguments = thesignature.bind(*args, **kwargs) bound_arguments.apply_defaults() return bound_arguments diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index f2ad204b..36178d3c 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -31,7 +31,7 @@ import inspect import typing -from .arity import resolve_bindings, getfunc +from .arity import getfunc, resolve_bindings, resolve_bindings_partial from .typecheck import isoftype from .regutil import register_decorator @@ -217,9 +217,14 @@ def example(): **Interaction with `curry`**: Starting with v0.15.0, `curry` supports `@generic`. In the case where the - *number* of positional arguments supplied so far matches at least one - multimethod, but there is no match for the given combination of argument - *types*, `curry` waits for more arguments (returning the curried function). + *number* of positional arguments supplied so far is acceptable for *some* + registered multimethod, but some parameters of that multimethod are still + missing bindings (i.e. it is not a full match), `curry` waits for more + arguments (returning the curried function). + + Passing an argument of an invalid type at any step of currying immediately + raises `TypeError`. Here "invalid type" means that for the partial application + constructed so far, no registered multimethod accepts the new argument(s). **CAUTION**: @@ -295,7 +300,6 @@ def typed(f): Once a `@typed` function has been created, no more multimethods can be attached to it. """ - # TODO: Fix the epic fail at fail-fast, and update the corresponding test. s = generic(f) del s._register # remove the ability to register more methods return s @@ -340,7 +344,8 @@ def format_methods(f): function, _ = getfunc(f) multimethods = list_methods(f) if multimethods: - methods_list = [f" {_format_method(x)}" for x in multimethods] + thecallables = [thecallable for thecallable, type_signature in multimethods] + methods_list = [f" {_format_callable(x)}" for x in thecallables] methods_str = "\n".join(methods_list) else: # pragma: no cover, in practice a generic should always have at least one method. methods_str = " " @@ -466,51 +471,80 @@ def _list_multimethods(dispatcher, self_or_cls=None): return list(chain.from_iterable(relevant_registries)) -def _format_method(multimethod): - """Format, as a string, a human-readable description of a multimethod. - - Input format is an item returned by `_list_multimethods`. +# TODO: move this utility to `unpythonic.fun`? Belongs there, but doing so introduces a circular dependency. +def _format_callable(thecallable): + """Format, as a string, a human-readable description of a callable. The returned string includes the call signature, and the source filename and starting line number. This output format takes a page from Julia, with some artistic liberty. """ - thecallable, type_signature = multimethod # Our `type_signature` is based on `typing.get_type_hints`, # but for the error message, we need something that formats # like source code. Hence we use `inspect.signature`. thesignature = inspect.signature(thecallable) - function, _ = getfunc(thecallable) + function, _ = getfunc(thecallable) # raw function for OOP methods, too + # TODO: Python 3.8: filename sometimes detected incorrectly + # - This is because `inspect.getsourcefile` uses `inspect.getfile`, which looks at + # the `co_filename` of the code object. If the function is decorated, then it sees + # the source file where the decorator was defined, not the original function. filename = inspect.getsourcefile(function) source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" -def _resolve_multimethod(dispatcher, args, kwargs): - """Find the first matching multimethod on `dispatcher` for the given `args` and `kwargs`.""" +def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): + """Return the first matching multimethod on `dispatcher` for the given `args` and `kwargs`. + + If `partial` is `True`, allow leaving some parameters of the function unbound, + and return the first multimethod that matches the given partial `args` and `kwargs`. + + The partial mode is useful for type-checking arguments for partial application of a generic + function. If any multimethod matches (this function returns something other than `None`), + then the generic function can accept those partial arguments. + + Note we can only dispatch, i.e. determine which multimethod is the one to be called, + only once we have full (non-partial) `args` and `kwargs`, because in general the + remaining not-yet-passed `args` or `kwargs` may cause the search to match a different + multimethod. + """ multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) for thecallable, type_signature in multimethods: try: - bound_arguments = resolve_bindings(thecallable, *args, **kwargs) + if _partial: + bound_arguments = resolve_bindings_partial(thecallable, *args, **kwargs) + else: + bound_arguments = resolve_bindings(thecallable, *args, **kwargs) except TypeError: # arity mismatch, so this multimethod is not acceptable for the given args/kwargs. continue - if _match_argument_types(type_signature, bound_arguments): + if not _get_argument_type_mismatches(type_signature, bound_arguments): return thecallable return None -def _match_argument_types(type_signature, bound_arguments): +def _get_argument_type_mismatches(type_signature, bound_arguments): """Match bound arguments against the given type signature. - Return whether the arguments match the signature. + Return a list of type mismatches. If it is empty, everything is ok. + When not, each item is of the form `(parameter, value, expected_type)`. + + `type_signature`: in the format returned by `typing.get_type_hints`. + + Is allowed to contain additional items not present + in `bound_arguments`, useful for type-checking during + partial application. - type_signature`: in the format returned by `typing.get_type_hints`. `bound_arguments`: see `unpythonic.arity.resolve_bindings`. + + `type_signature` must contain a corresponding parameter + for each argument in `bound_arguments`. (This is already + checked by `resolve_bindings`.) """ + mismatches = [] for parameter, value in bound_arguments.arguments.items(): assert parameter in type_signature # resolve_bindings should already TypeError when not. expected_type = type_signature[parameter] if not isoftype(value, expected_type): - return False - return True + mismatches.append((parameter, value, expected_type)) + return mismatches def _extract_self_or_cls(thecallable, args): """From `thecallable` and positional arguments `args`, extract the value of `self`/`cls`, if any. @@ -538,11 +572,30 @@ def _extract_self_or_cls(thecallable, args): self_or_cls = None return self_or_cls -def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): +def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates, _partial=False): """Raise a `TypeError` regarding a failed multiple dispatch (no matching multimethod). - `candidates`: a list of multimethods that were attempted, but did not match. + `candidates`: list of `(thecallable, type_signature)` that were attempted, but did not match. + `_partial`: if `True`, report a failure in a *partial application*. + if `False`, report a failure in a *call*. """ + # For `@typed` functions, which have just one valid call signature, we can easily + # report which args or kwargs failed to match. + if len(candidates) == 1: + # TODO: There's some repeated error-reporting code in `unpythonic.fun`. + thecallable, type_signature = candidates[0] + if _partial: + bound_arguments = resolve_bindings_partial(thecallable, *args, **kwargs) + else: + bound_arguments = resolve_bindings(thecallable, *args, **kwargs) + mismatches = _get_argument_type_mismatches(type_signature, bound_arguments) + mismatches_list = [f"{parameter}={repr(value)}, expected {expected_type}" + for parameter, value, expected_type in mismatches] + mismatches_str = "; ".join(mismatches_list) + one_multimethod_msg_str = f"\nParameter binding(s) do not match type specification: {mismatches_str}" + else: + one_multimethod_msg_str = "" + # TODO: It would be nice to show the type signature of the args actually given, # TODO: but in the general case this is difficult. We can't just `type(x)`, since # TODO: the signature may specify something like `Sequence[int]`. Knowing a `list` @@ -550,13 +603,22 @@ def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates): # TODO: was expected. The actual value at least implicitly contains the type information. args_list = [repr(x) for x in args] args_str = ", ".join(args_list) + if _partial and args_str: + args_str += ", ..." sep = ", " if args and kwargs else "" kws_list = [f"{k}={repr(v)}" for k, v in kwargs.items()] kws_str = ", ".join(kws_list) - methods_list = [f" {_format_method(x)}" for x in candidates] + if _partial and kws_str: + kws_str += ", ..." + if _partial and not args_str and not kws_str: + args_str = "..." + thecallables = [thecallable for thecallable, type_signature in candidates] + methods_list = [f" {_format_callable(x)}" for x in thecallables] methods_str = "\n".join(methods_list) - msg = (f"No multiple-dispatch match for the call {dispatcher.__qualname__}({args_str}{sep}{kws_str}).\n" - f"Multimethods for @{isgeneric(dispatcher)} {_function_fullname(dispatcher)} (most recent match attempt last):\n{methods_str}") + op = "partial application" if _partial else "call" + msg = (f"No multiple-dispatch match for the {op} {dispatcher.__qualname__}({args_str}{sep}{kws_str}).\n" + f"Multimethods for @{isgeneric(dispatcher)} {_function_fullname(dispatcher)} (most recent match attempt last):\n{methods_str}" + f"{one_multimethod_msg_str}") raise TypeError(msg) def _setup(fullname, multimethod): @@ -564,16 +626,16 @@ def _setup(fullname, multimethod): This is a low-level function; you'll likely want `@generic` or `@augment`. - fullname: str, fully qualified name of function to register the multimethod - on, used as key in the dispatcher registry. + `fullname`: str, fully qualified name of function to register the multimethod + on, used as key in the dispatcher registry. - Registering the first multimethod on a given `fullname` makes - that function generic, and creates the dispatcher for it. + Registering the first multimethod on a given `fullname` makes + that function generic, and creates the dispatcher for it. - Second and further registrations using the same `fullname` add - the new multimethod to the existing dispatcher. + Second and further registrations using the same `fullname` add + the new multimethod to the existing dispatcher. - multimethod: callable, the new multimethod to register. + `multimethod`: callable, the new multimethod to register. Return value is the dispatcher. """ @@ -660,7 +722,7 @@ def _register_to(dispatcher, multimethod): # Update entry point docstring to include docs for the new multimethod, # and its call signature. - call_signature_desc = _format_method((multimethod, type_signature)) + call_signature_desc = _format_callable(multimethod) our_doc = call_signature_desc if multimethod.__doc__: our_doc += "\n" + multimethod.__doc__ diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 925574b3..07e561fe 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -17,11 +17,15 @@ "to1st", "to2nd", "tokth", "tolast", "to", "withself"] -from functools import wraps, partial +from functools import wraps, partial as functools_partial +from typing import get_type_hints -from .arity import arities, resolve_bindings, tuplify_bindings, UnknownArity +from .arity import (arities, resolve_bindings, resolve_bindings_partial, + tuplify_bindings, UnknownArity) from .fold import reducel -from .dispatch import isgeneric, _resolve_multimethod +from .dispatch import (isgeneric, _resolve_multimethod, _format_callable, + _get_argument_type_mismatches, _raise_multiple_dispatch_error, + _list_multimethods, _extract_self_or_cls) from .dynassign import dyn, make_dynvar from .regutil import register_decorator from .symbol import sym @@ -72,6 +76,66 @@ def memoized(*args, **kwargs): # return memo[k] # return memoized +# Parameter naming is consistent with `functools.partial`. +# +# Note standard behavior of `functools.partial`: `kwargs` do not disappear from the call +# signature even if partially applied. The same kwarg can be sent multiple times, with the +# latest application winning. We must resist the temptation to override that behavior here, +# because there are other places in the stdlib, particularly `inspect._signature_get_partial` +# (as of Python 3.8), that expect the standard semantics. +def partial(func, *args, **kwargs): + """Wrapper over `functools.partial` that type-checks the arguments against the type annotations on `func`. + + The type annotations may use features from the `typing` stdlib module. + See `unpythonic.typecheck.isoftype` for details. + + Trying to pass an argument of a type that does not match the corresponding + parameter's type specification raises `TypeError` immediately. + + Note the check still occurs at run time, but at the use site of `partial`, + when the partially applied function is constructed. This makes it fail-faster + than an `isinstance` check inside the function. + + To conveniently make regular calls of the function type-check arguments, too, + see the decorator `unpythonic.dispatch.typed`. + """ + # HACK: As of Python 3.8, `typing.get_type_hints` does not know about `functools.partial` objects, + # HACK: but those objects have `args` and `keywords` attributes, so we can extract what we need. + # TODO: Remove this hack if `typing.get_type_hints` gets support for `functools.partial` at some point. + if isinstance(func, functools_partial): + thecallable = func.func + collected_args = func.args + args + collected_kwargs = {**func.keywords, **kwargs} + else: + thecallable = func + collected_args = args + collected_kwargs = kwargs + + if isgeneric(thecallable): # multiple dispatch + # For generic functions, at least one multimethod must match the partial signature + # for the partial application to be valid. + if not _resolve_multimethod(thecallable, collected_args, collected_kwargs, _partial=True): + _raise_multiple_dispatch_error(thecallable, collected_args, collected_kwargs, + candidates=_list_multimethods(thecallable, + _extract_self_or_cls(thecallable, + args)), + _partial=True) + else: # not `@generic` or `@typed`; just a function that has type annotations. + # TODO: There's some repeated error-reporting code in `unpythonic.dispatch`. + type_signature = get_type_hints(thecallable) + if type_signature: # TODO: Python 3.8+: use walrus assignment here + bound_arguments = resolve_bindings_partial(func, *collected_args, **collected_kwargs) + mismatches = _get_argument_type_mismatches(type_signature, bound_arguments) + if mismatches: + description = _format_callable(func) + mismatches_list = [f"{parameter}={repr(value)}, expected {expected_type}" + for parameter, value, expected_type in mismatches] + mismatches_str = "; ".join(mismatches_list) + raise TypeError(f"When partially applying {description}:\nParameter binding(s) do not match type specification: {mismatches_str}") + + # `functools.partial` already handles chaining partial applications, so send only the new args/kwargs to it. + return functools_partial(func, *args, **kwargs) + make_dynvar(curry_context=[]) @passthrough_lazy_args def _currycall(f, *args, **kwargs): @@ -100,9 +164,11 @@ def curry(f, *args, _curry_force_call=False, _curry_allow_uninspectable=False, * decision when the function is called. For a callable to be curryable, it must be possible to inpect its signature - to determine its minimum and maximum positional arities; builtin functions - such as ``operator.add`` won't work. In such cases ``UnknownArity`` will - be raised. + to determine its minimum and maximum positional arities. In some versions + of Python, builtin functions or methods such as ``operator.add`` or + ``list.append`` might not report that information. We work around some of + the most common cases (see the module `unpythonic.arity`), but when the + inspection fails and no workaround is available, we raise ``UnknownArity``. **Examples**:: @@ -199,7 +265,8 @@ def foo(a, b, *, c, d): return maybe_force_args(f, *args, **kwargs) return f # TODO: improve: all required name-only args should be present before calling f. - # Difficult, partial() doesn't remove an already-set kwarg from the signature. + # TODO: `functools.partial()` doesn't remove an already-set kwarg from the signature, but + # TODO: `functools.partial` objects have a `keywords` attribute, which contains what we want. try: min_arity, max_arity = arities(f) except UnknownArity: # likely a builtin @@ -215,15 +282,18 @@ def curried(*args, **kwargs): with dyn.let(curry_context=(outerctx + [f])): # If `f` is `@generic` (see `unpythonic.dispatch`), and `min_arity <= len(args) <= max_arity`, # we need to check for a multimethod match. If there is no match, it means that the arguments - # provided so far don't satisfy any registered multimethod call signature, but more arguments - # can still be accepted by the other call signatures. In that case, there are effectively - # too few arguments. + # provided so far don't satisfy any registered multimethod call signature (fully), but more + # arguments can still be accepted by the other call signatures. In that case, there are + # effectively too few arguments. # # For `@typed`, there is only one call signature, so the arity is the only important factor. nargs = len(args) if (nargs < min_arity or (isgeneric(f) == "generic" and min_arity <= nargs <= max_arity and not _resolve_multimethod(f, args, kwargs))): + # Fail-fast: use our `partial` wrapper to type-check the partial call signature + # when we build the curried function. It delegates to `functools.partial` if the + # type check passes. p = partial(f, *args, **kwargs) if islazy(f): p = passthrough_lazy_args(p) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 62e5ebba..a57f734f 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -109,6 +109,31 @@ def runtests(): test[kittify(x=1.0, y=2.0) == "float"] test_raises[TypeError, kittify(x=1, y=2.0)] + with testset("@generic integration with curry"): + @generic + def curryable(x: int, y: int): + return "int" + @generic + def curryable(x: float, y: float): # noqa: F811 + return "float" + f = curry(curryable, 1) + test[callable(the[f])] + test[f(2) == "int"] + + # When the final set of arguments does not match any multimethod, it is a type error. + test_raises[TypeError, f(2.0)] + + # CAUTION: Partially applying by name starts keyword-only processing in `inspect.signature`, + # which is used by `unpythonic.arity.arities`, which in turn is used by `unpythonic.fun.curry`. + # Hence, if we pass `x=1` by name here, the remaining positional arity becomes 0... + f = curry(curryable, x=1) + test[callable(the[f])] + # ...so, we must pass `y` by name here. + test[f(y=2) == "int"] + + # When no multimethod can match the given partial signature, it is a type error. + test_raises[TypeError, curry(curryable, "abc")] + with testset("@augment"): @generic def f1(x: typing.Any): @@ -281,10 +306,8 @@ def errorcase2(x): test[callable(the[f])] test[f(21.0) == 42] - # But be careful: - f = curry(blubnify, 2.0) # wrong argument type; error not triggered yet - test[callable(the[f])] - test_raises[TypeError, f(21.0) == 42] # error will occur now, when the call is triggered + # Wrong argument type during partial application of @typed function - error reported immediately. + test_raises[TypeError, curry(blubnify, 2.0)] with testset("holy traits in Python with @generic"): # Note we won't get the performance benefits of Julia, because this is a diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 91e56f94..12732bdf 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -1,13 +1,13 @@ # -*- coding: utf-8 -*- -from ..syntax import macros, test, test_raises, fail # noqa: F401 -from ..test.fixtures import session, testset +from ..syntax import macros, test, test_raises, fail, the # noqa: F401 +from ..test.fixtures import session, testset, returns_normally from collections import Counter import sys from ..dispatch import generic -from ..fun import (memoize, curry, apply, +from ..fun import (memoize, partial, curry, apply, identity, const, andf, orf, notf, flip, rotate, @@ -134,6 +134,14 @@ def t(): fail["memoize should not prevent exception propagation."] # pragma: no cover test[evaluations == 1] + with testset("partial (type-checking wrapper)"): + def nottypedfunc(x): + return "ok" + def typedfunc(x: int): + return "ok" + test[returns_normally(partial(typedfunc, 42))] + test_raises[TypeError, partial(typedfunc, "abc")] + with testset("@curry"): @curry def add3(a, b, c): @@ -144,6 +152,15 @@ def add3(a, b, c): test[add3(1)(2, 3) == 6] test[add3(1, 2, 3) == 6] + # curry uses the type-checking `partial` + @curry + def add3ints(a: int, b: int, c: int): + return a + b + c + test[add3ints(1)(2)(3) == 6] + test[callable(the[add3ints(1)])] + test_raises[TypeError, add3ints(1.0)] + test_raises[TypeError, add3ints(1)(2.0)] + @curry def lispyadd(*args): return sum(args) @@ -221,8 +238,9 @@ def f(x: float, y: str): # noqa: F811, new multimethod for the same generic fun test[callable(curry(f))] test[curry(f, 42) == "int"] # Although `f` has a multimethod that takes one argument, if that argument is a float, - # the call signature does not match, so in that case `curry` waits for more arguments - # (because it knows `f` has also a multimethod that accepts two arguments). + # the call signature does not match fully. But it does match partially, so in that case + # `curry` waits for more arguments (because there is at least one multimethod that matches + # the partial arguments given so far). test[callable(curry(f, 3.14))] test[curry(f, 3.14, "cat") == "float, str"] From f60120ebffffef03ef024689eac8606c48bee691 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 18 May 2021 00:54:46 +0300 Subject: [PATCH 309/832] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d8c764d..03da00d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -89,6 +89,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - It is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. - `curry` now supports `@generic` functions. - The utilities `arities`, `required_kwargs`, and `optional_kwargs` now support `@generic` functions. + - `curry` now errors out immediately on argument type mismatch. + - Add `partial`, a type-checking wrapper for `functools.partial`. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) From cb0ba1bf9cb6c6d3bd03584978e8b39fac8e61e8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 18 May 2021 01:48:55 +0300 Subject: [PATCH 310/832] update changelog --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 03da00d2..2359306e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,8 +87,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Add function `isgeneric` to detect whether a callable has been declared `@generic`. - Add function `methods`: display a list of multimethods of a generic function. - It is now possible to dispatch on a homogeneous type of contents collected by a `**kwargs` parameter. - - `curry` now supports `@generic` functions. - - The utilities `arities`, `required_kwargs`, and `optional_kwargs` now support `@generic` functions. + - `curry` now supports `@generic` functions. **This feature is experimental. Semantics may still change.** + - The utilities `arities`, `required_kwargs`, and `optional_kwargs` now support `@generic` functions. **This feature is experimental. Semantics may still change.** - `curry` now errors out immediately on argument type mismatch. - Add `partial`, a type-checking wrapper for `functools.partial`. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). From 7398280dd3b16a54ef4082c5d38d62c4d7686ca4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 18 May 2021 01:49:38 +0300 Subject: [PATCH 311/832] Sketch for better kwargs support in curry. Currently does not work fully (some macro tests error or fail), because `print` and `range` have no signature retrievable by `inspect.signature`, and `env.set` is for some reason detected incorrectly (so that it remains curried, though all needed args are present) - maybe supporting OOP methods needs some additional thought. So, archiving this idea for now. --- unpythonic/dispatch.py | 45 ++++++++------ unpythonic/fun.py | 97 ++++++++++++++++++++++++------- unpythonic/tests/test_dispatch.py | 4 ++ 3 files changed, 109 insertions(+), 37 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index 36178d3c..f5c955d1 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -31,7 +31,7 @@ import inspect import typing -from .arity import getfunc, resolve_bindings, resolve_bindings_partial +from .arity import getfunc, _resolve_bindings from .typecheck import isoftype from .regutil import register_decorator @@ -492,6 +492,23 @@ def _format_callable(thecallable): source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" +def _bind_and_check_arity(thecallable, args, kwargs, *, _partial=False): + try: + bound_arguments = _resolve_bindings(thecallable, args, kwargs, _partial=_partial) + except TypeError as err: + # TODO: searching the error message for particular text is a big HACK. + # But `curry` needs to know why the match failed. + msg = err.args[0] + if "too many" in msg: # too many positional args supplied + return "too many args" + elif "unexpected" in msg: # unexpected named arg supplied + return "unexpected kwarg" + elif "missing" in msg: # at least one parameter not bound + return "unbound parameter" + else: + raise NotImplementedError from err + return bound_arguments + def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): """Return the first matching multimethod on `dispatcher` for the given `args` and `kwargs`. @@ -502,21 +519,16 @@ def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): function. If any multimethod matches (this function returns something other than `None`), then the generic function can accept those partial arguments. - Note we can only dispatch, i.e. determine which multimethod is the one to be called, - only once we have full (non-partial) `args` and `kwargs`, because in general the - remaining not-yet-passed `args` or `kwargs` may cause the search to match a different - multimethod. + Note it is only possible to dispatch, i.e. determine which multimethod is the one to be + called, only once we have full (non-partial) `args` and `kwargs`, because in general + the remaining not-yet-passed `args` or `kwargs` may cause the search to match a + different multimethod. """ multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) for thecallable, type_signature in multimethods: - try: - if _partial: - bound_arguments = resolve_bindings_partial(thecallable, *args, **kwargs) - else: - bound_arguments = resolve_bindings(thecallable, *args, **kwargs) - except TypeError: # arity mismatch, so this multimethod is not acceptable for the given args/kwargs. - continue - if not _get_argument_type_mismatches(type_signature, bound_arguments): + bound_arguments = _bind_and_check_arity(thecallable, args, kwargs, _partial=_partial) + if (isinstance(bound_arguments, inspect.BoundArguments) and + not _get_argument_type_mismatches(type_signature, bound_arguments)): return thecallable return None @@ -528,6 +540,8 @@ def _get_argument_type_mismatches(type_signature, bound_arguments): `type_signature`: in the format returned by `typing.get_type_hints`. + Must contain an item for each key of `bound_arguments`. + Is allowed to contain additional items not present in `bound_arguments`, useful for type-checking during partial application. @@ -584,10 +598,7 @@ def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates, _par if len(candidates) == 1: # TODO: There's some repeated error-reporting code in `unpythonic.fun`. thecallable, type_signature = candidates[0] - if _partial: - bound_arguments = resolve_bindings_partial(thecallable, *args, **kwargs) - else: - bound_arguments = resolve_bindings(thecallable, *args, **kwargs) + bound_arguments = _resolve_bindings(thecallable, args, kwargs, _partial=_partial) mismatches = _get_argument_type_mismatches(type_signature, bound_arguments) mismatches_list = [f"{parameter}={repr(value)}, expected {expected_type}" for parameter, value, expected_type in mismatches] diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 07e561fe..e08ee51b 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -18,14 +18,16 @@ "withself"] from functools import wraps, partial as functools_partial +import inspect from typing import get_type_hints -from .arity import (arities, resolve_bindings, resolve_bindings_partial, - tuplify_bindings, UnknownArity) +from .arity import (arities, _resolve_bindings, tuplify_bindings, + UnknownArity) from .fold import reducel from .dispatch import (isgeneric, _resolve_multimethod, _format_callable, _get_argument_type_mismatches, _raise_multiple_dispatch_error, - _list_multimethods, _extract_self_or_cls) + _list_multimethods, _extract_self_or_cls, + _bind_and_check_arity) from .dynassign import dyn, make_dynvar from .regutil import register_decorator from .symbol import sym @@ -51,7 +53,7 @@ def memoize(f): memo = {} @wraps(f) def memoized(*args, **kwargs): - k = tuplify_bindings(resolve_bindings(f, *args, **kwargs)) + k = tuplify_bindings(_resolve_bindings(f, args, kwargs, _partial=False)) if k not in memo: try: result = (_success, maybe_force_args(f, *args, **kwargs)) @@ -124,7 +126,8 @@ def partial(func, *args, **kwargs): # TODO: There's some repeated error-reporting code in `unpythonic.dispatch`. type_signature = get_type_hints(thecallable) if type_signature: # TODO: Python 3.8+: use walrus assignment here - bound_arguments = resolve_bindings_partial(func, *collected_args, **collected_kwargs) + bound_arguments = _resolve_bindings(func, collected_args, collected_kwargs, _partial=True) + # TODO: allow having some parameters without type annotations. mismatches = _get_argument_type_mismatches(type_signature, bound_arguments) if mismatches: description = _format_callable(func) @@ -264,9 +267,6 @@ def foo(a, b, *, c, d): if args or kwargs or _curry_force_call: return maybe_force_args(f, *args, **kwargs) return f - # TODO: improve: all required name-only args should be present before calling f. - # TODO: `functools.partial()` doesn't remove an already-set kwarg from the signature, but - # TODO: `functools.partial` objects have a `keywords` attribute, which contains what we want. try: min_arity, max_arity = arities(f) except UnknownArity: # likely a builtin @@ -280,17 +280,65 @@ def foo(a, b, *, c, d): def curried(*args, **kwargs): outerctx = dyn.curry_context with dyn.let(curry_context=(outerctx + [f])): - # If `f` is `@generic` (see `unpythonic.dispatch`), and `min_arity <= len(args) <= max_arity`, - # we need to check for a multimethod match. If there is no match, it means that the arguments - # provided so far don't satisfy any registered multimethod call signature (fully), but more - # arguments can still be accepted by the other call signatures. In that case, there are - # effectively too few arguments. + # All of `f`'s parameters should be bound (whether by position or by name) before calling `f`. # - # For `@typed`, there is only one call signature, so the arity is the only important factor. - nargs = len(args) - if (nargs < min_arity or - (isgeneric(f) == "generic" and min_arity <= nargs <= max_arity and - not _resolve_multimethod(f, args, kwargs))): + # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by + # `inspect.signature`, used by `unpythonic.arity.arities`), but `functools.partial` objects + # have a `keywords` attribute, which contains what we want. + # + # Now that we must test argument bindings anyway (to support kwargs properly), we also use + # the `func` and `args` attributes. This allows us to test the bindings on the underlying function. + if isinstance(f, functools_partial): + function = f.func + collected_args = f.args + args + collected_kwargs = {**f.keywords, **kwargs} + else: + function = f + collected_args = args + collected_kwargs = kwargs + + # The `type_signature` is used for `@generic` and `@typed` functions. + def match_arguments(thecallable, type_signature=None): + bound_arguments = _bind_and_check_arity(thecallable, collected_args, + collected_kwargs, _partial=False) + if isinstance(bound_arguments, inspect.BoundArguments): # success + # The parameter types in the call signature affect multiple-dispatching, + # so we must type-check, too. + if not type_signature or not _get_argument_type_mismatches(type_signature, bound_arguments): + return "ok" + return "argument type mismatch" + return bound_arguments # error code + + if not isgeneric(function): + status = match_arguments(function) + else: + results = set() + multimethods = _list_multimethods(function, + _extract_self_or_cls(function, + collected_args)) + for thecallable, type_signature in multimethods: + result = match_arguments(thecallable, type_signature) + results.add(result) + if result == "ok": + break + # Any multimethod that can bind all args and kwargs (without type errors) is a match; + # prefer that first. + if "ok" in results: + status = "ok" + # No match. Figure out if we have too few or too many args/kwargs. + # At least one multimethod can accept more args or kwargs. Prefer that next. + elif "unbound parameter" in results: + status = "unbound parameter" + # Then prefer the case with too many positionals (and hope there aren't unexpected kwargs, too). + elif "too many args" in results: + status = "too many args" + elif "unexpected kwarg" in results or "argument type mismatch" in results: + _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, + candidates=multimethods, _partial=True) + else: # all cases should be accounted for # pragma: no cover + assert False + + if status == "unbound parameter": # at least one parameter not bound yet; wait for more args/kwargs # Fail-fast: use our `partial` wrapper to type-check the partial call signature # when we build the curried function. It delegates to `functools.partial` if the # type check passes. @@ -298,8 +346,9 @@ def curried(*args, **kwargs): if islazy(f): p = passthrough_lazy_args(p) return curry(p) - # passthrough on right, like https://github.com/Technologicat/spicy - if nargs > max_arity: + + # Too many positional args; passthrough on right, like https://github.com/Technologicat/spicy + elif status == "too many args": now_args, later_args = args[:max_arity], args[max_arity:] now_result = maybe_force_args(f, *now_args, **kwargs) # use up all kwargs now now_result = force(now_result) if not isinstance(now_result, tuple) else force1(now_result) @@ -316,6 +365,14 @@ def curried(*args, **kwargs): if isinstance(now_result, tuple): return now_result + later_args return (now_result,) + later_args + + # Unexpected kwarg, could not be bound to any parameter + elif status == "unexpected kwarg": + # TODO: report the unexpected kwarg(s) + raise NotImplementedError("curry: cannot pass-through unexpected named args") + + # All parameters bound to some arg or kwarg + assert status == "ok" return maybe_force_args(f, *args, **kwargs) if islazy(f): curried = passthrough_lazy_args(curried) diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index a57f734f..fb31df1e 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -131,6 +131,10 @@ def curryable(x: float, y: float): # noqa: F811 # ...so, we must pass `y` by name here. test[f(y=2) == "int"] + f = curry(curryable, 1) + test[callable(the[f])] + test[f(y=2) == "int"] + # When no multimethod can match the given partial signature, it is a type error. test_raises[TypeError, curry(curryable, "abc")] From b3eb3c87b53334c08e8ff7ff804543d30e98fb7b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 18 May 2021 02:46:53 +0300 Subject: [PATCH 312/832] fix silly mistake Now OOP methods work with the new `curry`. We should be very careful where to get the raw function and where to just deal with a bound OOP method. --- unpythonic/arity.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/unpythonic/arity.py b/unpythonic/arity.py index 0d21b369..6308faa8 100644 --- a/unpythonic/arity.py +++ b/unpythonic/arity.py @@ -355,11 +355,8 @@ def resolve_bindings(f, *args, **kwargs): our own implementation of the parameter binding algorithm (that ran also on Python 3.4), but it is no longer needed, since now we support only Python 3.6 and later. - The only things we do beside call `inspect.Signature.bind` are: - - - If `f` is a method, we extract the raw function first, and analyze the bindings of that. - - - We apply default values (from the definition of `f`) automatically. + The only thing we do beside call `inspect.Signature.bind` is that we apply default values + (from the definition of `f`) automatically. The return value is an `inspect.BoundArguments`. If you want a hashable result, postprocess the return value with `tuplify_bindings(result)`. @@ -413,7 +410,6 @@ def f(a): return _resolve_bindings(f, args, kwargs, _partial=False) def _resolve_bindings(f, args, kwargs, *, _partial): - f, _ = getfunc(f) thesignature = signature(f) if _partial: bound_arguments = thesignature.bind_partial(*args, **kwargs) From 2960e71c7d0eb6a744cfea0fcdd1b65daaf2e14b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 18 May 2021 12:38:56 +0300 Subject: [PATCH 313/832] Fix and improve kwargs handling in `curry` Now uninspectable builtins error out, like they did previously. The algorithm is different and we use a different part of `inspect`, so it may now be a *different subset* of builtins that has become uninspectable. Now all tests pass. --- unpythonic/dispatch.py | 28 ++----- unpythonic/fun.py | 172 +++++++++++++++++++++++++++-------------- 2 files changed, 120 insertions(+), 80 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index f5c955d1..f2dd5ccd 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -488,27 +488,11 @@ def _format_callable(thecallable): # - This is because `inspect.getsourcefile` uses `inspect.getfile`, which looks at # the `co_filename` of the code object. If the function is decorated, then it sees # the source file where the decorator was defined, not the original function. +# function = inspect.unwrap(function) # maybe this helps? filename = inspect.getsourcefile(function) source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" -def _bind_and_check_arity(thecallable, args, kwargs, *, _partial=False): - try: - bound_arguments = _resolve_bindings(thecallable, args, kwargs, _partial=_partial) - except TypeError as err: - # TODO: searching the error message for particular text is a big HACK. - # But `curry` needs to know why the match failed. - msg = err.args[0] - if "too many" in msg: # too many positional args supplied - return "too many args" - elif "unexpected" in msg: # unexpected named arg supplied - return "unexpected kwarg" - elif "missing" in msg: # at least one parameter not bound - return "unbound parameter" - else: - raise NotImplementedError from err - return bound_arguments - def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): """Return the first matching multimethod on `dispatcher` for the given `args` and `kwargs`. @@ -526,10 +510,12 @@ def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): """ multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) for thecallable, type_signature in multimethods: - bound_arguments = _bind_and_check_arity(thecallable, args, kwargs, _partial=_partial) - if (isinstance(bound_arguments, inspect.BoundArguments) and - not _get_argument_type_mismatches(type_signature, bound_arguments)): - return thecallable + try: + bound_arguments = _resolve_bindings(thecallable, args, kwargs, _partial=_partial) + if not _get_argument_type_mismatches(type_signature, bound_arguments): + return thecallable + except TypeError: # could not accept the given arguments; this isn't the multimethod we're looking for. + continue return None def _get_argument_type_mismatches(type_signature, bound_arguments): diff --git a/unpythonic/fun.py b/unpythonic/fun.py index e08ee51b..b4bcaa4b 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -18,7 +18,6 @@ "withself"] from functools import wraps, partial as functools_partial -import inspect from typing import get_type_hints from .arity import (arities, _resolve_bindings, tuplify_bindings, @@ -26,8 +25,7 @@ from .fold import reducel from .dispatch import (isgeneric, _resolve_multimethod, _format_callable, _get_argument_type_mismatches, _raise_multiple_dispatch_error, - _list_multimethods, _extract_self_or_cls, - _bind_and_check_arity) + _list_multimethods, _extract_self_or_cls) from .dynassign import dyn, make_dynvar from .regutil import register_decorator from .symbol import sym @@ -160,18 +158,19 @@ def curry(f, *args, _curry_force_call=False, _curry_allow_uninspectable=False, * """Decorator: curry the function f. Essentially, the resulting function automatically chains partial application - until the minimum positional arity of ``f`` is satisfied, at which point - ``f``is called. + until all parameters of ``f`` are bound, at which point ``f`` is called. - Also more kwargs can be passed at each step, but they do not affect the - decision when the function is called. + For a callable to be curryable, its signature must be inspectable by the stdlib + function `inspect.signature`. In some versions of Python, inspection may fail + for builtin functions or methods such as ``print``, ``range``, ``operator.add``, + or ``list.append``. - For a callable to be curryable, it must be possible to inpect its signature - to determine its minimum and maximum positional arities. In some versions - of Python, builtin functions or methods such as ``operator.add`` or - ``list.append`` might not report that information. We work around some of - the most common cases (see the module `unpythonic.arity`), but when the - inspection fails and no workaround is available, we raise ``UnknownArity``. + **CAUTION**: Up to v0.14.3, we looked at positional arity only, and there were + workarounds in place for some of the most common builtins. As of v0.15.0, we + compute argument bindings like Python itself does. Hence we use a different + algorithm, and thus a *different subset* of builtins may have become uninspectable. + + When inspection fails, we raise ``unpythonic.arity.UnknownArity``. **Examples**:: @@ -247,16 +246,32 @@ def foo(a, b, *, c, d): clip = lambda n1, n2: composel(*with_n((n1, drop), (n2, take))) assert tuple(curry(clip, 5, 10, range(20))) == tuple(range(5, 15)) - **CAUTION**: BUG: `curry` may fail to actually call the function even after - sufficient arguments have been collected, if some of the positional-or-keyword - arguments of the function being curried are passed by name (in the first call). - It seems those arguments don't reduce the expected remaining positional arity, - although they should. See issue #61: - https://github.com/Technologicat/unpythonic/issues/61 + **Kwargs support**: + + As of v0.15.0, `curry` supports passing arguments by name at any step during the currying. + + We collect both `args` and `kwargs` across all steps, and bind arguments to function + parameters the same way Python itself does, so it shouldn't matter whether the function + parameters end up bound by position or name. When all parameters have a binding, the call + triggers. + + That means, for example, that this now works as expected:: + + @curry + def f(x, y): + return x, y + + assert f(y=2)(x=1) == (1, 2) + + However, it is possible that the algorithm isn't perfect, so there may be small semantic + differences to regular one-step function calls. If you find any, please file an issue, + so these can at the very least be documented; and if doable with reasonable effort, + preferably fixed. - **Workaround**: if possible, at the definition site for your function, declare - any arguments you plan to pass by name as keyword-only; then they won't affect - the positional arity. + It is still an error if named arguments are left over when the top-level curry context + is reached. Treating this case would require generalizing return values so that functions + could return named outputs. See: + https://github.com/Technologicat/unpythonic/issues/32 """ f = force(f) # lazify support: we need the value of f # trivial case first: interaction with call_ec and other replace-def-with-value decorators @@ -267,27 +282,36 @@ def foo(a, b, *, c, d): if args or kwargs or _curry_force_call: return maybe_force_args(f, *args, **kwargs) return f - try: - min_arity, max_arity = arities(f) - except UnknownArity: # likely a builtin + + def fallback(): # what to do if inspection fails if not _curry_allow_uninspectable: # usual behavior raise # co-operate with unpythonic.syntax.autocurry; don't crash on builtins if args or kwargs or _curry_force_call: return maybe_force_args(f, *args, **kwargs) return f + + try: + min_arity, max_arity = arities(f) + except UnknownArity: # likely a builtin + return fallback() + @wraps(f) def curried(*args, **kwargs): outerctx = dyn.curry_context with dyn.let(curry_context=(outerctx + [f])): + # In order to decide what to do when the curried function is called, we first compute the + # parameter bindings. + # # All of `f`'s parameters should be bound (whether by position or by name) before calling `f`. # # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by # `inspect.signature`, used by `unpythonic.arity.arities`), but `functools.partial` objects # have a `keywords` attribute, which contains what we want. # - # Now that we must test argument bindings anyway (to support kwargs properly), we also use - # the `func` and `args` attributes. This allows us to test the bindings on the underlying function. + # To support kwargs properly, we must compute argument bindings anyway, so we also use the + # `func` and `args` attributes. This allows us to compute the bindings against the original + # function. if isinstance(f, functools_partial): function = f.func collected_args = f.args + args @@ -299,9 +323,25 @@ def curried(*args, **kwargs): # The `type_signature` is used for `@generic` and `@typed` functions. def match_arguments(thecallable, type_signature=None): - bound_arguments = _bind_and_check_arity(thecallable, collected_args, + try: + bound_arguments = _resolve_bindings(thecallable, collected_args, collected_kwargs, _partial=False) - if isinstance(bound_arguments, inspect.BoundArguments): # success + except TypeError as err: + # TODO: Searching the error message for a particular text snippet is a big HACK, + # TODO: but we need to know *why* the arguments could not be bound. + msg = err.args[0] + if "too many" in msg: # too many positional args supplied + return "too many args" + elif "unexpected" in msg: # unexpected named arg supplied + return "unexpected kwarg" + elif "missing" in msg: # at least one parameter not bound + return "unbound parameter" + elif "multiple values" in msg: # attempted to bind a parameter to more than one value + # This is a `TypeError` for regular calls, too, so let it propagate. + raise + else: # we should have accounted for all cases # pragma: no cover + raise NotImplementedError from err + else: # The parameter types in the call signature affect multiple-dispatching, # so we must type-check, too. if not type_signature or not _get_argument_type_mismatches(type_signature, bound_arguments): @@ -309,39 +349,51 @@ def match_arguments(thecallable, type_signature=None): return "argument type mismatch" return bound_arguments # error code - if not isgeneric(function): - status = match_arguments(function) - else: - results = set() - multimethods = _list_multimethods(function, - _extract_self_or_cls(function, - collected_args)) - for thecallable, type_signature in multimethods: - result = match_arguments(thecallable, type_signature) - results.add(result) - if result == "ok": - break - # Any multimethod that can bind all args and kwargs (without type errors) is a match; - # prefer that first. - if "ok" in results: - status = "ok" - # No match. Figure out if we have too few or too many args/kwargs. - # At least one multimethod can accept more args or kwargs. Prefer that next. - elif "unbound parameter" in results: - status = "unbound parameter" - # Then prefer the case with too many positionals (and hope there aren't unexpected kwargs, too). - elif "too many args" in results: - status = "too many args" - elif "unexpected kwarg" in results or "argument type mismatch" in results: - _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, - candidates=multimethods, _partial=True) - else: # all cases should be accounted for # pragma: no cover - assert False + # `@generic` functions have several call signatures, so we must aggregate the results + # in some sensible way to decide what to do. For non-generics, there's just one call signature. + try: + if not isgeneric(function): + status = match_arguments(function) + else: + results = set() + # We can't use the public `list_methods` here, because on OOP methods, + # decorators live on the unbound method (raw function). Thus we must + # extract `self`/`cls` from the arguments of the call (for linked + # dispatcher lookup in the MRO). + multimethods = _list_multimethods(function, + _extract_self_or_cls(function, + collected_args)) + for thecallable, type_signature in multimethods: + result = match_arguments(thecallable, type_signature) + results.add(result) + if result == "ok": + break + # Any multimethod that can bind all collected args and kwargs (without type errors) + # is a match; prefer that first. + if "ok" in results: + status = "ok" + # No match. Figure out if we have too few or too many args/kwargs. + # If at least one multimethod can accept more args or kwargs, prefer that next. + elif "unbound parameter" in results: + status = "unbound parameter" + # Then prefer the case with too many positionals (and hope there aren't unexpected kwargs, too). + elif "too many args" in results: + status = "too many args" + elif "unexpected kwarg" in results or "argument type mismatch" in results: + _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, + candidates=multimethods, _partial=True) + else: # all cases should be accounted for # pragma: no cover + assert False + except ValueError as err: # inspect.Signature.bind(), via our _resolve_bindings() + msg = err.args[0] + if "no signature found" in msg: + return fallback() + raise if status == "unbound parameter": # at least one parameter not bound yet; wait for more args/kwargs # Fail-fast: use our `partial` wrapper to type-check the partial call signature # when we build the curried function. It delegates to `functools.partial` if the - # type check passes. + # type check passes, and else raises a `TypeError` immediately. p = partial(f, *args, **kwargs) if islazy(f): p = passthrough_lazy_args(p) @@ -349,6 +401,8 @@ def match_arguments(thecallable, type_signature=None): # Too many positional args; passthrough on right, like https://github.com/Technologicat/spicy elif status == "too many args": + # TODO: The uniform thing to do would be to pass on any arguments that weren't + # TODO: bound to any parameter, regardless if they were passed positionally or by name. now_args, later_args = args[:max_arity], args[max_arity:] now_result = maybe_force_args(f, *now_args, **kwargs) # use up all kwargs now now_result = force(now_result) if not isinstance(now_result, tuple) else force1(now_result) @@ -372,7 +426,7 @@ def match_arguments(thecallable, type_signature=None): raise NotImplementedError("curry: cannot pass-through unexpected named args") # All parameters bound to some arg or kwarg - assert status == "ok" + assert status == "ok", status return maybe_force_args(f, *args, **kwargs) if islazy(f): curried = passthrough_lazy_args(curried) From 55d08a06fb4aafb2069602c76b75207e308482a8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 18 May 2021 12:44:40 +0300 Subject: [PATCH 314/832] improve comments --- unpythonic/fun.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index b4bcaa4b..507ba2cc 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -120,12 +120,18 @@ def partial(func, *args, **kwargs): _extract_self_or_cls(thecallable, args)), _partial=True) - else: # not `@generic` or `@typed`; just a function that has type annotations. + else: # Not `@generic` or `@typed`; just a function that has type annotations. + # It's not very unpythonic-ic to provide this since we already have `@typed` for this use case, + # but it's much more pythonic, if the type-checking `partial` works properly for code that does + # not opt in to `unpythonic`'s multiple-dispatch subsystem. # TODO: There's some repeated error-reporting code in `unpythonic.dispatch`. type_signature = get_type_hints(thecallable) if type_signature: # TODO: Python 3.8+: use walrus assignment here bound_arguments = _resolve_bindings(func, collected_args, collected_kwargs, _partial=True) - # TODO: allow having some parameters without type annotations. + # TODO: Allow having some parameters without type annotations. Requiring them for all + # TODO: parameters is a `@generic`-ism (because it uses them for dispatching). + # TODO: Alternatively, generalize `@generic` to ignore types for arguments whose parameters + # TODO: have no type annotation. But as said in the comments, that could be a footgun. mismatches = _get_argument_type_mismatches(type_signature, bound_arguments) if mismatches: description = _format_callable(func) From 4b8498df82ae293c9fde3df05bdbccefed2fb2ad Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 19 May 2021 17:22:34 +0300 Subject: [PATCH 315/832] fix whitespace --- unpythonic/fun.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 507ba2cc..ac98d4ac 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -457,7 +457,7 @@ def iscurried(f): def flip(f): """Decorator: flip (reverse) the positional arguments of f.""" - @ wraps(f) + @wraps(f) def flipped(*args, **kwargs): return maybe_force_args(f, *reversed(args), **kwargs) if islazy(f): @@ -480,7 +480,7 @@ def rotate(k): assert (rotate(1)(identity))(1, 2, 3) == (2, 3, 1) """ def rotate_k(f): - @ wraps(f) + @wraps(f) def rotated(*args, **kwargs): n = len(args) if not n: @@ -495,7 +495,7 @@ def rotated(*args, **kwargs): return rotated return rotate_k -@ passthrough_lazy_args +@passthrough_lazy_args def apply(f, arg0, *more, **kwargs): """Scheme/Racket-like apply. @@ -844,7 +844,7 @@ def to(*specs): """ return composeli(tokth(k, f) for k, f in specs) -@ register_decorator(priority=80) +@register_decorator(priority=80) def withself(f): """Decorator. Allow a lambda to refer to itself. @@ -869,7 +869,7 @@ def withself(f): assert fact(5) == 120 fact(5000) # no crash """ - @ wraps(f) + @wraps(f) def fwithself(*args, **kwargs): #return f(fwithself, *args, **kwargs) return maybe_force_args(f, fwithself, *args, **kwargs) # support unpythonic.syntax.lazify From 042499907a58c5dc9f8c26c06664af81675bee1d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 19 May 2021 17:23:09 +0300 Subject: [PATCH 316/832] update comments and docstrings --- unpythonic/dispatch.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index f2dd5ccd..bb7303fa 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -488,7 +488,7 @@ def _format_callable(thecallable): # - This is because `inspect.getsourcefile` uses `inspect.getfile`, which looks at # the `co_filename` of the code object. If the function is decorated, then it sees # the source file where the decorator was defined, not the original function. -# function = inspect.unwrap(function) # maybe this helps? + # function = inspect.unwrap(function) # maybe this helps? But now I can't reproduce the bug to test it. filename = inspect.getsourcefile(function) source, firstlineno = inspect.getsourcelines(function) return f"{thecallable.__qualname__}{str(thesignature)} from {filename}:{firstlineno}" @@ -496,7 +496,7 @@ def _format_callable(thecallable): def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): """Return the first matching multimethod on `dispatcher` for the given `args` and `kwargs`. - If `partial` is `True`, allow leaving some parameters of the function unbound, + If `_partial` is `True`, allow leaving some parameters of the function unbound, and return the first multimethod that matches the given partial `args` and `kwargs`. The partial mode is useful for type-checking arguments for partial application of a generic @@ -506,7 +506,8 @@ def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): Note it is only possible to dispatch, i.e. determine which multimethod is the one to be called, only once we have full (non-partial) `args` and `kwargs`, because in general the remaining not-yet-passed `args` or `kwargs` may cause the search to match a - different multimethod. + different multimethod. In partial mode, this function says only that there is + *at least one* match when given those partial arguments. """ multimethods = _list_multimethods(dispatcher, _extract_self_or_cls(dispatcher, args)) for thecallable, type_signature in multimethods: @@ -573,7 +574,7 @@ def _extract_self_or_cls(thecallable, args): return self_or_cls def _raise_multiple_dispatch_error(dispatcher, args, kwargs, *, candidates, _partial=False): - """Raise a `TypeError` regarding a failed multiple dispatch (no matching multimethod). + """Raise a nicely formatted `TypeError` regarding a failed multiple dispatch (no matching multimethod). `candidates`: list of `(thecallable, type_signature)` that were attempted, but did not match. `_partial`: if `True`, report a failure in a *partial application*. From 72ced7667f9c9f8b2ff8f7e17b5feaa834456a5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 00:40:01 +0300 Subject: [PATCH 317/832] add back a custom implementation of function parameter binding This one is borrowed from `inspect.Signature.bind`. But we need to treat the case with extra args/kwargs without raising an error, for passthrough support in `curry`. It's frustrating that sometimes you're practically forced to import or customize some private stuff from the Python stdlib just to be able to do certain things. (In this case, we must import `inspect._empty`, because `inspect.signature` emits those - and the `bind` method itself is hardcoded to raise an error on extra args. The clean solution would be some conditions and restarts, but I suppose that train already sailed for Python.) --- unpythonic/arity.py | 157 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 156 insertions(+), 1 deletion(-) diff --git a/unpythonic/arity.py b/unpythonic/arity.py index 6308faa8..ec4e3b9e 100644 --- a/unpythonic/arity.py +++ b/unpythonic/arity.py @@ -11,8 +11,10 @@ "resolve_bindings", "resolve_bindings_partial", "tuplify_bindings", "UnknownArity"] +from collections import OrderedDict import copy -from inspect import signature, Parameter, ismethod +from inspect import signature, Parameter, ismethod, BoundArguments, _empty +import itertools import operator class UnknownArity(ValueError): @@ -457,3 +459,156 @@ def tuplify(ordereddict): thearguments = bound_arguments.arguments return tuplify(thearguments) + +# This is `inspect.Signature.bind` from Python 3.8.5, modified for our purposes so we can determine +# unbound *and extra* arguments (both positional and by-name) without raising a `TypeError`. +# We need this for kwargs support in `curry`, because we want to pass through unmatched args and kwargs +# (which otherwise trigger a `TypeError`). +# +# This is only for `curry`; all other code uses the standard implementation. +# +# Used under the PSF license. Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, +# 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Python Software Foundation; All Rights Reserved +def _bind(thesignature, args, kwargs, *, partial): + """Private method. Don't use directly.""" + + arguments = OrderedDict() + + parameters = iter(thesignature.parameters.values()) + parameters_ex = () + arg_vals = iter(args) + + # These are added for `unpythonic`. + unbound_parameters = [] + extra_args = [] + extra_kwargs = OrderedDict() + kwargs = copy.copy(kwargs) # the caller might need the original later + + while True: + # Let's iterate through the positional arguments and corresponding + # parameters + try: + arg_val = next(arg_vals) + except StopIteration: + # No more positional arguments + try: + param = next(parameters) + except StopIteration: + # No more parameters. That's it. Just need to check that + # we have no `kwargs` after this while loop + break + else: + if param.kind == Parameter.VAR_POSITIONAL: + # That's OK, just empty *args. Let's start parsing + # kwargs + break + elif param.name in kwargs: + if param.kind == Parameter.POSITIONAL_ONLY: + msg = '{arg!r} parameter is positional only, ' \ + 'but was passed as a keyword' + msg = msg.format(arg=param.name) + raise TypeError(msg) from None + parameters_ex = (param,) + break + elif (param.kind == Parameter.VAR_KEYWORD or + param.default is not _empty): + # That's fine too - we have a default value for this + # parameter. So, lets start parsing `kwargs`, starting + # with the current parameter + parameters_ex = (param,) + break + else: + # No default, not VAR_KEYWORD, not VAR_POSITIONAL, + # not in `kwargs` + if partial: + parameters_ex = (param,) + break + else: + # msg = 'missing a required argument: {arg!r}' + # msg = msg.format(arg=param.name) + # raise TypeError(msg) from None + unbound_parameters.append(param) + else: + # We have a positional argument to process + try: + param = next(parameters) + except StopIteration: + # raise TypeError('too many positional arguments') from None + extra_args.append(arg_val) + else: + if param.kind in (Parameter.VAR_KEYWORD, Parameter.KEYWORD_ONLY): + # Looks like we have no parameter for this positional + # argument + # raise TypeError( + # 'too many positional arguments') from None + extra_args.append(arg_val) + + if param.kind == Parameter.VAR_POSITIONAL: + # We have an '*args'-like argument, let's fill it with + # all positional arguments we have left and move on to + # the next phase + values = [arg_val] + values.extend(arg_vals) + arguments[param.name] = tuple(values) + break + + if param.name in kwargs and param.kind != Parameter.POSITIONAL_ONLY: + raise TypeError( + 'multiple values for argument {arg!r}'.format( + arg=param.name)) from None + + arguments[param.name] = arg_val + + # Now, we iterate through the remaining parameters to process + # keyword arguments + kwargs_param = None + for param in itertools.chain(parameters_ex, parameters): + if param.kind == Parameter.VAR_KEYWORD: + # Memorize that we have a '**kwargs'-like parameter + kwargs_param = param + continue + + if param.kind == Parameter.VAR_POSITIONAL: + # Named arguments don't refer to '*args'-like parameters. + # We only arrive here if the positional arguments ended + # before reaching the last parameter before *args. + continue + + param_name = param.name + try: + arg_val = kwargs.pop(param_name) + except KeyError: + # We have no value for this parameter. It's fine though, + # if it has a default value, or it is an '*args'-like + # parameter, left alone by the processing of positional + # arguments. + if (not partial and param.kind != Parameter.VAR_POSITIONAL and + param.default is _empty): + # raise TypeError('missing a required argument: {arg!r}'. + # format(arg=param_name)) from None + unbound_parameters.append(param) + + else: + if param.kind == Parameter.POSITIONAL_ONLY: + # This should never happen in case of a properly built + # Signature object (but let's have this check here + # to ensure correct behaviour just in case) + raise TypeError('{arg!r} parameter is positional only, ' + 'but was passed as a keyword'. + format(arg=param.name)) + + arguments[param_name] = arg_val + + if kwargs: + if kwargs_param is not None: + # Process our '**kwargs'-like parameter + arguments[kwargs_param.name] = kwargs + else: + # raise TypeError( + # 'got an unexpected keyword argument {arg!r}'.format( + # arg=next(iter(kwargs)))) + extra_kwargs.update(kwargs) + + return (BoundArguments(thesignature, arguments), + tuple(unbound_parameters), + (tuple(extra_args), extra_kwargs)) From cd72003fb108d2aca8952a533957e3bb336d958c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 00:43:54 +0300 Subject: [PATCH 318/832] add mode to function argument type checker to skip unannotated args This is needed for `unpythonic.fun.partial`, to treat functions that just have type annotations on their parameters and want nothing to do with `@generic` or `@typed`. --- unpythonic/dispatch.py | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index bb7303fa..a0268ecc 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -519,7 +519,7 @@ def _resolve_multimethod(dispatcher, args, kwargs, *, _partial=False): continue return None -def _get_argument_type_mismatches(type_signature, bound_arguments): +def _get_argument_type_mismatches(type_signature, bound_arguments, *, skip_unannotated=False): """Match bound arguments against the given type signature. Return a list of type mismatches. If it is empty, everything is ok. @@ -527,21 +527,33 @@ def _get_argument_type_mismatches(type_signature, bound_arguments): `type_signature`: in the format returned by `typing.get_type_hints`. - Must contain an item for each key of `bound_arguments`. - Is allowed to contain additional items not present in `bound_arguments`, useful for type-checking during partial application. - `bound_arguments`: see `unpythonic.arity.resolve_bindings`. + If `skip_unannotated=False` (default), `type_signature` + **must** contain an item for each key of `bound_arguments`. + + If `skip_unannotated=True`, then any binding whose key + is not in `type_signature` will not be type-checked. - `type_signature` must contain a corresponding parameter - for each argument in `bound_arguments`. (This is already - checked by `resolve_bindings`.) + In plain English, the function can have some unannotated + parameters, to denote those parameters should not be + type-checked. + + `unpythonic`'s multiple-dispatch subsystem requires + explicitly annotating `typing.Any` instead of omitting + the type annotation, but this function can be used + by other parts of `unpythonic`. + + `bound_arguments`: see `unpythonic.arity.resolve_bindings`. """ mismatches = [] for parameter, value in bound_arguments.arguments.items(): - assert parameter in type_signature # resolve_bindings should already TypeError when not. + if parameter not in type_signature: + if skip_unannotated: + continue + raise ValueError(f"type_signature has no item for parameter {parameter}, which was supplied in `bound_arguments`. If that was intended, please use `skip_unannotated=True`.") expected_type = type_signature[parameter] if not isoftype(value, expected_type): mismatches.append((parameter, value, expected_type)) From 7bba88a8a22d0e9a2b1835343d7169dbd3e3cb01 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 00:45:03 +0300 Subject: [PATCH 319/832] Support kwargs properly in `curry`. Resolves #61. --- unpythonic/fun.py | 314 +++++++++++++++++++++++++--------------------- 1 file changed, 171 insertions(+), 143 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index ac98d4ac..c4f4c26e 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -17,11 +17,12 @@ "to1st", "to2nd", "tokth", "tolast", "to", "withself"] +from collections import namedtuple from functools import wraps, partial as functools_partial +from inspect import signature from typing import get_type_hints -from .arity import (arities, _resolve_bindings, tuplify_bindings, - UnknownArity) +from .arity import (_resolve_bindings, tuplify_bindings, _bind) from .fold import reducel from .dispatch import (isgeneric, _resolve_multimethod, _format_callable, _get_argument_type_mismatches, _raise_multiple_dispatch_error, @@ -86,12 +87,17 @@ def memoized(*args, **kwargs): def partial(func, *args, **kwargs): """Wrapper over `functools.partial` that type-checks the arguments against the type annotations on `func`. + Arguments can be passed by position or by name; we compute their bindings + to function parameters like Python itself does. + The type annotations may use features from the `typing` stdlib module. See `unpythonic.typecheck.isoftype` for details. Trying to pass an argument of a type that does not match the corresponding parameter's type specification raises `TypeError` immediately. + Any parameter that does not have a type annotation will not be type-checked. + Note the check still occurs at run time, but at the use site of `partial`, when the partially applied function is constructed. This makes it fail-faster than an `isinstance` check inside the function. @@ -127,12 +133,12 @@ def partial(func, *args, **kwargs): # TODO: There's some repeated error-reporting code in `unpythonic.dispatch`. type_signature = get_type_hints(thecallable) if type_signature: # TODO: Python 3.8+: use walrus assignment here + # Partial mode: allow leaving some parameters unbound. bound_arguments = _resolve_bindings(func, collected_args, collected_kwargs, _partial=True) - # TODO: Allow having some parameters without type annotations. Requiring them for all - # TODO: parameters is a `@generic`-ism (because it uses them for dispatching). - # TODO: Alternatively, generalize `@generic` to ignore types for arguments whose parameters - # TODO: have no type annotation. But as said in the comments, that could be a footgun. - mismatches = _get_argument_type_mismatches(type_signature, bound_arguments) + # Allow having some parameters without type annotations, in which case those parameters + # will not be type-checked. `@generic` requires them for all parameters except + # `self`/`cls`, but type annotations in general have no such requirement. + mismatches = _get_argument_type_mismatches(type_signature, bound_arguments, skip_unannotated=True) if mismatches: description = _format_callable(func) mismatches_list = [f"{parameter}={repr(value)}, expected {expected_type}" @@ -176,7 +182,7 @@ def curry(f, *args, _curry_force_call=False, _curry_allow_uninspectable=False, * compute argument bindings like Python itself does. Hence we use a different algorithm, and thus a *different subset* of builtins may have become uninspectable. - When inspection fails, we raise ``unpythonic.arity.UnknownArity``. + When inspection fails, we raise ``ValueError``, like `inspect.signature` does. **Examples**:: @@ -202,9 +208,9 @@ def foo(a, b, *, c, d): **Passthrough**: - If too many args are given, any extra ones are passed through on the right. - If an intermediate result is callable, it is invoked on the remaining - positional args:: + If too many args or unacceptable kwargs are given, any extra ones are passed + through. Positional args are passed through on the right. If an intermediate + result is callable, it is invoked on the remaining args and kwargs:: map_one = lambda f: (curry(foldr))(composer(cons, to1st(f)), nil) assert curry(map_one)(double, ll(1, 2, 3)) == ll(2, 4, 6) @@ -213,9 +219,6 @@ def foo(a, b, *, c, d): is extra. The result of ``map_one`` is a callable, so it is then invoked on this tuple. - For simplicity, in passthrough, all kwargs are consumed in the first step - for which too many positional args were supplied. - By default, if any passed-through positional args are still remaining when the currently top-level curry context exits, ``curry`` raises ``TypeError``, because such usage often indicates a bug. @@ -274,9 +277,9 @@ def f(x, y): so these can at the very least be documented; and if doable with reasonable effort, preferably fixed. - It is still an error if named arguments are left over when the top-level curry context - is reached. Treating this case would require generalizing return values so that functions - could return named outputs. See: + It is still an error if **named** arguments are left over for an outer curry context. + Treating this case would require generalizing return values so that functions could + return named outputs. See: https://github.com/Technologicat/unpythonic/issues/32 """ f = force(f) # lazify support: we need the value of f @@ -289,151 +292,176 @@ def f(x, y): return maybe_force_args(f, *args, **kwargs) return f - def fallback(): # what to do if inspection fails - if not _curry_allow_uninspectable: # usual behavior - raise - # co-operate with unpythonic.syntax.autocurry; don't crash on builtins - if args or kwargs or _curry_force_call: - return maybe_force_args(f, *args, **kwargs) - return f - - try: - min_arity, max_arity = arities(f) - except UnknownArity: # likely a builtin - return fallback() + # TODO: To make `curry` pay-as-you-go, look for opportunities to speed this up + # for non-`@generic` functions. Currently this more general `curry` for v0.15.0 + # (that handles kwargs correctly) can be even 50% slower than the more limited one + # (based on positional arity only) that was in v0.14.3. + + # actions + _call = sym("_call") + _call_with_passthrough = sym("_call_with_passthrough") + _keep_currying = sym("_keep_currying") + Analysis = namedtuple("Analysis", ["bound_arguments", "unbound_parameters", "extra_args", "extra_kwargs"]) + def analyze_parameter_bindings(f, args, kwargs): + # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by + # `inspect.signature`), but `functools.partial` objects have a `keywords` attribute, which + # contains what we want. + # + # To support kwargs properly, we must compute argument bindings anyway, so we also use the + # `func` and `args` attributes. This allows us to compute the bindings of all arguments + # against the original function. + if isinstance(f, functools_partial): + function = f.func + collected_args = f.args + args + collected_kwargs = {**f.keywords, **kwargs} + else: + function = f + collected_args = args + collected_kwargs = kwargs + + def _bind_arguments(thecallable): + # For this check we look for a complete match, hence `_partial=False`. + bound_arguments, unbound_parameters, (extra_args, extra_kwargs) = _bind(signature(thecallable), + collected_args, + collected_kwargs, + partial=False) + return Analysis(bound_arguments, unbound_parameters, extra_args, extra_kwargs) + + # `@generic` functions have several call signatures, so we must aggregate the results + # in a sensible way. For non-generics, there's just one call signature. + if not isgeneric(function): + # For non-generics, the curry-time type check occurs when we later call `partial`, + # so we don't need to do that here. We just compute the bindings of arguments to parameters. + analysis = _bind_arguments(function) + if not analysis.unbound_parameters and not analysis.extra_args and not analysis.extra_kwargs: + return _call, analysis + elif not analysis.unbound_parameters and (analysis.extra_args or analysis.extra_kwargs): + return _call_with_passthrough, analysis + assert analysis.unbound_parameters + return _keep_currying, analysis + + # Curry resolver for `@generic`/`@typed` (generic functions, multimethods, multiple dispatch). + # + # Iterate over multimethods, once per step: + # + # 1. If there is an exact match (all parameters bound, type check passes, no extra + # `args`/`kwargs`), call it. + # 2. If there is a complete match (all parameters bound, type check passes), but + # with extra `args`/`kwargs` (that cannot be accepted by the call signature), + # call it, arranging passthrough for the extra `args`/`kwargs`. + # 3. If there is at least one partial match (type check passes for bound arguments, + # unbound parameters remain), keep currying. In this case extra `args`/`kwargs`, + # if any, do not matter. This will fall into case 1 or 2 above after we get + # additional `args`/`kwargs` to complete a match. + # + # If none of the above match, we know at least one parameter got a binding + # that fails the type check. Raise `TypeError`. + # + # In steps 1 and 2, we use the same lookup order as the multiple dispatcher does; + # the first matching multimethod wins. Actual dispatch is still done by the dispatcher; + # we only compute the bindings to determine which case above the call falls into. + # + # `@typed` is a special case of `@generic` with just one multimethod registered. + # The resulting behavior is the same as for a non-generic function, because the + # above algorithm reduces to that. + + # We can't use the public `list_methods` here, because on OOP methods, + # decorators live on the unbound method (raw function). Thus we must + # extract `self`/`cls` from the arguments of the call (for linked + # dispatcher lookup in the MRO). + multimethods = _list_multimethods(function, + _extract_self_or_cls(function, + collected_args)) + # Step 1: exact match + for thecallable, type_signature in multimethods: + analysis = _bind_arguments(thecallable) + if not analysis.unbound_parameters and not analysis.extra_args and not analysis.extra_kwargs: + if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): + return _call, analysis + # Step 2: complete match, with extra args/kwargs + for thecallable, type_signature in multimethods: + analysis = _bind_arguments(thecallable) + if not analysis.unbound_parameters and (analysis.extra_args or analysis.extra_kwargs): + if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): + return _call_with_passthrough, analysis + # Step 3: partial match + for thecallable, type_signature in multimethods: + analysis = _bind_arguments(thecallable) + if analysis.unbound_parameters: + if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): + return _keep_currying, analysis + # No matter which multimethod we pick, at least one parameter gets a binding + # that fails the type check. + _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, + candidates=multimethods, _partial=True) @wraps(f) def curried(*args, **kwargs): outerctx = dyn.curry_context with dyn.let(curry_context=(outerctx + [f])): - # In order to decide what to do when the curried function is called, we first compute the - # parameter bindings. - # - # All of `f`'s parameters should be bound (whether by position or by name) before calling `f`. - # - # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by - # `inspect.signature`, used by `unpythonic.arity.arities`), but `functools.partial` objects - # have a `keywords` attribute, which contains what we want. - # - # To support kwargs properly, we must compute argument bindings anyway, so we also use the - # `func` and `args` attributes. This allows us to compute the bindings against the original - # function. - if isinstance(f, functools_partial): - function = f.func - collected_args = f.args + args - collected_kwargs = {**f.keywords, **kwargs} - else: - function = f - collected_args = args - collected_kwargs = kwargs - - # The `type_signature` is used for `@generic` and `@typed` functions. - def match_arguments(thecallable, type_signature=None): - try: - bound_arguments = _resolve_bindings(thecallable, collected_args, - collected_kwargs, _partial=False) - except TypeError as err: - # TODO: Searching the error message for a particular text snippet is a big HACK, - # TODO: but we need to know *why* the arguments could not be bound. - msg = err.args[0] - if "too many" in msg: # too many positional args supplied - return "too many args" - elif "unexpected" in msg: # unexpected named arg supplied - return "unexpected kwarg" - elif "missing" in msg: # at least one parameter not bound - return "unbound parameter" - elif "multiple values" in msg: # attempted to bind a parameter to more than one value - # This is a `TypeError` for regular calls, too, so let it propagate. - raise - else: # we should have accounted for all cases # pragma: no cover - raise NotImplementedError from err - else: - # The parameter types in the call signature affect multiple-dispatching, - # so we must type-check, too. - if not type_signature or not _get_argument_type_mismatches(type_signature, bound_arguments): - return "ok" - return "argument type mismatch" - return bound_arguments # error code - - # `@generic` functions have several call signatures, so we must aggregate the results - # in some sensible way to decide what to do. For non-generics, there's just one call signature. + # In order to decide what to do when the curried function is called, we must first compute + # the parameter bindings. All of `f`'s parameters must be bound (whether by position or by + # name) before calling `f`. try: - if not isgeneric(function): - status = match_arguments(function) - else: - results = set() - # We can't use the public `list_methods` here, because on OOP methods, - # decorators live on the unbound method (raw function). Thus we must - # extract `self`/`cls` from the arguments of the call (for linked - # dispatcher lookup in the MRO). - multimethods = _list_multimethods(function, - _extract_self_or_cls(function, - collected_args)) - for thecallable, type_signature in multimethods: - result = match_arguments(thecallable, type_signature) - results.add(result) - if result == "ok": - break - # Any multimethod that can bind all collected args and kwargs (without type errors) - # is a match; prefer that first. - if "ok" in results: - status = "ok" - # No match. Figure out if we have too few or too many args/kwargs. - # If at least one multimethod can accept more args or kwargs, prefer that next. - elif "unbound parameter" in results: - status = "unbound parameter" - # Then prefer the case with too many positionals (and hope there aren't unexpected kwargs, too). - elif "too many args" in results: - status = "too many args" - elif "unexpected kwarg" in results or "argument type mismatch" in results: - _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, - candidates=multimethods, _partial=True) - else: # all cases should be accounted for # pragma: no cover - assert False - except ValueError as err: # inspect.Signature.bind(), via our _resolve_bindings() + action, analysis = analyze_parameter_bindings(f, args, kwargs) + except ValueError as err: # inspection failed in inspect.signature()? msg = err.args[0] if "no signature found" in msg: - return fallback() + if not _curry_allow_uninspectable: # usual behavior + raise + # co-operate with unpythonic.syntax.autocurry; don't crash on builtins + if args or kwargs or _curry_force_call: + return maybe_force_args(f, *args, **kwargs) + return f raise - if status == "unbound parameter": # at least one parameter not bound yet; wait for more args/kwargs - # Fail-fast: use our `partial` wrapper to type-check the partial call signature - # when we build the curried function. It delegates to `functools.partial` if the - # type check passes, and else raises a `TypeError` immediately. - p = partial(f, *args, **kwargs) - if islazy(f): - p = passthrough_lazy_args(p) - return curry(p) - - # Too many positional args; passthrough on right, like https://github.com/Technologicat/spicy - elif status == "too many args": - # TODO: The uniform thing to do would be to pass on any arguments that weren't - # TODO: bound to any parameter, regardless if they were passed positionally or by name. - now_args, later_args = args[:max_arity], args[max_arity:] - now_result = maybe_force_args(f, *now_args, **kwargs) # use up all kwargs now + if action is _call: + return maybe_force_args(f, *args, **kwargs) + + elif action == _call_with_passthrough: + # To avoid subtle errors, we must pass the arguments the same way the user did: + # - Any arguments passed to us positionally must be passed through positionally, + # - Any arguments passed to us by name must be passed through by name. + # + # Note the impedance mismatch with our use of `functools.partial`; the `args`/`kwargs` + # here are **NOT** the full `args`/`kwargs`, but only the new ones from this step. + # + # We know these args/kwargs were extra when matched against the function's call signature: + later_args = analysis.extra_args + later_kwargs = analysis.extra_kwargs + # Hence, we should avoid passing **now** any args/kwargs that should be passed later: + now_args = args[:-len(later_args)] + now_kwargs = {k: v for k, v in kwargs.items() if k not in later_kwargs} + + now_result = maybe_force_args(f, *now_args, **now_kwargs) now_result = force(now_result) if not isinstance(now_result, tuple) else force1(now_result) if callable(now_result): - # curry it now, to sustain the chain in case we have - # too many (or too few) args for it. + # Curry it now, to sustain the chain in case we have too many (or too few) args for it. if not iscurried(now_result): now_result = curry(now_result) - return now_result(*later_args) + return now_result(*later_args, **later_kwargs) if not outerctx: - raise TypeError(f"Top-level curry context exited with {len(later_args)} arg(s) remaining: {later_args}") - # pass through to the curried procedure waiting in outerctx - # (e.g. in a curried compose chain) + raise TypeError(f"Top-level curry context exited with {len(later_args) + len(later_kwargs)} arg(s) remaining. Positional: {later_args}, named: {later_kwargs}") + # Pass through to the curried procedure waiting in outerctx (e.g. in a curried compose chain). + # TODO: To handle later_kwargs here, we need named return values. See issue #32. + # https://github.com/Technologicat/unpythonic/issues/32 + if later_kwargs: + raise NotImplementedError(f"Passing through named arguments to an outer curry context not implemented; got {later_kwargs}") if isinstance(now_result, tuple): return now_result + later_args return (now_result,) + later_args - # Unexpected kwarg, could not be bound to any parameter - elif status == "unexpected kwarg": - # TODO: report the unexpected kwarg(s) - raise NotImplementedError("curry: cannot pass-through unexpected named args") + elif action is _keep_currying: + # Fail-fast: use our `partial` wrapper to type-check the partial call signature + # when we build the curried function. It delegates to `functools.partial` if the + # type check passes, and else raises a `TypeError` immediately. + p = partial(f, *args, **kwargs) + if islazy(f): + p = passthrough_lazy_args(p) + return curry(p) - # All parameters bound to some arg or kwarg - assert status == "ok", status - return maybe_force_args(f, *args, **kwargs) + else: # pragma: no cover + assert False, action if islazy(f): curried = passthrough_lazy_args(curried) curried._is_curried_function = True # stash for detection @@ -446,7 +474,7 @@ def iscurried(f): """Return whether f is a curried function.""" return hasattr(f, "_is_curried_function") -#def curry_simple(f): # essential idea, without the extra features +#def curry_simple(f): # essential idea, without any extra features # min_arity, _ = arities(f) # @wraps(f) # def curried(*args, **kwargs): From 99b21418d3168e3f6c3c130485865014a4bb4f1b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 00:45:46 +0300 Subject: [PATCH 320/832] update CHANGELOG --- CHANGELOG.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2359306e..ac01b550 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,9 @@ -**0.15.0** (in progress; updated 10 May 2021) - *"We say 'howdy' around these parts"* edition: +**0.15.0** (in progress; updated 19 May 2021) - *"We say 'howdy' around these parts"* edition: Beside introducing **dialects** (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This introduces some breaking changes, so we have also taken the opportunity to apply any such that were previously scheduled. +We have sneaked in some upgrades for other subsystems, too. Particularly `curry`, the multiple dispatch system (`@generic`), and the integration between these two have been improved significantly. + **IMPORTANT**: - Minimum Python language version is now 3.6. @@ -90,9 +92,10 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `curry` now supports `@generic` functions. **This feature is experimental. Semantics may still change.** - The utilities `arities`, `required_kwargs`, and `optional_kwargs` now support `@generic` functions. **This feature is experimental. Semantics may still change.** - `curry` now errors out immediately on argument type mismatch. - - Add `partial`, a type-checking wrapper for `functools.partial`. + - Add `partial`, a type-checking wrapper for `functools.partial`, that errors out immediately on argument type mismatch. - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. + - Add `resolve_bindings_partial`, useful for analyzing partial application. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) - Python 3.8 and 3.9 support added. @@ -109,6 +112,11 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `mcpyrate.debug.step_expansion` is able to show the intermediate result after the `do` or `do0` has expanded, but before anything else has been done to the tree. - **Miscellaneous.** + - Resolve issue [#61](https://github.com/Technologicat/unpythonic/issues/61): `curry` now supports kwargs properly. + - We now analyze parameter bindings like Python itself does, so it should no longer matter whether arguments are passed by position or by name. + - Positional passthrough works as before. + - Now any remaining named arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result. + - However, passing named arguments to an outer curry context is not supported, because the clean solution for that requires support for named return values (see issue [#32](https://github.com/Technologicat/unpythonic/issues/32)). - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). From 1185228b3d4a1d48b28e410461ccd52a8311e2e8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 01:57:08 +0300 Subject: [PATCH 321/832] update curry docs --- doc/features.md | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/doc/features.md b/doc/features.md index 19202201..8fa3ab3a 100644 --- a/doc/features.md +++ b/doc/features.md @@ -995,16 +995,19 @@ Things missing from the standard library. - Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. - For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)). - `curry`, with some extra features: - - Passthrough on the right when too many args (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket) - - If the intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining positional args. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). - - For simplicity, all remaining keyword args are fed in at the first step that has too many positional args. + - **Changed in v0.15.0.** `curry` supports both positional and named arguments, and binds arguments to function parameters like Python itself does. The call triggers when all parameters are bound, regardless of whether they were passed by position or by name, and at which step of the currying process they were passed. + - **Changed in v0.15.0.** `unpythonic`'s multiple-dispatch system (`@generic`, `@typed`) is supported. `curry` looks for an exact match first, then a match with extra args/kwargs, and finally a partial match. If there is still no match, this implies that at least one parameter would get a binding that fails the type check. In such a case `TypeError` regarding failed multiple dispatch is raised. + - **Changed in v0.15.0.** If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`. + - Passthrough when too many args (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket). Positional args are passed through **on the right**. + - If the intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining args and kwargs. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. - To override, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. - To set the dynvar, `from unpythonic import dyn`, and then `with dyn.let(curry_context=...):`. + - Even with the upgrades in v0.15.0, passing through *named* args to an outer curry context is not supported. This may or may not change in the future; fixing this requires support for named return values. See issue [#32](https://github.com/Technologicat/unpythonic/issues/32). - Can be used both as a decorator and as a regular function. - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. - - **Caution**: If the positional arities of ``f`` cannot be inspected, currying fails, raising ``UnknownArity``. This may happen with builtins such as ``list.append``. - - `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. Type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions. Our `curry` uses this type-checking `partial` instead of the standard one, so currying supports fail-fast, too. **Added in v0.15.0.** + - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print`` or ``range``. + - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values packed into a tuple are unpacked to the argument list of the next function in the chain. - `composelc`, `composerc`: curry each function before composing them. Useful with passthrough. @@ -1150,7 +1153,9 @@ Finally, keep in mind this exercise is intended as a feature demonstration. In p #### ``curry`` and reduction rules -The provided variant of ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. When we say: +**Changed in v0.15.0.** *`curry` now supports kwargs, too, and binds parameters like Python itself does. Also, `@generic` and `@typed` functions are supported.* + +Our ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. Let's consider a simple example with positional arguments only. When we say: ```python curry(f, a0, a1, ..., a[n-1]) @@ -1167,7 +1172,26 @@ it means the following. Let ``m1`` and ``m2`` be the minimum and maximum positio - If ``n < m1``, partially apply ``f`` to the given arguments, yielding a new function with smaller ``m1``, ``m2``. Then curry the result and return it. - Internally we stack ``functools.partial`` applications, but there will be only one ``curried`` wrapper no matter how many invocations are used to build up arguments before ``f`` eventually gets called. -In the above example: +As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the presence of kwargs and `@generic` functions, is: + + - If `f` is **not** `@generic` or `@typed`: + - Compute parameter bindings of the args and kwargs collected so far, against the call signature of `f`. + - Note we keep track of which arguments were passed positionally and which by name. To avoid subtle errors, they are eventually passed to `f` the same way they were passed to `curry`. (Positional args are passed positionally, and kwargs are passed by name.) + - If there are no unbound parameters, and no args/kwargs are left over, we have an exact match. Call `f` and return its result, like a normal function call. + - Any sequence of curried calls that ends up binding all parameters of `f` triggers the call. + - As before, beware when working with variadic functions. Particularly, keep in mind that `*args` matches **zero or more** positional arguments (as the [Kleene star](https://en.wikipedia.org/wiki/Kleene_star)-ish notation indeed suggests). + - If there are no unbound parameters, but there are args/kwargs left over, arrange passthrough for the leftover args/kwargs (that were rejected by the call signature of `f`), and call `f`. If the result is a callable, curry it, and recurse. Else form a tuple... (as above). + - If neither of the above match, we know there is at least one unbound parameter, i.e. we have a partial match. Keep currying. + - If `f` is `@generic` or `@typed`: + - Iterate over multimethods registered on `f`, **up to three times**. + - First, try for an exact match that passes the type check. **If any such match is found**, pick that multimethod. Call it and return its result (as above). + - Then, try for a match that passes the type check, but has extra args/kwargs. **If any such match is found**, pick that multimethod. Arrange passthrough... (as above). + - Then, try for a partial match that passes the type check. **If any such match is found**, keep currying. + - If none of the above match, it implies that no matter which multimethod we pick, at least one parameter would get a binding that fails the type check. Raise `TypeError`. + +(If *really* interested in the gritty details, look at the source code of `unpythonic.fun.curry`. It calls some functions from `unpythonic.dispatch` for its `@generic` support, but otherwise it's pretty much self-contained.) + +Getting back to the simple case, in the above example: ```python curry(mapl_one, double, ll(1, 2, 3)) From 61d0d28d2c3d551bda421eae07f9c49b4c1cea96 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 02:24:36 +0300 Subject: [PATCH 322/832] fix bug --- unpythonic/fun.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index c4f4c26e..2247bb6c 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -430,7 +430,10 @@ def curried(*args, **kwargs): later_args = analysis.extra_args later_kwargs = analysis.extra_kwargs # Hence, we should avoid passing **now** any args/kwargs that should be passed later: - now_args = args[:-len(later_args)] + if later_args: + now_args = args[:-len(later_args)] + else: + now_args = args now_kwargs = {k: v for k, v in kwargs.items() if k not in later_kwargs} now_result = maybe_force_args(f, *now_args, **now_kwargs) From 6ce916b3dd4f2f948e6300ad84e97bee4206b6a4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 02:24:43 +0300 Subject: [PATCH 323/832] change ordering of which error takes priority --- unpythonic/fun.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 2247bb6c..ff8004c8 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -443,13 +443,13 @@ def curried(*args, **kwargs): if not iscurried(now_result): now_result = curry(now_result) return now_result(*later_args, **later_kwargs) - if not outerctx: - raise TypeError(f"Top-level curry context exited with {len(later_args) + len(later_kwargs)} arg(s) remaining. Positional: {later_args}, named: {later_kwargs}") - # Pass through to the curried procedure waiting in outerctx (e.g. in a curried compose chain). # TODO: To handle later_kwargs here, we need named return values. See issue #32. # https://github.com/Technologicat/unpythonic/issues/32 if later_kwargs: raise NotImplementedError(f"Passing through named arguments to an outer curry context not implemented; got {later_kwargs}") + if not outerctx: + raise TypeError(f"Top-level curry context exited with {len(later_args) + len(later_kwargs)} arg(s) remaining. Positional: {later_args}, named: {later_kwargs}") + # Pass through to the curried procedure waiting in outerctx (e.g. in a curried compose chain). if isinstance(now_result, tuple): return now_result + later_args return (now_result,) + later_args From 4ddee095955b7c961b75b1bd7fdfbc2520cc6ca6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 02:26:08 +0300 Subject: [PATCH 324/832] wording --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 805b083d..dff0cfad 100644 --- a/README.md +++ b/README.md @@ -147,6 +147,8 @@ Optionally, if you have [mcpyrate](https://github.com/Technologicat/mcpyrate), t [[docs](doc/features.md#batteries-for-itertools)] +Scan and fold accept multiple iterables, like in Racket. + ```python from operator import add from unpythonic import scanl, foldl, unfold, take @@ -155,7 +157,7 @@ assert tuple(scanl(add, 0, range(1, 5))) == (0, 1, 3, 6, 10) def op(e1, e2, acc): return acc + e1 * e2 -assert foldl(op, 0, (1, 2), (3, 4)) == 11 # we accept multiple input sequences, like Racket +assert foldl(op, 0, (1, 2), (3, 4)) == 11 def nextfibo(a, b): # *oldstates return (a, b, a + b) # value, *newstates From 2c4df925868e4fb249dbe7831a8f99199bc30214 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 02:26:14 +0300 Subject: [PATCH 325/832] advertise curry in README --- README.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/README.md b/README.md index dff0cfad..d9bcc1c5 100644 --- a/README.md +++ b/README.md @@ -164,6 +164,60 @@ def nextfibo(a, b): # *oldstates assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) ```
+
Industrial-strength curry. + +[[docs](doc/features.md#batteries-for-functools)] + +We bind arguments to parameters like Python itself does, so it does not matter whether arguments are passed by position or by name during currying. We support `@generic` multiple-dispatch functions. + +```python +from unpythonic import curry, generic + +@curry +def f(x, y): + return x, y + +assert f(1, 2) == (1, 2) +assert f(1)(2) == (1, 2) +assert f(1)(y=2) == (1, 2) +assert f(y=2)(x=1) == (1, 2) + +@curry +def add3(x, y, z): + return x + y + z + +# actually uses partial application so these work, too +assert add3(1)(2)(3) == 6 +assert add3(1, 2)(3) == 6 +assert add3(1)(2, 3) == 6 +assert add3(1, 2, 3) == 6 + +@curry +def lispyadd(*args): + return sum(args) +assert lispyadd() == 0 # no args is a valid arity here + +@generic +def g(x: int, y: int): + return "int" +@generic +def g(x: float, y: float): + return "float" +@generic +def g(s: str): + return "str" +g = curry(g) + +assert callable(g(1)) +assert g(1)(2) == "int" + +assert callable(g(1.0)) +assert g(1.0)(2.0) == "float" + +assert g("cat") == "str" +assert g(s="cat") == "str" +``` +
Allow a lambda to call itself. Name a lambda. [[docs for `withself`](doc/features.md#batteries-for-functools)] [[docs for `namelambda`](doc/features.md#namelambda-rename-a-function)] From bd9d2b34cb9c863808499f24697dd88db535828d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 02:58:26 +0300 Subject: [PATCH 326/832] advertise lispy symbols in README --- README.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/README.md b/README.md index d9bcc1c5..f11bf743 100644 --- a/README.md +++ b/README.md @@ -312,6 +312,39 @@ assert s == (10, 2, 10, 4, 10) assert t == (1, 2, 3, 4, 5) ```
+
Lispy symbol type. + +[[docs](doc/features.md#sym-gensym-Singleton-symbols-and-singletons)] + +Roughly, a [symbol](https://stackoverflow.com/questions/8846628/what-exactly-is-a-symbol-in-lisp-scheme) is a guaranteed-[interned](https://en.wikipedia.org/wiki/String_interning) string. + +A [gensym](http://clhs.lisp.se/Body/f_gensym.htm) is a guaranteed-unique string, which is useful as a nonce value. It's similar to the pythonic idiom `nonce = object()`, but with a nice repr, and object-identity-preserving pickle support. + +```python +from unpythonic import sym # lispy symbol +sandwich = sym("sandwich") +hamburger = sym("sandwich") # symbol's identity is determined by its name, only +assert hamburger is sandwich + +assert str(sandwich) == "sandwich" # symbols have a nice str() +assert repr(sandwich) == 'sym("sandwich")' # and eval-able repr() +assert eval(repr(sandwich)) is sandwich + +from pickle import dumps, loads +pickled_sandwich = dumps(sandwich) +unpickled_sandwich = loads(pickled_sandwich) +assert unpickled_sandwich is sandwich # symbols survive a pickle roundtrip + +from unpythonic import gensym # gensym: make new uninterned symbol +tabby = gensym("cat") +scottishfold = gensym("cat") +assert tabby is not scottishfold + +pickled_tabby = dumps(tabby) +unpickled_tabby = loads(pickled_tabby) +assert unpickled_tabby is tabby # also gensyms survive a pickle roundtrip +``` +
Lispy data structures. [[docs for `box`](doc/features.md#box-a-mutable-single-item-container)] [[docs for `cons`](doc/features.md#cons-and-friends-pythonic-lispy-linked-lists)] [[docs for `frozendict`](doc/features.md#frozendict-an-immutable-dictionary)] From c308240b69013b4e955e581416343868fc77bbc0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:03:03 +0300 Subject: [PATCH 327/832] more generic functions further up in the README --- README.md | 118 +++++++++++++++++++++++++++--------------------------- 1 file changed, 59 insertions(+), 59 deletions(-) diff --git a/README.md b/README.md index f11bf743..add584b4 100644 --- a/README.md +++ b/README.md @@ -218,6 +218,65 @@ assert g("cat") == "str" assert g(s="cat") == "str" ```
+
Multiple-dispatch generic functions, like in CLOS or Julia. + +[[docs](doc/features.md#generic-typed-isoftype-multiple-dispatch)] + +```python +from unpythonic import generic + +@generic +def my_range(stop: int): # create the generic function and the first multimethod + return my_range(0, 1, stop) +@generic +def my_range(start: int, stop: int): # further registrations add more multimethods + return my_range(start, 1, stop) +@generic +def my_range(start: int, step: int, stop: int): + return start, step, stop +``` + +This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. + +*Holy traits* are also a possibility: + +```python +import typing +from unpythonic import generic, augment + +class FunninessTrait: + pass +class IsFunny(FunninessTrait): + pass +class IsNotFunny(FunninessTrait): + pass + +@generic +def funny(x: typing.Any): # default + raise NotImplementedError(f"`funny` trait not registered for anything matching {type(x)}") + +@augment(funny) +def funny(x: str): # noqa: F811 + return IsFunny() +@augment(funny) +def funny(x: int): # noqa: F811 + return IsNotFunny() + +@generic +def laugh(x: typing.Any): + return laugh(funny(x), x) + +@augment(laugh) +def laugh(traitvalue: IsFunny, x: typing.Any): + return f"Ha ha ha, {x} is funny!" +@augment(laugh) +def laugh(traitvalue: IsNotFunny, x: typing.Any): + return f"{x} is not funny." + +assert laugh("that") == "Ha ha ha, that is funny!" +assert laugh(42) == "42 is not funny." +``` +
Allow a lambda to call itself. Name a lambda. [[docs for `withself`](doc/features.md#batteries-for-functools)] [[docs for `namelambda`](doc/features.md#namelambda-rename-a-function)] @@ -403,65 +462,6 @@ assert x == 85 The point is usability: in a function composition using pipe syntax, data flows from left to right.
-
Multiple-dispatch generic functions, like in CLOS or Julia. - -[[docs](doc/features.md#generic-typed-isoftype-multiple-dispatch)] - -```python -from unpythonic import generic - -@generic -def my_range(stop: int): # create the generic function and the first multimethod - return my_range(0, 1, stop) -@generic -def my_range(start: int, stop: int): # further registrations add more multimethods - return my_range(start, 1, stop) -@generic -def my_range(start: int, step: int, stop: int): - return start, step, stop -``` - -This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. - -*Holy traits* are also a possibility: - -```python -import typing -from unpythonic import generic, augment - -class FunninessTrait: - pass -class IsFunny(FunninessTrait): - pass -class IsNotFunny(FunninessTrait): - pass - -@generic -def funny(x: typing.Any): # default - raise NotImplementedError(f"`funny` trait not registered for anything matching {type(x)}") - -@augment(funny) -def funny(x: str): # noqa: F811 - return IsFunny() -@augment(funny) -def funny(x: int): # noqa: F811 - return IsNotFunny() - -@generic -def laugh(x: typing.Any): - return laugh(funny(x), x) - -@augment(laugh) -def laugh(traitvalue: IsFunny, x: typing.Any): - return f"Ha ha ha, {x} is funny!" -@augment(laugh) -def laugh(traitvalue: IsNotFunny, x: typing.Any): - return f"{x} is not funny." - -assert laugh("that") == "Ha ha ha, that is funny!" -assert laugh(42) == "42 is not funny." -``` -
Conditions: resumable, modular error handling, like in Common Lisp. [[docs](doc/features.md#handlers-restarts-conditions-and-restarts)] From ea89edc2a2c1b96eff5f7b193d69fe666c79b399 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:07:32 +0300 Subject: [PATCH 328/832] reorganize highlighted pure-Python features --- README.md | 234 +++++++++++++++++++++++++++--------------------------- 1 file changed, 117 insertions(+), 117 deletions(-) diff --git a/README.md b/README.md index add584b4..c80649a0 100644 --- a/README.md +++ b/README.md @@ -276,6 +276,123 @@ def laugh(traitvalue: IsNotFunny, x: typing.Any): assert laugh("that") == "Ha ha ha, that is funny!" assert laugh(42) == "42 is not funny." ``` +
+
Conditions: resumable, modular error handling, like in Common Lisp. + +[[docs](doc/features.md#handlers-restarts-conditions-and-restarts)] + +Contrived example: + +```python +from unpythonic import error, restarts, handlers, invoke, use_value, unbox + +class MyError(ValueError): + def __init__(self, value): # We want to act on the value, so save it. + self.value = value + +def lowlevel(lst): + _drop = object() # gensym/nonce + out = [] + for k in lst: + # Provide several different error recovery strategies. + with restarts(use_value=(lambda x: x), + halve=(lambda x: x // 2), + drop=(lambda: _drop)) as result: + if k > 9000: + error(MyError(k)) + # This is reached when no error occurs. + # `result` is a box, send k into it. + result << k + # Now the result box contains either k, + # or the return value of one of the restarts. + r = unbox(result) # get the value from the box + if r is not _drop: + out.append(r) + return out + +def highlevel(): + # Choose which error recovery strategy to use... + with handlers((MyError, lambda c: use_value(c.value))): + assert lowlevel([17, 10000, 23, 42]) == [17, 10000, 23, 42] + + # ...on a per-use-site basis... + with handlers((MyError, lambda c: invoke("halve", c.value))): + assert lowlevel([17, 10000, 23, 42]) == [17, 5000, 23, 42] + + # ...without changing the low-level code. + with handlers((MyError, lambda: invoke("drop"))): + assert lowlevel([17, 10000, 23, 42]) == [17, 23, 42] + +highlevel() +``` + +Conditions only shine in larger systems, with restarts set up at multiple levels of the call stack; this example is too small to demonstrate that. The single-level case here could be implemented as a error-handling mode parameter for the example's only low-level function. + +With multiple levels, it becomes apparent that this mode parameter must be threaded through the API at each level, unless it is stored as a dynamic variable (see [`unpythonic.dyn`](doc/features.md#dyn-dynamic-assignment)). But then, there can be several types of errors, and the error-handling mode parameters - one for each error type - have to be shepherded in an intricate manner. A stack is needed, so that an inner level may temporarily override the handler for a particular error type... + +The condition system is the clean, general solution to this problem. It automatically scopes handlers to their dynamic extent, and manages the handler stack automatically. In other words, it dynamically binds error-handling modes (for several types of errors, if desired) in a controlled, easily understood manner. The local programmability (i.e. the fact that a handler is not just a restart name, but an arbitrary function) is a bonus for additional flexibility. + +If this sounds a lot like an exception system, that's because conditions are the supercharged sister of exceptions. The condition model cleanly separates mechanism from policy, while otherwise remaining similar to the exception model. +
+
Lispy symbol type. + +[[docs](doc/features.md#sym-gensym-Singleton-symbols-and-singletons)] + +Roughly, a [symbol](https://stackoverflow.com/questions/8846628/what-exactly-is-a-symbol-in-lisp-scheme) is a guaranteed-[interned](https://en.wikipedia.org/wiki/String_interning) string. + +A [gensym](http://clhs.lisp.se/Body/f_gensym.htm) is a guaranteed-unique string, which is useful as a nonce value. It's similar to the pythonic idiom `nonce = object()`, but with a nice repr, and object-identity-preserving pickle support. + +```python +from unpythonic import sym # lispy symbol +sandwich = sym("sandwich") +hamburger = sym("sandwich") # symbol's identity is determined by its name, only +assert hamburger is sandwich + +assert str(sandwich) == "sandwich" # symbols have a nice str() +assert repr(sandwich) == 'sym("sandwich")' # and eval-able repr() +assert eval(repr(sandwich)) is sandwich + +from pickle import dumps, loads +pickled_sandwich = dumps(sandwich) +unpickled_sandwich = loads(pickled_sandwich) +assert unpickled_sandwich is sandwich # symbols survive a pickle roundtrip + +from unpythonic import gensym # gensym: make new uninterned symbol +tabby = gensym("cat") +scottishfold = gensym("cat") +assert tabby is not scottishfold + +pickled_tabby = dumps(tabby) +unpickled_tabby = loads(pickled_tabby) +assert unpickled_tabby is tabby # also gensyms survive a pickle roundtrip +``` +
+
Lispy data structures. + +[[docs for `box`](doc/features.md#box-a-mutable-single-item-container)] [[docs for `cons`](doc/features.md#cons-and-friends-pythonic-lispy-linked-lists)] [[docs for `frozendict`](doc/features.md#frozendict-an-immutable-dictionary)] + +```python +from unpythonic import box, unbox # mutable single-item container +cat = object() +cardboardbox = box(cat) +assert cardboardbox is not cat # the box is not the cat +assert unbox(cardboardbox) is cat # but the cat is inside the box +assert cat in cardboardbox # ...also syntactically +dog = object() +cardboardbox << dog # hey, it's my box! (replace contents) +assert unbox(cardboardbox) is dog + +from unpythonic import cons, nil, ll, llist # lispy linked lists +lst = cons(1, cons(2, cons(3, nil))) +assert ll(1, 2, 3) == lst # make linked list out of elements +assert llist([1, 2, 3]) == lst # convert iterable to linked list + +from unpythonic import frozendict # immutable dictionary +d1 = frozendict({'a': 1, 'b': 2}) +d2 = frozendict(d1, c=3, a=4) +assert d1 == frozendict({'a': 1, 'b': 2}) +assert d2 == frozendict({'a': 4, 'b': 2, 'c': 3}) +```
Allow a lambda to call itself. Name a lambda. @@ -371,66 +488,6 @@ assert s == (10, 2, 10, 4, 10) assert t == (1, 2, 3, 4, 5) ```
-
Lispy symbol type. - -[[docs](doc/features.md#sym-gensym-Singleton-symbols-and-singletons)] - -Roughly, a [symbol](https://stackoverflow.com/questions/8846628/what-exactly-is-a-symbol-in-lisp-scheme) is a guaranteed-[interned](https://en.wikipedia.org/wiki/String_interning) string. - -A [gensym](http://clhs.lisp.se/Body/f_gensym.htm) is a guaranteed-unique string, which is useful as a nonce value. It's similar to the pythonic idiom `nonce = object()`, but with a nice repr, and object-identity-preserving pickle support. - -```python -from unpythonic import sym # lispy symbol -sandwich = sym("sandwich") -hamburger = sym("sandwich") # symbol's identity is determined by its name, only -assert hamburger is sandwich - -assert str(sandwich) == "sandwich" # symbols have a nice str() -assert repr(sandwich) == 'sym("sandwich")' # and eval-able repr() -assert eval(repr(sandwich)) is sandwich - -from pickle import dumps, loads -pickled_sandwich = dumps(sandwich) -unpickled_sandwich = loads(pickled_sandwich) -assert unpickled_sandwich is sandwich # symbols survive a pickle roundtrip - -from unpythonic import gensym # gensym: make new uninterned symbol -tabby = gensym("cat") -scottishfold = gensym("cat") -assert tabby is not scottishfold - -pickled_tabby = dumps(tabby) -unpickled_tabby = loads(pickled_tabby) -assert unpickled_tabby is tabby # also gensyms survive a pickle roundtrip -``` -
-
Lispy data structures. - -[[docs for `box`](doc/features.md#box-a-mutable-single-item-container)] [[docs for `cons`](doc/features.md#cons-and-friends-pythonic-lispy-linked-lists)] [[docs for `frozendict`](doc/features.md#frozendict-an-immutable-dictionary)] - -```python -from unpythonic import box, unbox # mutable single-item container -cat = object() -cardboardbox = box(cat) -assert cardboardbox is not cat # the box is not the cat -assert unbox(cardboardbox) is cat # but the cat is inside the box -assert cat in cardboardbox # ...also syntactically -dog = object() -cardboardbox << dog # hey, it's my box! (replace contents) -assert unbox(cardboardbox) is dog - -from unpythonic import cons, nil, ll, llist # lispy linked lists -lst = cons(1, cons(2, cons(3, nil))) -assert ll(1, 2, 3) == lst # make linked list out of elements -assert llist([1, 2, 3]) == lst # convert iterable to linked list - -from unpythonic import frozendict # immutable dictionary -d1 = frozendict({'a': 1, 'b': 2}) -d2 = frozendict(d1, c=3, a=4) -assert d1 == frozendict({'a': 1, 'b': 2}) -assert d2 == frozendict({'a': 4, 'b': 2, 'c': 3}) -``` -
Live list slices. [[docs](doc/features.md#view-writable-sliceable-view-into-a-sequence)] @@ -461,63 +518,6 @@ assert x == 85 ``` The point is usability: in a function composition using pipe syntax, data flows from left to right. -
-
Conditions: resumable, modular error handling, like in Common Lisp. - -[[docs](doc/features.md#handlers-restarts-conditions-and-restarts)] - -Contrived example: - -```python -from unpythonic import error, restarts, handlers, invoke, use_value, unbox - -class MyError(ValueError): - def __init__(self, value): # We want to act on the value, so save it. - self.value = value - -def lowlevel(lst): - _drop = object() # gensym/nonce - out = [] - for k in lst: - # Provide several different error recovery strategies. - with restarts(use_value=(lambda x: x), - halve=(lambda x: x // 2), - drop=(lambda: _drop)) as result: - if k > 9000: - error(MyError(k)) - # This is reached when no error occurs. - # `result` is a box, send k into it. - result << k - # Now the result box contains either k, - # or the return value of one of the restarts. - r = unbox(result) # get the value from the box - if r is not _drop: - out.append(r) - return out - -def highlevel(): - # Choose which error recovery strategy to use... - with handlers((MyError, lambda c: use_value(c.value))): - assert lowlevel([17, 10000, 23, 42]) == [17, 10000, 23, 42] - - # ...on a per-use-site basis... - with handlers((MyError, lambda c: invoke("halve", c.value))): - assert lowlevel([17, 10000, 23, 42]) == [17, 5000, 23, 42] - - # ...without changing the low-level code. - with handlers((MyError, lambda: invoke("drop"))): - assert lowlevel([17, 10000, 23, 42]) == [17, 23, 42] - -highlevel() -``` - -Conditions only shine in larger systems, with restarts set up at multiple levels of the call stack; this example is too small to demonstrate that. The single-level case here could be implemented as a error-handling mode parameter for the example's only low-level function. - -With multiple levels, it becomes apparent that this mode parameter must be threaded through the API at each level, unless it is stored as a dynamic variable (see [`unpythonic.dyn`](doc/features.md#dyn-dynamic-assignment)). But then, there can be several types of errors, and the error-handling mode parameters - one for each error type - have to be shepherded in an intricate manner. A stack is needed, so that an inner level may temporarily override the handler for a particular error type... - -The condition system is the clean, general solution to this problem. It automatically scopes handlers to their dynamic extent, and manages the handler stack automatically. In other words, it dynamically binds error-handling modes (for several types of errors, if desired) in a controlled, easily understood manner. The local programmability (i.e. the fact that a handler is not just a restart name, but an arbitrary function) is a bonus for additional flexibility. - -If this sounds a lot like an exception system, that's because conditions are the supercharged sister of exceptions. The condition model cleanly separates mechanism from policy, while otherwise remaining similar to the exception model.
From f953266f5b9744f74944e11083a0b9c076ae69dc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:21:41 +0300 Subject: [PATCH 329/832] try to fail-fast on missing builtin call signatures --- unpythonic/fun.py | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index ff8004c8..3364ab20 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -292,6 +292,23 @@ def f(x, y): return maybe_force_args(f, *args, **kwargs) return f + def fallback(): + if not _curry_allow_uninspectable: # usual behavior + raise + # co-operate with unpythonic.syntax.autocurry; don't crash on builtins + if args or kwargs or _curry_force_call: + return maybe_force_args(f, *args, **kwargs) + return f + + # Try to fail-fast with uninspectable builtins. + try: + signature(f) + except ValueError as err: # inspection failed in inspect.signature()? + msg = err.args[0] + if "no signature found" in msg: + return fallback() + raise + # TODO: To make `curry` pay-as-you-go, look for opportunities to speed this up # for non-`@generic` functions. Currently this more general `curry` for v0.15.0 # (that handles kwargs correctly) can be even 50% slower than the more limited one @@ -407,12 +424,7 @@ def curried(*args, **kwargs): except ValueError as err: # inspection failed in inspect.signature()? msg = err.args[0] if "no signature found" in msg: - if not _curry_allow_uninspectable: # usual behavior - raise - # co-operate with unpythonic.syntax.autocurry; don't crash on builtins - if args or kwargs or _curry_force_call: - return maybe_force_args(f, *args, **kwargs) - return f + return fallback() raise if action is _call: From d30afcba1a65ec5a08ae0faea9e1ca8d41355927 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:24:14 +0300 Subject: [PATCH 330/832] fix tests --- unpythonic/tests/test_arity.py | 12 +++++++++++- unpythonic/tests/test_fun.py | 30 ++++++++---------------------- 2 files changed, 19 insertions(+), 23 deletions(-) diff --git a/unpythonic/tests/test_arity.py b/unpythonic/tests/test_arity.py index fccda24e..5a3307d4 100644 --- a/unpythonic/tests/test_arity.py +++ b/unpythonic/tests/test_arity.py @@ -3,10 +3,12 @@ from ..syntax import macros, test, test_raises, the # noqa: F401 from ..test.fixtures import session, testset +import sys + from ..arity import (arities, arity_includes, required_kwargs, optional_kwargs, kwargs, resolve_bindings, tuplify_bindings, - getfunc) + getfunc, UnknownArity) def runtests(): def barefunction(x): @@ -102,6 +104,14 @@ def instmeth(self): test[arities(target.classmeth) == (1, 1)] test[arities(target.staticmeth) == (1, 1)] + # Methods of builtin types have uninspectable arity up to Python 3.6. + # Python 3.7 seems to fix this at least for `list`, and PyPy3 (7.3.0; Python 3.6.9) + # doesn't have this error either. + if sys.version_info < (3, 7, 0) and sys.implementation.name == "cpython": + with testset("uninspectable builtin methods"): + lst = [] + test_raises[UnknownArity, arities(lst.append)] + # resolve_bindings: resolve parameter bindings established by a function # when it is called with the given args and kwargs. # diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 12732bdf..1f9109ab 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -4,7 +4,6 @@ from ..test.fixtures import session, testset, returns_normally from collections import Counter -import sys from ..dispatch import generic from ..fun import (memoize, partial, curry, apply, @@ -17,7 +16,6 @@ withself) from ..dynassign import dyn -from ..arity import UnknownArity def runtests(): with testset("identity function"): @@ -207,26 +205,14 @@ def double(x): # a `with test` can optionally return a value, which becomes the asserted expr. return curry(double, 2, "foo") == (4, "foo") - # Methods of builtin types have uninspectable arity up to Python 3.6. - # Python 3.7 seems to fix this at least for `list`, and PyPy3 (7.3.0; Python 3.6.9) - # doesn't have this error either. - if sys.version_info < (3, 7, 0) and sys.implementation.name == "cpython": - with testset("uninspectable builtins"): - lst = [] - test_raises[UnknownArity, curry(lst.append)] # uninspectable method of builtin type - - # Internal feature, used by curry macro. If uninspectables are said to be ok, - # then attempting to curry an uninspectable simply returns the original function. - # - # Due to Python's method binding machinery re-triggering the descriptor on each lookup, - # each lookup of `lst.append` will produce a *new* instance of the object that - # represents the bound method (builtin method, in this case). They print the same, - # they look the same... but they `is not` the same. - # - # To avoid this pitfall, we do the lookup exactly once - and then reuse the result. - m1 = lst.append - m2 = curry(m1, _curry_allow_uninspectable=True) - test[m2 is m1] + with testset("uninspectable builtin functions"): + test_raises[ValueError, curry(print)] # builtin function that fails `inspect.signature` + + # Internal feature, used by curry macro. If uninspectables are said to be ok, + # then attempting to curry an uninspectable simply returns the original function. + m1 = print + m2 = curry(print, _curry_allow_uninspectable=True) + test[the[m2] is the[m1]] with testset("curry integration with @generic"): # v0.15.0+ @generic From 292967f51e5a002f0f344f1a78b467cd9a360d5a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:30:00 +0300 Subject: [PATCH 331/832] fix tests, this time for sure --- unpythonic/tests/test_arity.py | 2 +- unpythonic/tests/test_fun.py | 17 ++++++++++------- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/unpythonic/tests/test_arity.py b/unpythonic/tests/test_arity.py index 5a3307d4..8716ecd0 100644 --- a/unpythonic/tests/test_arity.py +++ b/unpythonic/tests/test_arity.py @@ -107,7 +107,7 @@ def instmeth(self): # Methods of builtin types have uninspectable arity up to Python 3.6. # Python 3.7 seems to fix this at least for `list`, and PyPy3 (7.3.0; Python 3.6.9) # doesn't have this error either. - if sys.version_info < (3, 7, 0) and sys.implementation.name == "cpython": + if sys.version_info < (3, 7, 0) and sys.implementation.name == "cpython": # pragma: no cover with testset("uninspectable builtin methods"): lst = [] test_raises[UnknownArity, arities(lst.append)] diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 1f9109ab..20166b5e 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -4,6 +4,7 @@ from ..test.fixtures import session, testset, returns_normally from collections import Counter +import sys from ..dispatch import generic from ..fun import (memoize, partial, curry, apply, @@ -205,14 +206,16 @@ def double(x): # a `with test` can optionally return a value, which becomes the asserted expr. return curry(double, 2, "foo") == (4, "foo") - with testset("uninspectable builtin functions"): - test_raises[ValueError, curry(print)] # builtin function that fails `inspect.signature` + # This doesn't occur on PyPy3. + if sys.implementation.name == "cpython": # pragma: no cover + with testset("uninspectable builtin functions"): + test_raises[ValueError, curry(print)] # builtin function that fails `inspect.signature` - # Internal feature, used by curry macro. If uninspectables are said to be ok, - # then attempting to curry an uninspectable simply returns the original function. - m1 = print - m2 = curry(print, _curry_allow_uninspectable=True) - test[the[m2] is the[m1]] + # Internal feature, used by curry macro. If uninspectables are said to be ok, + # then attempting to curry an uninspectable simply returns the original function. + m1 = print + m2 = curry(print, _curry_allow_uninspectable=True) + test[the[m2] is the[m1]] with testset("curry integration with @generic"): # v0.15.0+ @generic From bf23eb74a6472b2d41d5a26403a263e09cd4e2fb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:37:57 +0300 Subject: [PATCH 332/832] add first tests for curry kwargs support --- unpythonic/tests/test_fun.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 20166b5e..da69994a 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -217,6 +217,15 @@ def double(x): m2 = curry(print, _curry_allow_uninspectable=True) test[the[m2] is the[m1]] + with testset("curry kwargs support"): + @curry + def testing12(x, y): + return (x, y) + test[testing12(1)(2) == (1, 2)] + test[testing12(1)(y=2) == (1, 2)] + test[testing12(x=1)(y=2) == (1, 2)] + test[testing12(y=2)(x=1) == (1, 2)] + with testset("curry integration with @generic"): # v0.15.0+ @generic def f(x: int): From 31d5942b53cb095210b999bb3fe61ffce76bf545 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:50:27 +0300 Subject: [PATCH 333/832] add comment --- unpythonic/fun.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 3364ab20..4822f585 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -292,7 +292,7 @@ def f(x, y): return maybe_force_args(f, *args, **kwargs) return f - def fallback(): + def fallback(): # what to do when inspection fails if not _curry_allow_uninspectable: # usual behavior raise # co-operate with unpythonic.syntax.autocurry; don't crash on builtins From e1f81e5ee0c42383af99d52bf3568cb12f7b2462 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 20 May 2021 03:50:42 +0300 Subject: [PATCH 334/832] improve curry tests --- unpythonic/tests/test_fun.py | 39 +++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index da69994a..ac859678 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -220,12 +220,24 @@ def double(x): with testset("curry kwargs support"): @curry def testing12(x, y): - return (x, y) + return x, y test[testing12(1)(2) == (1, 2)] test[testing12(1)(y=2) == (1, 2)] test[testing12(x=1)(y=2) == (1, 2)] test[testing12(y=2)(x=1) == (1, 2)] + @curry + def makemul(x): + def mymul(y): + return x * y + return mymul + test[callable(makemul())] # not enough args/kwargs yet + test[makemul(2)(3) == 6] # just enough args + test[makemul(2, 3) == 6] # extra args + test[makemul(2, y=3) == 6] # extra kwargs, fine if callable intermediate result can accept them + test[makemul(x=2, y=3) == 6] + test[makemul(y=3, x=2) == 6] + with testset("curry integration with @generic"): # v0.15.0+ @generic def f(x: int): @@ -239,8 +251,29 @@ def f(x: float, y: str): # noqa: F811, new multimethod for the same generic fun # the call signature does not match fully. But it does match partially, so in that case # `curry` waits for more arguments (because there is at least one multimethod that matches # the partial arguments given so far). - test[callable(curry(f, 3.14))] - test[curry(f, 3.14, "cat") == "float, str"] + test[callable(curry(f, 3.14))] # partial match + test[curry(f, 3.14, "cat") == "float, str"] # exact match + + # Partial match, but let's use the return value of `curry` (does it chain correctly?). + tmp = curry(f, 3.14) + test[tmp("cat") == "float, str"] + + @curry + @generic + def makemul_typed(x: int): + @generic + def mymul_typed(y: int): + return x * y + return mymul_typed + test[callable(makemul_typed())] # not enough args/kwargs yet + test[makemul_typed(2)(3) == 6] # just enough args + test[makemul_typed(2, 3) == 6] # extra args + test[makemul_typed(2, y=3) == 6] # extra kwargs, fine if callable intermediate result can accept them + test[makemul_typed(x=2, y=3) == 6] + test[makemul_typed(y=3, x=2) == 6] + test_raises[TypeError, makemul_typed(2.0)] # only defined for int + test_raises[TypeError, makemul_typed(2.0, 3)] # should notice it even with extra args + test_raises[TypeError, makemul_typed(2, 3.0)] with testset("compose"): double = lambda x: 2 * x From aa547e2ddeef805a15badaa5ad9db3332228e8a3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 21 May 2021 01:50:53 +0300 Subject: [PATCH 335/832] improve comment --- unpythonic/fun.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 4822f585..2fce4439 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -748,9 +748,10 @@ def composed(*args): bindings = {"curry_context": dyn.curry_context + [composed]} with dyn.let(**bindings): a = maybe_force_args(g, *args) - # we could duck-test, but this is more predictable for the user - # (consider chaining functions that manipulate a generator), and - # tuple specifically is the pythonic multiple-return-values thing. + # we could duck-test for an iterable, but this is more predictable + # for the user (consider chaining functions that manipulate a + # generator), and tuple specifically is the pythonic + # multiple-return-values thing. if isinstance(a, tuple): return maybe_force_args(f, *a) return maybe_force_args(f, a) From 8115ad7f0c61cd898c301fc8b23dddf884e76f54 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:01:23 +0300 Subject: [PATCH 336/832] Revise multiple-return-values, add named return values. Also, improve lazify support for function composition utilities such as the `pipe` family. --- CHANGELOG.md | 9 + CONTRIBUTING.md | 3 +- doc/dialects/lispython.md | 1 + doc/features.md | 31 ++- unpythonic/collections.py | 223 +++++++++++++++++++- unpythonic/dialects/lispython.py | 2 +- unpythonic/dialects/tests/test_lispython.py | 2 +- unpythonic/dialects/tests/test_pytkell.py | 3 +- unpythonic/fun.py | 173 ++++++++++----- unpythonic/seq.py | 184 +++++++++++----- unpythonic/syntax/lazify.py | 7 +- unpythonic/syntax/tailtools.py | 74 ++++--- unpythonic/syntax/tests/test_conts.py | 35 +-- unpythonic/syntax/tests/test_lazify.py | 7 +- unpythonic/syntax/tests/test_tco.py | 3 +- unpythonic/tests/test_fold.py | 10 +- unpythonic/tests/test_fun.py | 53 +++-- unpythonic/tests/test_it.py | 7 +- unpythonic/tests/test_seq.py | 72 ++++--- 19 files changed, 666 insertions(+), 233 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac01b550..2b0bf990 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -145,6 +145,15 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Drop support for deprecated argument format for `raisef`. Now the usage is `raisef(exc)` or `raisef(exc, cause=...)`. These correspond exactly to `raise exc` and `raise exc from ...`, respectively. - **Other backward-incompatible API changes.** + - Multiple-return-value handling changed. Resolves issue [#32](https://github.com/Technologicat/unpythonic/issues/32). + - Multiple return values are now denoted as `Values`, available from the top-level namespace of `unpythonic`. + - The `Values` constructor accepts both positional and named arguments. Passing in named arguments creates **named return values**. This completes the symmetry between argument passing and returns. + - Most of the time, it's still fine to return a tuple and destructure that; but in contexts where it is important to distinguish between a single `tuple` return value and multiple return values, it is preferable to use `Values`. + - In any utilities that deal with function composition, if your intent is multiple-return-values, **it is now mandatory to return a `Values`** instead of a `tuple`: + - `curry` + - `pipe` family + - `compose` family + - All multiple-return-values in code using the `with continuations` macro. (The continuations system essentially composes continuation functions.) - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **API differences.** - The macros `lazy` and `f` can be imported from the syntax interface module, `unpythonic.syntax`, and the class `Lazy` is available at the top level of `unpythonic`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3975b693..50257a42 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -62,7 +62,8 @@ - For example: - Not only a summarizing `minmax` utility, but `running_minmax` as well. The former is then just a one-liner expressed in terms of the latter. - `foldl` accepts multiple iterables, has a switch to terminate either on the shortest or on the longest input, and takes its arguments in a curry-friendly order. It also *requires* at least one iterable, so that `curry` knows to not trigger the call until at least one iterable has been provided. - - `curry` changes Python's reduction semantics to be more similar to Haskell's, to pass extra arguments through on the right, and keep calling if an intermediate result is a function, and there are still such passed-through arguments remaining. This extends what can be expressed concisely, [for example](http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html) a classic lispy `map` is `curry(lambda f: curry(foldr, composerc(cons, f), nil))`. Feed that a function and an iterable, and get a linked list with the mapped results. Note the arity mismatch; `f` is 1-to-1, but `cons` is 2-to-1. + - `curry` changes Python's reduction semantics to be more similar to Haskell's, to pass extra arguments through, and keep calling if an intermediate result is a function, and there are still such passed-through arguments remaining. This extends what can be expressed concisely, [for example](http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html) a classic lispy `map` is `curry(lambda f: curry(foldr, composerc(cons, f), nil))`. Feed that a function and an iterable, and get a linked list with the mapped results. Note the arity mismatch; `f` is 1-to-1, but `cons` is 2-to-1. + - `curry` also supports our `@generic` functions, and named return values... - **Make features work together** when it makes sense. Aim at composability. Try to make features orthogonal when reasonably possible, so that making them work together requires no extra effort. When not possible, purposefully minimizing friction in interaction between features makes for a coherent, easily understandable language extension. - **Be concise but readable**, like in mathematics. diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index b462c343..3874914a 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -85,6 +85,7 @@ We also import some macros and functions to serve as dialect builtins: - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax`` - ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil``, ``prod`` - ``dyn``, for dynamic assignment + - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, the `pipe` family, the `compose` family, and the `with continuations` macro.) For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. diff --git a/doc/features.md b/doc/features.md index 8fa3ab3a..abfcbe04 100644 --- a/doc/features.md +++ b/doc/features.md @@ -998,15 +998,17 @@ Things missing from the standard library. - **Changed in v0.15.0.** `curry` supports both positional and named arguments, and binds arguments to function parameters like Python itself does. The call triggers when all parameters are bound, regardless of whether they were passed by position or by name, and at which step of the currying process they were passed. - **Changed in v0.15.0.** `unpythonic`'s multiple-dispatch system (`@generic`, `@typed`) is supported. `curry` looks for an exact match first, then a match with extra args/kwargs, and finally a partial match. If there is still no match, this implies that at least one parameter would get a binding that fails the type check. In such a case `TypeError` regarding failed multiple dispatch is raised. - **Changed in v0.15.0.** If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`. - - Passthrough when too many args (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket). Positional args are passed through **on the right**. - - If the intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining args and kwargs. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). - - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. + - Passthrough for args/kwargs that are incompatible with the target function's call signature (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket). + - Here *incompatible* means too many positional args, or named args that have no corresponding parameter. (Note that if the function has a `**kwargs` parameter, then all named args are considered compatible, because it absorbs anything.) + - Multiple return values (both positional and named) are denoted using `Values` (which see). A standard return value is considered to consist of one positional return value only. + - Positional args are passed through **on the right**. Any positional return values of the curried function are prepended, on the left. + - If the first positional return value of an intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining args and kwargs, after merging the rest of the return values into the args and kwargs. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). + - If more args/kwargs are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. - To override, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. - To set the dynvar, `from unpythonic import dyn`, and then `with dyn.let(curry_context=...):`. - - Even with the upgrades in v0.15.0, passing through *named* args to an outer curry context is not supported. This may or may not change in the future; fixing this requires support for named return values. See issue [#32](https://github.com/Technologicat/unpythonic/issues/32). - Can be used both as a decorator and as a regular function. - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. - - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print`` or ``range``. + - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python you have (and whether CPython or PyPy3). - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values packed into a tuple are unpacked to the argument list of the next function in the chain. @@ -1090,7 +1092,7 @@ assert myzipr((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) # zip first assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # reverse first -# curry with passthrough on the right +# curry with passthrough (positionals passed through on the right) # final result is a tuple of the result(s) and the leftover args double = lambda x: 2 * x with dyn.let(curry_context=["whatever"]): # set a context to allow passthrough to the top level @@ -1134,7 +1136,7 @@ Yet another way to write ``map_one`` is: mymap = lambda f: curry(foldr, composer(cons, curry(f)), nil) ``` -The curried ``f`` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; this two-tuple then ends up as the arguments to ``cons``. +The curried ``f`` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; these two values then end up as the arguments to ``cons``. Using a currying compose function (name suffixed with ``c``), the inner curry can be dropped: @@ -1146,9 +1148,11 @@ assert curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9) This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy)) as we're gonna get in Python. -Notice how the last two versions accept multiple input iterables; this is thanks to currying ``f`` inside the composition. An element from each of the iterables is taken by the processing function ``f``. Being the last argument, ``acc`` is passed through on the right. The output from the processing function - one new item - and ``acc`` then become a two-tuple, passed into cons. +Notice how the last two versions accept multiple input iterables; this is thanks to currying ``f`` inside the composition. An element from each of the iterables is taken by the processing function ``f``. Being the last argument, ``acc`` is passed through on the right. The output from the processing function - one new item - and ``acc`` then become two arguments, passed into cons. -Finally, keep in mind this exercise is intended as a feature demonstration. In production code, the builtin ``map`` is much better. +Finally, keep in mind this exercise is intended as a feature demonstration. In production code, the builtin ``map`` is much better. It produces a lazy iterable, and does not care which kind of actual data structure the items will be stored in (once computed). + +The example we have here evaluates all items immediately, and specifically produces a linked list. It's just a nice example of function composition involving incompatible arities, thus demonstrating the kind of situation where the passthrough feature of `curry` is useful. It is taken from a paper by [John Hughes (1984)](https://www.cse.chalmers.se/~rjmh/Papers/whyfp.html). #### ``curry`` and reduction rules @@ -1172,7 +1176,7 @@ it means the following. Let ``m1`` and ``m2`` be the minimum and maximum positio - If ``n < m1``, partially apply ``f`` to the given arguments, yielding a new function with smaller ``m1``, ``m2``. Then curry the result and return it. - Internally we stack ``functools.partial`` applications, but there will be only one ``curried`` wrapper no matter how many invocations are used to build up arguments before ``f`` eventually gets called. -As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the presence of kwargs and `@generic` functions, is: +As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the presence of kwargs, `@generic` functions, and `Values` multiple-return-values, is: - If `f` is **not** `@generic` or `@typed`: - Compute parameter bindings of the args and kwargs collected so far, against the call signature of `f`. @@ -1180,7 +1184,12 @@ As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the - If there are no unbound parameters, and no args/kwargs are left over, we have an exact match. Call `f` and return its result, like a normal function call. - Any sequence of curried calls that ends up binding all parameters of `f` triggers the call. - As before, beware when working with variadic functions. Particularly, keep in mind that `*args` matches **zero or more** positional arguments (as the [Kleene star](https://en.wikipedia.org/wiki/Kleene_star)-ish notation indeed suggests). - - If there are no unbound parameters, but there are args/kwargs left over, arrange passthrough for the leftover args/kwargs (that were rejected by the call signature of `f`), and call `f`. If the result is a callable, curry it, and recurse. Else form a tuple... (as above). + - If there are no unbound parameters, but there are args/kwargs left over, arrange passthrough for the leftover args/kwargs (that were rejected by the call signature of `f`), and call `f`. Any leftover positional arguments are passed through **on the right**. + - Merge the return value of `f` with the leftover args/kwargs, thus forming updated leftover args/kwargs. + - If the return value of `f` is a `Values`: prepend positional return values into the leftover args (i.e. insert them **on the left**), and update the leftover kwargs with the named return values. (I.e. a key name conflict causes an overwrite in the leftover kwargs.) + - Else: there is just one positional return value. Prepend it to the leftover args. + - If the first positional return value is a callable: remove it from the leftover args, curry it, and recurse with the (updated) leftover args/kwargs. + - Else: form a `Values` from the leftover args/kwargs, and return it. (This return goes to the next outer curry context, or at the top level, to the original caller.) - If neither of the above match, we know there is at least one unbound parameter, i.e. we have a partial match. Keep currying. - If `f` is `@generic` or `@typed`: - Iterate over multimethods registered on `f`, **up to three times**. diff --git a/unpythonic/collections.py b/unpythonic/collections.py index 0a7d6dbe..ea324c87 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -5,7 +5,8 @@ "frozendict", "roview", "view", "ShadowedSequence", "mogrify", "get_abcs", "in_slice", "index_in_slice", - "SequenceView", "MutableSequenceView"] # ABCs + "SequenceView", "MutableSequenceView", # ABCs + "Values", "valuify"] from functools import wraps from itertools import repeat @@ -19,10 +20,12 @@ from operator import lt, le, ge, gt import threading -from .llist import cons, Nil -from .misc import getattrrec from .env import env from .dynassign import _Dyn +from .lazyutil import passthrough_lazy_args +from .llist import cons, Nil +from .misc import getattrrec +from .regutil import register_decorator def get_abcs(cls): """Return a set of the collections.abc superclasses of cls (virtuals too).""" @@ -75,8 +78,12 @@ def mogrify(func, container): just like in ``map``. """ def doit(x): + if isinstance(x, Values): + new_rets = doit(x.rets) + new_kwrets = doit(x.kwrets) + return Values(*new_rets, **new_kwrets) # mutable containers - if isinstance(x, MutableSequence): + elif isinstance(x, MutableSequence): y = [doit(elt) for elt in x] if hasattr(x, "clear"): x.clear() # list has this, but not guaranteed by MutableSequence @@ -142,6 +149,9 @@ def doit(x): # ----------------------------------------------------------------------------- +# TODO: Make `box` support pickle with per-use-site uuids, to keep a shared box shared across a pickle roundtrip? +# TODO: See the `gsym` implementation in `unpythonic.symbol` for how to do this in a thread-safe manner. +# TODO: Think how those semantics should work with `ThreadLocalBox`. class box: """Minimalistic, mutable single-item container à la Racket. @@ -893,3 +903,208 @@ def _canonize_slice(s, length=None, wrap=None): # convert negatives, inject def stop = -1 # yes, really -1 to have index 0 inside the slice return start, stop, step + +# ----------------------------------------------------------------------------- + +@passthrough_lazy_args +class Values: + """Structured multiple-return-values. + + That is, return multiple values positionally and by name. This completes + the symmetry between passing function arguments and returning values + from a function: Python itself allows passing arguments by name, but has + no concept of returning values by name. This class adds that concept. + + Having a `Values` type separate from `tuple` also helps with semantic + accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value now + means just that - one value that is a `tuple`. It is different from a + `Values` that contains several positional return values (that are meant + to be treated separately). + + **When to use**: + + Most of the time, returning a tuple to denote multiple-return-values + and unpacking it is just fine, and that is exactly what `unpythonic` + does internally in many places. + + But the distinction is critically important for function composition, + so that positional return values can be automatically mapped into + positional arguments to the next function in the chain, and named + return values into named arguments. + + Accordingly, various parts of `unpythonic` that deal with function + composition use the `Values` abstraction; particularly `curry`, and + the `compose` and `pipe` families. + + **Behavior**: + + `Values` is a duck-type with some features of both sequences and mappings, + but not the full `collections.abc` API of either. + + Each operation that obviously and without ambiguity makes sense only + for the positional or named part, accesses that part. + + The only exception is `__getitem__` (subscripting), which makes sense + for both parts, unambiguously, because the key types differ. If the index + expression is an `int` or a `slice`, it is an index/slice for the + positional part. If it is an `str`, it is a key for the named part. + + If you need to explicitly access either part (and its full API), + use the `rets` and `kwrets` attributes. The names are in analogy + with `args` and `kwargs`. + + `rets` is a `tuple`, and `kwrets` is a `frozendict`. + + `Values` objects can be compared for equality. Two `Values` objects + are equal if both their `rets` and `kwrets` (respectively) are. + + Examples:: + + def f(): + return Values(1, 2, 3) + result = f() + assert isinstance(result, Values) + assert result.rets == (1, 2, 3) + assert not result.kwrets + assert result[0] == 1 + assert result[:-1] == (1, 2) + a, b, c = result # if no kwrets, can be unpacked like a tuple + a, b, c = f() + + def g(): + return Values(x=3) # named return value + result = g() + assert isinstance(result, Values) + assert not result.rets + assert result.kwrets == {"x": 3} # actually a `frozendict` + assert "x" in result # `in` looks in the named part + assert result["x"] == 3 + assert result.get("x", None) == 3 + assert result.get("y", None) == None + assert tuple(results.keys()) == ("x",) # also `values()`, `items()` + + def h(): + return Values(1, 2, x=3) + result = h() + assert isinstance(result, Values) + assert result.rets == (1, 2) + assert result.kwrets == {"x": 3} + a, b = result.rets # positionals can always be unpacked explicitly + assert result[0] == 1 + assert "x" in result + assert result["x"] == 3 + + def silly_but_legal(): + return Values(42) + result = silly_but_legal() + assert result.rets[0] == 42 + assert result.ret == 42 # shorthand for single-value case + + The last example is silly, but legal, because it is preferable to just omit + the `Values` if it is known that there is only one return value. (This also + applies when that value is a `tuple`, when the intent is to return it as a + single `tuple`, in contexts where this distinction matters.) + """ + def __init__(self, *rets, **kwrets): + """Create a `Values` object. + + `rets`: positional return values + `kwrets`: named return values + """ + self.rets = rets + self.kwrets = frozendict(kwrets) + + # Shorthand for one-value case + def _ret(self): + return self.rets[0] + ret = property(fget=_ret, doc="Shorthand for `self.rets[0]`. Read-only.") + + # Iterable + def __iter__(self): + """Values is iterable when there are no `kwrets`; this then iterates over `rets`. + + This is meant to minimize impact on existing code that receives a `tuple` + as a pythonic multiple-return-values idiom. Changing the `return` to + return a `Values` instead requires no changes at the receiving end + (unless you change the sending end to return some named values; + if you do, then it *should* yell, to avoid silently discarding + those named values). + + Note that you can iterate over `rets` or `kwrets` to explicitly state + which you mean; that always works. + """ + if self.kwrets: + raise ValueError(f"Named values present, cannot iterate over all values. Got: {self.kwrets}") + return iter(self.rets) + + # Sequence (no full support: no `__len__`, `__reversed__`, `index`, `count`) + def __getitem__(self, idx): + """Subscripting. + + Indexing by an `int` or `slice` indexes the positional part. + Indexing by an `str` indexes the named part. + + Indexing by any other type raises `TypeError`. + """ + # multi-headed hydra + if isinstance(idx, (int, slice)): + return self.rets[idx] + elif isinstance(idx, str): + return self.kwrets[idx] + raise TypeError(f"Expected either int, slice or str subscript, got {type(idx)} with value {repr(idx)}") + + # Container + def __contains__(self, k): + """The `in` operator, looks in the named part.""" + return k in self.kwrets + + # Mapping (no full support: no `__len__`) + def items(self): + """Items of the named part.""" + return self.kwrets.items() + def keys(self): + """Keys of the named part.""" + return self.kwrets.keys() + def values(self): + """Values of the named part.""" + return self.kwrets.values() + def get(self, k, default=None): + """Dict-like `get` for the named part.""" + return self[k] if k in self else default + + # comparison + def __eq__(self, other): + """Equality comparison. + + Two `Values` objects are equal if both their `rets` and `kwrets` + (respectively) are. + """ + if not isinstance(other, Values): + return False + return other.rets == self.rets and other.kwrets == self.kwrets + def __ne__(self, other): + """Inequality comparison.""" + return not (self == other) + + # no `__len__`, because we have two candidates + + # pretty-printing + def __repr__(self): # pragma: no cover + """Pretty-printing. Eval-able if the contents are.""" + rets_list = [repr(x) for x in self.rets] + rets_str = ", ".join(rets_list) + kwrets_list = [f"{name}={repr(value)}" for name, value in self.kwrets.items()] + kwrets_str = ", ".join(kwrets_list) + sep = ", " if self.rets and self.kwrets else "" + return f"Values({rets_str}{sep}{kwrets_str})" + +@register_decorator(priority=30) +def valuify(f): + """Decorator. If `f` returns `tuple` (exactly, no subclass), convert into `Values`, else pass through.""" + @wraps(f) + def valuified(*args, **kwargs): + result = f(*args, **kwargs) + if type(result) is tuple: # yes, exactly tuple + result = Values(*result) + return result + return valuified diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index 6949db6b..32c50cf4 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -36,7 +36,7 @@ def transform_ast(self, tree): # tree is an ast.Module local, delete, do, do0, let_syntax, abbrev, block, expr, cond) - from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn # noqa: F401, F811 + from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn, Values # noqa: F401, F811 with autoreturn, quicklambda, multilambda, tco, namedlambda: __paste_here__ # noqa: F821, just a splicing marker. tree.body = splice_dialect(tree.body, template, "__paste_here__") diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index 35988992..9e3cba42 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -125,7 +125,7 @@ def f(k, acc): def setk(*args, cc): nonlocal k k = cc # current continuation, i.e. where to go after setk() finishes - args # tuple means multiple-return-values + Values(*args) # multiple-return-values # noqa: F821, Lispython imports Values by default. def doit(): lst = ['the call returned'] *more, = call_cc[setk('A')] diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index 55fb016e..e4bc69e5 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -8,6 +8,7 @@ from ...test.fixtures import session, testset from ...syntax import macros, continuations, call_cc, tco # noqa: F401, F811 +from ...collections import Values from ...misc import timer from types import FunctionType @@ -180,7 +181,7 @@ def f(a, b): def setk(*args, cc): nonlocal k k = cc # current continuation, i.e. where to go after setk() finishes - return args # tuple means multiple-return-values + return Values(*args) # multiple-return-values def doit(): lst = ['the call returned'] *more, = call_cc[setk('A')] diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 2fce4439..121d1499 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -282,6 +282,8 @@ def f(x, y): return named outputs. See: https://github.com/Technologicat/unpythonic/issues/32 """ + from .collections import Values # circular import + f = force(f) # lazify support: we need the value of f # trivial case first: interaction with call_ec and other replace-def-with-value decorators if not callable(f): @@ -449,22 +451,66 @@ def curried(*args, **kwargs): now_kwargs = {k: v for k, v in kwargs.items() if k not in later_kwargs} now_result = maybe_force_args(f, *now_args, **now_kwargs) - now_result = force(now_result) if not isinstance(now_result, tuple) else force1(now_result) - if callable(now_result): - # Curry it now, to sustain the chain in case we have too many (or too few) args for it. - if not iscurried(now_result): - now_result = curry(now_result) - return now_result(*later_args, **later_kwargs) - # TODO: To handle later_kwargs here, we need named return values. See issue #32. - # https://github.com/Technologicat/unpythonic/issues/32 - if later_kwargs: - raise NotImplementedError(f"Passing through named arguments to an outer curry context not implemented; got {later_kwargs}") + now_result = force1(now_result) # just in case it's a `Lazy` + + # Inspect the return value(s). + # - Inject the appropriate items to `later_args` and `later_kwargs`. + if isinstance(now_result, Values): # multiple-return-values + if now_result.rets: + # `leftmost`, not `first`, for unambiguous stack traces. + leftmost, *others = now_result.rets + leftmost = force1(leftmost) + + # Extra positional arguments (`later_args`) are passed through *on the right*. + # Hence any further positional return values are inserted before them. + if callable(leftmost): + # If the leftmost return value is a callable, omit it from `later_args`, + # since we will call it. + later_args = tuple(others) + later_args + else: + later_args = (leftmost,) + tuple(others) + later_args + else: + # No positional return values; no changes to `later_args`. + leftmost = None + + # In case of name conflicts, named return values override earlier extra named arguments. + # (This follows the execution order: arguments were passed in, then the function ran.) + # TODO: This way, or allow named arguments to override a named return value? + # TODO: Which choice is more useful practically or mathematically? + if now_result.kwrets: + later_kwargs = {**later_kwargs, **now_result.kwrets} + else: + # The only return value is also the leftmost one. + leftmost = force1(now_result) + if callable(leftmost): + pass + else: + later_args = (leftmost,) + later_args + + # If the first positional return value is a callable, curry it and recurse. + # Currying sustains the chain in case the next action is `_call_with_passthrough` + # or `_keep_currying`. + if callable(leftmost): + if not iscurried(leftmost): + leftmost = curry(leftmost) + return leftmost(*later_args, **later_kwargs) + + # The first positional return value is not a callable. Pass the return value(s) through + # to the curried procedure waiting in outerctx (e.g. in a curried compose chain). + # + # If there is no outer curry context (i.e. we are the top-level curry context), + # by default it is an error to have any args/kwargs left over, to avoid common + # human error. (To explicitly state such intent, `with dyn.let(curry_context=["whatever"])`.) if not outerctx: - raise TypeError(f"Top-level curry context exited with {len(later_args) + len(later_kwargs)} arg(s) remaining. Positional: {later_args}, named: {later_kwargs}") - # Pass through to the curried procedure waiting in outerctx (e.g. in a curried compose chain). - if isinstance(now_result, tuple): - return now_result + later_args - return (now_result,) + later_args + num_positional_msg = f"{len(later_args)} positional" + num_named_msg = f"{len(later_kwargs)} named" + num_sep = " and " if later_args and later_kwargs else "" + plural = "s" if len(later_args) + len(later_kwargs) != 1 else "" + positional_msg = f"positional: {later_args}" + named_msg = f"named: {later_kwargs}" + sep = "; " if later_args and later_kwargs else "" + raise TypeError(f"Top-level curry context exited with {num_positional_msg}{num_sep}{num_named_msg} argument{plural} remaining; {positional_msg}{sep}{named_msg}") + return Values(*later_args, **later_kwargs) elif action is _keep_currying: # Fail-fast: use our `partial` wrapper to type-check the partial call signature @@ -564,37 +610,39 @@ def apply(f, arg0, *more, **kwargs): # Not marking this as lazy-aware works better with continuations (since this # is the default cont, and return values should be values, not lazy[]) -def identity(*args): +def identity(*args, **kwargs): """Identity function. - Accepts any positional arguments, and returns them. + Accepts any args and kwargs, and returns them. - Packs into a tuple if there is more than one. + Packs into a `Values` if anything other than one positional arg. Example:: - assert identity(1, 2, 3) == (1, 2, 3) + assert identity(1, 2, 3) == Values(1, 2, 3) assert identity(42) == 42 assert identity() is None """ - if not args: + from .collections import Values # circular import + if not args and not kwargs: return None - return args if len(args) > 1 else args[0] + return Values(*args, **kwargs) if kwargs or len(args) > 1 else args[0] # In lazify, return values are always just values, so we have to force args # to compute the return value; as a shortcut, just don't mark this as lazy. -def const(*args): +def const(*args, **kwargs): """Constant function. Returns a function that accepts any arguments (also kwargs) - and returns the args given here (packed into a tuple if more than one). + and returns the args and kwargs given here (packed into a `Values` + if anything other than one positional arg). Example:: c = const(1, 2, 3) - assert c(42, "foo") == (1, 2, 3) - assert c("anything") == (1, 2, 3) - assert c() == (1, 2, 3) + assert c(42, "foo") == Values(1, 2, 3) + assert c("anything") == Values(1, 2, 3) + assert c() == Values(1, 2, 3) c = const(42) assert c("anything") == 42 @@ -602,10 +650,11 @@ def const(*args): c = const() assert c("anything") is None """ - if not args: + from .collections import Values # circular import + if not args and not kwargs: ret = None else: - ret = args if len(args) > 1 else args[0] + ret = Values(*args, **kwargs) if kwargs or len(args) > 1 else args[0] def constant(*a, **kw): return ret return constant @@ -681,7 +730,7 @@ def disjoined(*args, **kwargs): def _make_compose1(direction): # "left", "right" def compose1_two(f, g): # return lambda x: f(g(x)) - return lambda x: maybe_force_args(f, maybe_force_args(g, x)) + return lambda x: maybe_force_args(f, force1(maybe_force_args(g, x))) if direction == "right": compose1_two = flip(compose1_two) def compose1(fs): @@ -737,28 +786,56 @@ def composel1i(iterable): """Like composel1, but read the functions from an iterable.""" return _compose1_left(iterable) -def _make_compose(direction): # "left", "right" +def _make_compose(direction): + """Make a function that composes functions from an iterable. + + Return value is a function `compose(fs)` -> `composed(*args, **kwargs)`. + + `direction`: str, one of "left", "right". Which way to compose. + + For example, let `fs = (f1, f2, f3)`. + + If `direction == "left"`, `composed` computes f3(f2(f1(...))); + the functions apply leftmost first. + + If `direction == "right"`, `composed` computes f1(f2(f3(...))); + the functions apply rightmost first. + + Standard mathematical function composition notation f1 ∘ f2 ∘ f3 takes rightmost first, + but we refuse the temptation to guess. We provide only explicit `l` and `r` variants + of all the `compose` utilities. + """ def compose_two(f, g): - def composed(*args): + """g is applied first, then f. + + (f ∘ g)(...) ≡ f(g(...)) + """ + from .collections import Values # circular import + def composed(*args, **kwargs): bindings = {} if iscurried(f): - # co-operate with curry: provide a top-level curry context + # Co-operate with curry: provide a top-level curry context # to allow passthrough from the function that is applied first # to the function that is applied second. bindings = {"curry_context": dyn.curry_context + [composed]} with dyn.let(**bindings): - a = maybe_force_args(g, *args) - # we could duck-test for an iterable, but this is more predictable - # for the user (consider chaining functions that manipulate a - # generator), and tuple specifically is the pythonic - # multiple-return-values thing. - if isinstance(a, tuple): - return maybe_force_args(f, *a) + a = maybe_force_args(g, *args, **kwargs) + a = force1(a) # just in case it's a `Lazy` + # We could duck-test for an iterable, but this is more predictable. + if isinstance(a, Values): + return maybe_force_args(f, *a.rets, **a.kwrets) return maybe_force_args(f, a) return composed if direction == "right": compose_two = flip(compose_two) def compose(fs): + """Compose functions from iterable `fs`. + + **CAUTION**: This is a closure. Which way to compose (left or right) + was chosen when this closure instance was created. Please use the + public API functions whose names explicitly state the direction. + """ + fs = force(fs) composed = reducel(compose_two, fs) # op(elt, acc) if all(islazy(f) for f in fs): composed = passthrough_lazy_args(composed) @@ -769,16 +846,15 @@ def compose(fs): _compose_right = _make_compose("right") def composer(*fs): - """Compose functions accepting only positional args. Right to left. + """Compose functions. Right to left. This mirrors the standard mathematical convention (f ∘ g)(x) ≡ f(g(x)). - At each step, if the output from a function is a tuple, - it is unpacked to the argument list of the next function. Otherwise, - we assume the output is intended to be fed to the next function as-is. + We support passing both positional and named values. - Especially, generators, namedtuples and any custom classes will **not** be - unpacked, regardless of whether or not they support the iterator protocol. + At each step, if the output from a function is a `Values`, it is unpacked + to the args and kwargs of the next function. Otherwise, we feed the output + to the next function as a single positional argument. """ return composeri(fs) @@ -827,12 +903,15 @@ def composelci(iterable): def tokth(k, f): """Return a function to apply f to args[k], pass the rest through. + The output is a `Values`. Named arguments are passed through as-is. + Negative indices also supported. Especially useful in multi-arg compose chains. See ``unpythonic.test.test_fun`` for examples. """ - def apply_f_to_kth_arg(*args): + from .collections import Values # circular import + def apply_f_to_kth_arg(*args, **kwargs): n = len(args) if not n: raise TypeError("Expected at least one argument") @@ -844,7 +923,7 @@ def apply_f_to_kth_arg(*args): out.append(maybe_force_args(f, args[j])) # mth argument if n > m: out.extend(args[m:]) - return tuple(out) + return Values(*out, **kwargs) if islazy(f): apply_f_to_kth_arg = passthrough_lazy_args(apply_f_to_kth_arg) return apply_f_to_kth_arg diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 1d325b7a..e9ee21e0 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -8,10 +8,13 @@ "do", "do0", "assign"] from collections import namedtuple + +from .arity import arity_includes, UnknownArity +from .collections import Values +from .dynassign import dyn from .env import env from .fun import curry, iscurried -from .dynassign import dyn -from .arity import arity_includes, UnknownArity +from .lazyutil import force1, force, maybe_force_args, passthrough_lazy_args from .symbol import sym # sequence side effects in a lambda @@ -23,6 +26,10 @@ def begin(*vals): f = lambda x: begin(print("hi"), 42*x) print(f(1)) # 42 + + **CAUTION**: For regular code only. If you use macros, prefer `do[]`; + the macro layer of `unpythonic` recognizes only the `do` constructs + as a sequencing abstraction. """ return vals[-1] if len(vals) else None @@ -34,6 +41,10 @@ def begin0(*vals): # eager, bodys already evaluated when this is called g = lambda x: begin0(23*x, print("hi")) print(g(1)) # 23 + + **CAUTION**: For regular code only. If you use macros, prefer `do0[]`; + the macro layer of `unpythonic` recognizes only the `do` constructs + as a sequencing abstraction. """ return vals[0] if len(vals) else None @@ -46,6 +57,10 @@ def lazy_begin(*bodys): f = lambda x: lazy_begin(lambda: print("hi"), lambda: 42*x) print(f(1)) # 42 + + **CAUTION**: For regular code only. If you use macros, prefer `do[]`; + the macro layer of `unpythonic` recognizes only the `do` constructs + as a sequencing abstraction. """ n = len(bodys) if not n: @@ -67,6 +82,10 @@ def lazy_begin0(*bodys): g = lambda x: lazy_begin0(lambda: 23*x, lambda: print("hi")) print(g(1)) # 23 + + **CAUTION**: For regular code only. If you use macros, prefer `do0[]`; + the macro layer of `unpythonic` recognizes only the `do` constructs + as a sequencing abstraction. """ n = len(bodys) if not n: @@ -80,7 +99,15 @@ def lazy_begin0(*bodys): body() return out +# TODO: check use of maybe_force_args and force1 in all function composition utilities +# TODO: finish the Values upgrade (grep: "multiple return values", "isinstance tuple") +# TODO: test the new lazify support in piping constructs +# TODO: test multiple-return-values support in all function composition utilities +# TODO: expand tests of `continuations` to cases with named return values +# TODO: update code examples + # sequence one-input, one-output functions +@passthrough_lazy_args def pipe1(value0, *bodys): """Perform a sequence of operations on an initial value. @@ -140,11 +167,14 @@ def pipe1(value0, *bodys): # return x x = value0 for update in bodys: - x = update(x) + update = force1(update) + x = maybe_force_args(update, x) return x +# Singleton value for exiting the pipe abstraction. exitpipe = sym("exitpipe") +@passthrough_lazy_args class piped1: """Shell-like piping syntax. @@ -172,13 +202,15 @@ def __or__(self, f): assert y | inc | exitpipe == 85 assert y | exitpipe == 84 # y is not modified """ + f = force1(f) if f is exitpipe: return self._x cls = self.__class__ - return cls(f(self._x)) # functional update + return cls(maybe_force_args(f, self._x)) # functional update def __repr__(self): # pragma: no cover return f"" +@passthrough_lazy_args class lazy_piped1: """Like piped, but apply the functions later. @@ -201,7 +233,7 @@ def __init__(self, x, *, _funcs=None): The ``_funcs`` parameter is for internal use. """ self._x = x - self._funcs = _funcs or () + self._funcs = force(_funcs or ()) def __or__(self, f): """Pipe the value into f; but just plan to do so, don't perform it yet. @@ -230,45 +262,55 @@ def nextfibo(state): p | exitpipe print(fibos) """ + f = force1(f) if f is exitpipe: # compute now v = self._x for g in self._funcs: - v = g(v) + v = force1(v) + v = maybe_force_args(g, v) return v # just pass on the reference to the original x. cls = self.__class__ - return cls(x=self._x, _funcs=self._funcs + (f,)) + return cls(x=self._x, _funcs=self._funcs + (force1(f),)) def __repr__(self): # pragma: no cover return f"" +@passthrough_lazy_args def pipe(values0, *bodys): """Like pipe1, but with arbitrary number of inputs/outputs at each step. - The only restriction is that each function must take as many positional - arguments as the previous one returns. + The only restriction is that the call and return signatures must match: + each function must take those positional/named arguments the previous one + returns. - At each step, if the output from a function is a tuple, - it is unpacked to the argument list of the next function. Otherwise, - we assume the output is intended to be fed to the next function as-is. + At each step, if the output from a function is a `Values`, it is unpacked + to the args and kwargs of the next function. Otherwise, we feed the output + to the next function as a single positional argument. + + At the beginning of the pipe, `values0` is treated the same way; so to + feed multiple args/kwargs to the first function, use a `Values`. + + If the final return value is a `Values`, and contains only one positional + return value, we unwrap it. Otherwise the `Values` object is returned as-is. If you only need a one-in-one-out chain, ``pipe1`` is faster. Examples:: - a, b = pipe((2, 3), - lambda x, y: (x + 1, 2 * y), - lambda x, y: (x * 2, y + 1)) + a, b = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y), + lambda x, y: Values(x * 2, y + 1)) assert (a, b) == (6, 7) - a, b, c = pipe((2, 3), - lambda x, y: (x + 1, 2 * y, "foo"), - lambda x, y, s: (x * 2, y + 1, f"got {s}")) + a, b, c = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, "foo"), + lambda x, y, s: Values(x * 2, y + 1, f"got {s}")) assert (a, b, c) == (6, 7, "got foo") - a, b = pipe((2, 3), - lambda x, y: (x + 1, 2 * y, "foo"), - lambda x, y, s: (x * 2, y + 1, f"got {s}"), - lambda x, y, s: (x + y, s)) + a, b = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, "foo"), + lambda x, y, s: Values(x * 2, y + 1, f"got {s}"), + lambda x, y, s: Values(x + y, s)) assert (a, b) == (13, "got foo") """ xs = values0 @@ -276,105 +318,131 @@ def pipe(values0, *bodys): for k, update in enumerate(bodys): islast = (k == n - 1) bindings = {} + update = force1(update) if iscurried(update) and not islast: # co-operate with curry: provide a top-level curry context # to allow passthrough from a pipelined function to the next # (except the last one, since it exits the curry context). bindings = {"curry_context": dyn.curry_context + [update]} with dyn.let(**bindings): - if isinstance(xs, tuple): - xs = update(*xs) + xs = force1(xs) + if isinstance(xs, Values): + xs = maybe_force_args(update, *xs.rets, **xs.kwrets) else: - xs = update(xs) - if isinstance(xs, tuple): - return xs if len(xs) > 1 else xs[0] + xs = maybe_force_args(update, xs) + xs = force1(xs) + if isinstance(xs, Values): + return xs if xs.kwrets or len(xs.rets) > 1 else xs[0] return xs +@passthrough_lazy_args def pipec(values0, *bodys): """Like pipe, but curry each function before piping. Useful with the passthrough in ``curry``. Each function only needs to declare as many of the (leftmost) arguments as it needs to access or modify:: - a, b = pipec((1, 2), - lambda x: x + 1, # extra args passed through on the right - lambda x, y: (x * 2, y + 1)) + a, b = pipec(Values(1, 2), + # extra values passed through by curry, positionals on the right + lambda x: x + 1, + lambda x, y: Values(x * 2, y + 1)) assert (a, b) == (4, 3) """ return pipe(values0, *map(curry, bodys)) +@passthrough_lazy_args class piped: """Like piped1, but for any number of inputs/outputs at each step.""" - def __init__(self, *xs): - """Set up a pipe and load the initial values xs into it.""" - self._xs = xs + def __init__(self, *xs, **kws): + """Set up a pipe and load the initial values xs and kws into it. + + The inputs are automatically packed into a `Values`. + """ + self._xs = Values(*xs, **kws) def __or__(self, f): """Pipe the values through the function f. Example:: - f = lambda x, y: (2*x, y+1) - g = lambda x, y: (x+1, 2*y) - x = piped(2, 3) | f | g | exitpipe # --> (5, 8) + f = lambda x, y: Values(2*x, y+1) + g = lambda x, y: Values(x+1, 2*y) + x = piped(2, 3) | f | g | exitpipe # --> Values(5, 8) + + If the final return value is a `Values`, and contains only one positional + return value, we unwrap it. Otherwise the `Values` object is returned as-is. """ + f = force1(f) xs = self._xs + assert isinstance(xs, Values) # __init__ ensures this if f is exitpipe: - return xs if len(xs) > 1 else xs[0] + return xs if xs.kwrets or len(xs.rets) > 1 else xs[0] cls = self.__class__ - assert isinstance(xs, tuple) # __init__ ensures this - newxs = f(*xs) - if isinstance(newxs, tuple): - return cls(*newxs) + newxs = maybe_force_args(f, *xs.rets, **xs.kwrets) + newxs = force1(newxs) + if isinstance(newxs, Values): + return cls(*newxs.rets, **newxs.kwrets) return cls(newxs) def __repr__(self): # pragma: no cover return f"" +@passthrough_lazy_args class lazy_piped: """Like lazy_piped1, but for any number of inputs/outputs at each step. Examples:: p1 = lazy_piped(2, 3) - p2 = p1 | (lambda x, y: (x + 1, 2 * y, "foo")) - p3 = p2 | (lambda x, y, s: (x * 2, y + 1, f"got {s}")) - p4 = p3 | (lambda x, y, s: (x + y, s)) + p2 = p1 | (lambda x, y: Values(x + 1, 2 * y, "foo")) + p3 = p2 | (lambda x, y, s: Values(x * 2, y + 1, f"got {s}")) + p4 = p3 | (lambda x, y, s: Values(x + y, s)) # nothing done yet! - assert (p4 | exitpipe) == (13, "got foo") + assert (p4 | exitpipe) == Values(13, "got foo") # lazy pipe as an unfold fibos = [] def nextfibo(a, b): # now two arguments fibos.append(a) - return (b, a + b) # two return values, still expressed as a tuple + return Values(a=b, b=(a + b)) # can return by name too p = lazy_piped(1, 1) for _ in range(10): p = p | nextfibo - p | exitpipe - print(fibos) + assert p | exitpipe == Values(a=89, b=144) # final state + assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] """ - def __init__(self, *xs, _funcs=None): - """Set up a lazy pipe and load the initial values xs into it. + def __init__(self, *xs, _funcs=None, **kws): + """Set up a lazy pipe and load the initial values xs and kws into it. + + The inputs are automatically packed into a `Values`. The ``_funcs`` parameter is for internal use. """ - self._xs = xs - self._funcs = _funcs or () + self._xs = Values(*xs, **kws) + self._funcs = force(_funcs or ()) def __or__(self, f): - """Pipe the values into f; but just plan to do so, don't perform it yet.""" + """Pipe the values into f; but just plan to do so, don't perform it yet. + + When f is `exitpipe`, perform the planned computation. + + If the final return value is a `Values`, and contains only one positional + return value, we unwrap it. Otherwise the `Values` object is returned as-is. + """ + f = force1(f) if f is exitpipe: # compute now vs = self._xs for g in self._funcs: - if isinstance(vs, tuple): - vs = g(*vs) + vs = force1(vs) + if isinstance(vs, Values): + vs = g(*vs.rets, **vs.kwrets) else: vs = g(vs) - if isinstance(vs, tuple): - return vs if len(vs) > 1 else vs[0] + vs = force1(vs) + if isinstance(vs, Values): + return vs if vs.kwrets or len(vs.rets) > 1 else vs[0] else: return vs # just pass on the references to the original xs. cls = self.__class__ - return cls(*self._xs, _funcs=self._funcs + (f,)) + return cls(*self._xs.rets, _funcs=self._funcs + (force1(f),), **self._xs.kwrets) def __repr__(self): # pragma: no cover return f"" diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index e786b21c..7f7a8a04 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -716,9 +716,12 @@ def transform_starred(tree, dstarred=False): # namelambda() is used by let[] and do[] # Lazy() is a strict function, takes a lambda, constructs a Lazy object # _autoref_resolve doesn't need any special handling + # Values() doesn't need any special handling elif (isdo(tree) or is_decorator(tree.func, "namelambda") or - any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, _expanded_lazy_name) or - isx(tree.func, "_autoref_resolve")): + any(isx(tree.func, s) for s in _ctorcalls_all) or + isx(tree.func, _expanded_lazy_name) or + isx(tree.func, "_autoref_resolve") or + isx(tree.func, "Values")): # here we know the operator (.func) to be one of specific names; # don't transform it to avoid confusing lazyrec[] (important if this # is an inner call in the arglist of an outer, lazy call, since it diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 6836a655..7d22a8a0 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -27,18 +27,19 @@ from .astcompat import getconstant, NameConstant from .ifexprs import aif, it +from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView from .util import (isx, isec, detect_callec, detect_lambda, has_tco, sort_lambda_decorators, suggest_decorator_index, UnpythonicASTMarker, ExpandedContinuationsMarker) -from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView +from ..collections import Values from ..dynassign import dyn -from ..it import uniqify from ..fun import identity +from ..it import uniqify +from ..lazyutil import force1, passthrough_lazy_args from ..tco import trampolined, jump -from ..lazyutil import passthrough_lazy_args # In `continuations`, we use `aif` and `it` as hygienically captured macros. # Note the difference between `aif[..., it, ...]` and `q[a[_our_aif][..., a[_our_it], ...]]`. @@ -295,12 +296,11 @@ def myfunc(a, b, cc): - ``return somevalue`` actually means a tail-call to ``cc`` with the given ``somevalue``. - Multiple values can be returned as a ``tuple``. Tupleness is tested - at run-time. + Multiple values can be returned as a ``Values``. Multiple-valueness + is tested at run time. - Any tuple return value is automatically unpacked to the positional - args of ``cc``. To return multiple things as one without the implicit - unpacking, use a ``list``. + Any ``Values`` return value is automatically unpacked to the args + and kwargs of ``cc``. - An explicit ``return somefunc(arg0, ..., k0=v0, ...)`` actually means a tail-call to ``somefunc``, with its ``cc`` automatically set to our @@ -369,9 +369,13 @@ def myfunc(a, b, cc): Assignment targets: - - To destructure a multiple-values (from a tuple return value), + - To destructure positional multiple-values (from a `Values` return value), use a tuple assignment target (comma-separated names, as usual). + Destructuring named return values from a `call_cc` is currently not supported. + Instead, use a single assignment target to capture the whole `Values` object, + and then destructure it manually. + - The last assignment target may be starred. It is transformed into the vararg (a.k.a. ``*args``) of the continuation function. (It will capture a whole tuple, or any excess items, as usual.) @@ -729,16 +733,16 @@ def _tco(block_body): # _pcc/cc chaining handler, to be exported to client code via q[h[]]. # # We handle multiple-return-values like the rest of unpythonic does: -# returning a tuple means returning multiple values. Unpack them -# to cc's arglist. +# returning a `Values` means returning multiple values. Unpack them +# to cc's args/kwargs. # def chain_conts(cc1, cc2, with_star=False): # cc1=_pcc, cc2=cc """Internal function, used in code generated by the continuations macro.""" if with_star: # to be chainable from a tail call, accept a multiple-values arglist if cc1 is not None: @passthrough_lazy_args - def cc(*value): - return jump(cc1, cc=cc2, *value) + def cc(*rets, **kwrets): + return jump(cc1, cc=cc2, *rets, **kwrets) else: # Beside a small optimization, it is important to preserve # "identity" as "identity", so that the call_cc logic that @@ -748,18 +752,20 @@ def cc(*value): else: # for inert data value returns (this produces the multiple-values arglist) if cc1 is not None: @passthrough_lazy_args - def cc(value): - if isinstance(value, tuple): - return jump(cc1, cc=cc2, *value) + def cc(return_value): + return_value = force1(return_value) + if isinstance(return_value, Values): + return jump(cc1, cc=cc2, *return_value.rets, **return_value.kwrets) else: - return jump(cc1, value, cc=cc2) + return jump(cc1, return_value, cc=cc2) else: @passthrough_lazy_args - def cc(value): - if isinstance(value, tuple): - return jump(cc2, *value) + def cc(return_value): + return_value = force1(return_value) + if isinstance(return_value, Values): + return jump(cc2, *return_value.rets, **return_value.kwrets) else: - return jump(cc2, value) + return jump(cc2, return_value) return cc @@ -815,7 +821,7 @@ def transform_args(tree): tree.args.kw_defaults[j] = q[h[identity]] # implicitly add "parent cc" arg for treating the tail of a computation # as one entity (only actually used in continuation definitions created by - # call_cc; everywhere else, it's None). See callcc_topology.pdf for clarifying pictures. + # call_cc; everywhere else, it's None). See doc/callcc_topology.pdf for clarifying pictures. if "_pcc" not in kwonlynames: non = q[None] non = copy_location(non, tree) @@ -834,15 +840,17 @@ def transform_args(tree): # Already performed by the TCO machinery: # return f(...) --> return jump(f, ...) # - # Additional transformations needed here: + # Additional transformations needed for `continuations`. + # Function calls, after the TCO transform: # return jump(f, ...) --> return jump(f, cc=cc, ...) # customize the transform to add the cc kwarg + # Bare data: # return value --> return jump(cc, value) - # return v1, ..., vn --> return jump(cc, *(v1, ..., vn)) + # return Values(a0, ..., k0=v0, ...) --> return jump(cc, a0, ..., k0=v0, ...) # # Here we only customize the transform_retexpr callback to pass our # current continuation (if no continuation already specified by user). def call_cb(tree): # add the cc kwarg (this plugs into the TCO transformation) - # we're a postproc; our input is "jump(some_target_func, *args)" + # we're a postproc; our input is "jump(some_target_func, *args, **kwargs)" hascc = any(kw.arg == "cc" for kw in tree.keywords) if hascc: # chain our _pcc and the cc=... manually provided by the user @@ -853,6 +861,7 @@ def call_cb(tree): # add the cc kwarg (this plugs into the TCO transformation) # chain our _pcc and the current value of cc tree.keywords = [keyword(arg="cc", value=q[h[chain_conts](n["_pcc"], n["cc"], with_star=True)])] + tree.keywords return tree + # The `data_cb` handles also `Values`; `_transform_retexpr` detects those and treats them as bare data. def data_cb(tree): # transform an inert-data return value into a tail-call to cc. tree = q[h[chain_conts](n["_pcc"], n["cc"])(a[tree])] return tree @@ -1076,8 +1085,8 @@ def examine(self, tree): # into tail calls (to cc), we must insert any missing implicit bare "return" # statements so that _tco_transform_return() sees them. # - # Note that a bare "return" returns `None`, but in the AST `return` looks - # different from `return None`. + # Note that a bare "return" returns `None` at run time, but in the AST, + # `return` looks different from `return None`. class ImplicitBareReturnInjector(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -1239,7 +1248,7 @@ def transform(tree): thelambda = lastitem thelambda.body = transform(thelambda.body) elif type(tree) is Call: - # Apply TCO to tail calls. + # Apply TCO to tail calls ("jumpify" them). # - If already an explicit jump() or loop(), leave it alone. # - If a call to an ec, leave it alone. # - Because an ec call may appear anywhere, a tail-position @@ -1249,7 +1258,14 @@ def transform(tree): # - Hence, transform_return() calls us on the content of # all ec nodes directly. ec(...) is like return; the # argument is the retexpr. - if not (isx(tree.func, _isjump) or isec(tree, known_ecs)): + # - If a Values(...), leave it alone; that just constructs + # a multiple-return-values object so it doesn't need TCO. + # But it acts like bare data. + if isx(tree.func, _isjump) or isec(tree, known_ecs): + pass + elif isx(tree.func, "Values"): + tree = transform_data(tree) + else: tree.args = [tree.func] + tree.args tree.func = q[h[jump]] tree = transform_call(tree) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 5dd8502a..3f083345 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -6,6 +6,7 @@ from ...syntax import macros, continuations, call_cc, multilambda, autoreturn, autocurry, let # noqa: F401, F811 +from ...collections import Values from ...ec import call_ec from ...fploop import looped from ...tco import trampolined, jump @@ -19,7 +20,13 @@ def add1(x): test[add1(2) == 3] def message(cc): - return ("hello", "there") + # The continuations system essentially deals with function composition, + # so we make a distinction between a single `tuple` return value and + # multiple-return-values. + # + # Use Values(...) to return multiple values from a function that you + # intend to `call_cc`. + return Values("hello", "there") def baz(): m, n = call_cc[message()] # The cc arg is passed implicitly. return [m, n] @@ -28,8 +35,8 @@ def baz(): # The cc arg must be declared as the last one that has no default value, # or declared as by-name-only. It's always passed by name. def f(a, b, cc): - return 2 * a, 3 * b - test[f(3, 4) == (6, 12)] + return Values(2 * a, 3 * b) + test[f(3, 4) == Values(6, 12)] x, y = f(3, 4) test[x == 6 and y == 12] @@ -49,7 +56,7 @@ def g(a, b): def h1(a, b): x, y = call_cc[f(a, b)] return None or f(3, 4) # the f from the previous "with continuations" block - test[h1(3, 4) == (6, 12)] + test[h1(3, 4) == Values(6, 12)] def h2(a, b): x, y = call_cc[f(a, b)] @@ -60,7 +67,7 @@ def h2(a, b): def h3(a, b): x, y = call_cc[f(a, b)] return None or False or f(3, 4) - test[h3(3, 4) == (6, 12)] + test[h3(3, 4) == Values(6, 12)] def h4(a, b): x, y = call_cc[f(a, b)] @@ -76,7 +83,7 @@ def h5(a, b): def i1(a, b): x, y = call_cc[f(a, b)] return True and f(3, 4) - test[i1(3, 4) == (6, 12)] + test[i1(3, 4) == Values(6, 12)] def i2(a, b): x, y = call_cc[f(a, b)] @@ -87,7 +94,7 @@ def i2(a, b): def i3(a, b): x, y = call_cc[f(a, b)] return True and 42 and f(3, 4) - test[i3(3, 4) == (6, 12)] + test[i3(3, 4) == Values(6, 12)] def i4(a, b): x, y = call_cc[f(a, b)] @@ -103,7 +110,7 @@ def i5(a, b): def j1(a, b): x, y = call_cc[f(a, b)] return None or True and f(3, 4) - test[j1(3, 4) == (6, 12)] + test[j1(3, 4) == Values(6, 12)] with testset("let in tail position"): with continuations: @@ -111,19 +118,19 @@ def j2(a, b): x, y = call_cc[f(a, b)] return let[[c << a, # noqa: F821 d << b] in f(c, d)] # noqa: F821 - test[j2(3, 4) == (6, 12)] + test[j2(3, 4) == Values(6, 12)] with testset("if-expression in tail position"): with continuations: def j3(a, b): x, y = call_cc[f(a, b)] return f(a, b) if True else None - test[j3(3, 4) == (6, 12)] + test[j3(3, 4) == Values(6, 12)] def j4(a, b): x, y = call_cc[f(a, b)] return None if False else f(a, b) - test[j4(3, 4) == (6, 12)] + test[j4(3, 4) == Values(6, 12)] with testset("integration with a lambda that has TCO"): with continuations: @@ -194,8 +201,6 @@ def setk(*args, cc): # and list() is a regular function, not a continuation-enabled one # (so it would immediately terminate the TCO chain; besides, # it takes only 1 argument and doesn't know what to do with "cc".) - # - list instead of tuple to return it as one value - # (a tuple return value is interpreted as multiple-return-values) return xs def doit(): lst = ['the call returned'] @@ -213,7 +218,7 @@ def doit(): def setk(*args, cc): # noqa: F811, the previous one is no longer used. nonlocal k k = cc # current continuation, i.e. where to go after setk() finishes - return args # tuple means multiple-return-values + return Values(*args) # multiple-return-values def doit(): lst = ['the call returned'] *more, = call_cc[setk('A')] @@ -235,7 +240,7 @@ def doit(): def setk(*args, cc): # noqa: F811, the previous one is no longer used. nonlocal k k = cc - return args # tuple return value (if not literal, tested at run-time) --> multiple-values + return Values(*args) # multiple-return-values x, y = call_cc[setk(*vals)] test[x, y == vals] # end the block to end capture, and start another one to resume programming diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 18e9e8ec..c600fa6b 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -10,9 +10,7 @@ autocurry, continuations, call_cc) -# Doesn't really override the earlier curry import. The first one is a macro, -# and this one is a regular run-time function. -from ...collections import frozendict +from ...collections import frozendict, Values from ...ec import call_ec from ...excutil import raisef from ...fun import (curry, memoize, flip, rotate, apply, @@ -439,7 +437,8 @@ def h1(x): test[islazy(h2)] # args 0 and 2 never *used* by h2, so we need to force() # to get their values to compare the reference answer to. - test[force(h2(1, 2, 3)) == (1, 84, 3)] + # Also, h2 uses tokth, which wraps its multiple-return-values in a Values. + test[force(h2(1, 2, 3)) == Values(1, 84, 3)] fact = withself(lambda self, n, acc=1: self(n - 1, acc * n) if n > 1 else acc) # linear process test[islazy(fact)] diff --git a/unpythonic/syntax/tests/test_tco.py b/unpythonic/syntax/tests/test_tco.py index fd49f3e1..02a1a106 100644 --- a/unpythonic/syntax/tests/test_tco.py +++ b/unpythonic/syntax/tests/test_tco.py @@ -7,6 +7,7 @@ from ...syntax import (macros, tco, autoreturn, autocurry, do, let, letseq, dletrec, # noqa: F401, F811 quicklambda, f, continuations, call_cc) +from ...collections import Values from ...ec import call_ec from ...fploop import looped_over from ...fun import withself, curry @@ -168,7 +169,7 @@ def g(x): def setk(*args, cc): nonlocal k k = cc # current continuation, i.e. where to go after setk() finishes - return args # tuple means multiple-return-values + return Values(*args) # multiple-return-values def doit(): lst = ['the call returned'] *more, = call_cc[setk('A')] diff --git a/unpythonic/tests/test_fold.py b/unpythonic/tests/test_fold.py index af2103ef..7e442960 100644 --- a/unpythonic/tests/test_fold.py +++ b/unpythonic/tests/test_fold.py @@ -93,9 +93,11 @@ def mymap_one2(f, iterable): doubler = mymap_one4(double) test[doubler(ll(1, 2, 3)) == ll(2, 4, 6)] - # curry supports passing through on the right any args over the max arity. - # If an intermediate result is a callable, it is invoked on the remaining - # positional args: + # curry supports passthrough for any args/kwargs that can't be accepted by + # the function's call signature (too many positionals or unknown named args). + # Positionals are passed through on the right. + # If the first positional return value of an intermediate result is a callable, + # it is curried, and invoked on the remaining args/kwargs: test[curry(mymap_one4, double, ll(1, 2, 3)) == ll(2, 4, 6)] # But having any args remaining when the top-level curry context exits @@ -118,7 +120,7 @@ def mymap_one2(f, iterable): # # The iterables are taken by the processing function. acc, being the last # argument, is passed through on the right. The output from the processing - # function - one new item - and acc then become a two-tuple, which gets + # function - one new item - and acc then become two arguments, which get # passed into cons. myadd = lambda x, y: x + y # can't inspect signature of builtin add test[curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9)] diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index ac859678..fbfd76af 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -6,6 +6,7 @@ from collections import Counter import sys +from ..collections import Values from ..dispatch import generic from ..fun import (memoize, partial, curry, apply, identity, const, @@ -20,12 +21,12 @@ def runtests(): with testset("identity function"): - test[identity(1, 2, 3) == (1, 2, 3)] + test[identity(1, 2, 3) == Values(1, 2, 3)] test[identity(42) == 42] test[identity() is None] # no args, default value with testset("constant function"): - test[const(1, 2, 3)(42, "foo") == (1, 2, 3)] + test[const(1, 2, 3)(42, "foo") == Values(1, 2, 3)] test[const(42)("anything") == 42] test[const()("anything") is None] @@ -183,28 +184,36 @@ def t(): a = curry(add) test[curry(a) is a] # curry wrappers should not stack - # Curry passes through extra args on the right, like in Haskell. Each - # call consumes args up to the maximum arity of the function being - # called. If the return value is callable, it is the next function - # to be (implicitly curried and then) called. + # curry supports passthrough for any args/kwargs that can't be accepted by + # the function's call signature (too many positionals or unknown named args). + # Positionals are passed through on the right. + # If the first positional return value of an intermediate result is a callable, + # it is curried, and invoked on the remaining args/kwargs: @curry def f(x): # note f takes only one arg return lambda y: x * y test[f(2, 21) == 42] - # Curry raises by default when the top-level curry context exits with - # args remaining. This is so that providing too many args will still - # raise `TypeError`. + # By default, `curry` raises `TypeError` when the top-level curry context exits + # with args/kwargs remaining. This is a safety feature: providing args/kwargs + # not consumed during the curry chain will raise an error, rather than silently + # produce results that are likely not what was intended. def double(x): return 2 * x - with test_raises[TypeError, "leftover args should not be allowed by default"]: + with test_raises[TypeError, "leftover positional args should not be allowed by default"]: curry(double, 2, "foo") + with test_raises[TypeError, "leftover named args should not be allowed by default"]: + curry(double, 2, nosucharg="foo") - # To disable the error, use this trick to explicitly state you want to do so: - with test["leftover args should be allowed with manually created surrounding context"]: + # The check can be disabled, by stating explicitly that you want to do so: + with test["leftover positional args should be allowed with manually created surrounding context"]: with dyn.let(curry_context=["whatever"]): # any human-readable label is fine. # a `with test` can optionally return a value, which becomes the asserted expr. - return curry(double, 2, "foo") == (4, "foo") + return the[curry(double, 2, "foo")] == Values(4, "foo") + + with test["leftover named args should be allowed with manually created surrounding context"]: + with dyn.let(curry_context=["whatever"]): + return the[curry(double, 2, nosucharg="foo")] == Values(4, nosucharg="foo") # This doesn't occur on PyPy3. if sys.implementation.name == "cpython": # pragma: no cover @@ -300,7 +309,7 @@ def mymul_typed(y: int): with testset("curry in compose chain"): def f1(a, b): - return 2 * a, 3 * b + return Values(2 * a, 3 * b) def f2(a, b): return a + b f1_then_f2_a = composelc(f1, f2) @@ -308,22 +317,22 @@ def f2(a, b): test[f1_then_f2_a(2, 3) == f1_then_f2_b(2, 3) == 13] def f3(a, b): - return a, b + return Values(a, b) def f4(a, b, c): return a + b + c f1_then_f3_then_f4 = composelc(f1, f3, f4) test[f1_then_f3_then_f4(2, 3, 5) == 18] # extra arg passed through on the right with testset("to1st, to2nd, tolast, to (argument shunting)"): - test[to1st(double)(1, 2, 3) == (2, 2, 3)] - test[to2nd(double)(1, 2, 3) == (1, 4, 3)] - test[tolast(double)(1, 2, 3) == (1, 2, 6)] + test[to1st(double)(1, 2, 3) == Values(2, 2, 3)] + test[to2nd(double)(1, 2, 3) == Values(1, 4, 3)] + test[tolast(double)(1, 2, 3) == Values(1, 2, 6)] processor = to((0, double), (-1, inc), (1, composer(double, double)), (0, inc)) - test[processor(1, 2, 3) == (3, 8, 4)] + test[processor(1, 2, 3) == Values(3, 8, 4)] with testset("tokth error cases"): test_raises[TypeError, tokth(3, double)()] # expect at least one argument @@ -337,11 +346,11 @@ def f(a, b): test[(flip(f))(1, b=2) == (1, 2)] # b -> kwargs with testset("rotate arglist"): - test[(rotate(-1)(identity))(1, 2, 3) == (3, 1, 2)] - test[(rotate(1)(identity))(1, 2, 3) == (2, 3, 1)] + test[(rotate(-1)(identity))(1, 2, 3) == Values(3, 1, 2)] + test[(rotate(1)(identity))(1, 2, 3) == Values(2, 3, 1)] # inner to outer: (a, b, c) -> (b, c, a) -> (a, c, b) - test[flip(rotate(-1)(identity))(1, 2, 3) == (1, 3, 2)] + test[flip(rotate(-1)(identity))(1, 2, 3) == Values(1, 3, 2)] with testset("rotate error cases"): test_raises[TypeError, (rotate(1)(identity))()] # expect at least one argument diff --git a/unpythonic/tests/test_it.py b/unpythonic/tests/test_it.py index 2145aaa8..1617ba46 100644 --- a/unpythonic/tests/test_it.py +++ b/unpythonic/tests/test_it.py @@ -30,6 +30,7 @@ subset, powerset, allsame) +from ..collections import Values from ..fun import composel, identity, curry from ..gmemo import imemoize, gmemoize from ..mathseq import s @@ -73,11 +74,11 @@ def noneadd(a, b): # but actually requires 2. Solution: use partial instead of curry. lzip2 = partial(map, identity) rzip2 = lambda *iterables: map(identity, *(rev(s) for s in iterables)) - test[tuple(lzip2((1, 2, 3), (4, 5, 6), (7, 8))) == ((1, 4, 7), (2, 5, 8))] - test[tuple(rzip2((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7))] + test[tuple(lzip2((1, 2, 3), (4, 5, 6), (7, 8))) == (Values(1, 4, 7), Values(2, 5, 8))] + test[tuple(rzip2((1, 2, 3), (4, 5, 6), (7, 8))) == (Values(3, 6, 8), Values(2, 5, 7))] rzip3 = partial(rmap, identity) - test[tuple(rzip3((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7))] + test[tuple(rzip3((1, 2, 3), (4, 5, 6), (7, 8))) == (Values(3, 6, 8), Values(2, 5, 7))] with testset("first, second, nth, last"): test[first(range(5)) == 0] diff --git a/unpythonic/tests/test_seq.py b/unpythonic/tests/test_seq.py index 5218a677..dbc05c5c 100644 --- a/unpythonic/tests/test_seq.py +++ b/unpythonic/tests/test_seq.py @@ -3,6 +3,7 @@ from ..syntax import macros, test, test_raises, fail # noqa: F401 from ..test.fixtures import session, testset +from ..collections import Values from ..seq import (begin, begin0, lazy_begin, lazy_begin0, pipe1, pipe, pipec, piped1, piped, exitpipe, @@ -43,32 +44,45 @@ def runtests(): test[pipe(42, inc, double) == 86] # 2-in-2-out - a, b = pipe((2, 3), - lambda x, y: (x + 1, 2 * y), - lambda x, y: (x * 2, y + 1)) + a, b = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y), + lambda x, y: Values(x * 2, y + 1)) test[(a, b) == (6, 7)] + # 2-in-2-out, pass intermediate result by name + a, b = pipe(Values(2, 3), + lambda x, y: Values(x=(x + 1), y=(2 * y)), + lambda x, y: Values(x * 2, y + 1)) + test[(a, b) == (6, 7)] + + # 2-in-2-out, also return final result by name + v = pipe(Values(2, 3), + lambda x, y: Values(x=(x + 1), y=(2 * y)), + lambda x, y: Values(a=(x * 2), b=(y + 1))) + test[v == Values(a=6, b=7)] + test[v["a"] == 6 and v["b"] == 7] # can access them via subscripting too + # 2-in-eventually-3-out - a, b, c = pipe((2, 3), - lambda x, y: (x + 1, 2 * y, "foo"), - lambda x, y, z: (x * 2, y + 1, f"got {z}")) + a, b, c = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, "foo"), + lambda x, y, z: Values(x * 2, y + 1, f"got {z}")) test[(a, b, c) == (6, 7, "got foo")] # 2-in-3-in-between-2-out - a, b = pipe((2, 3), - lambda x, y: (x + 1, 2 * y, "foo"), - lambda x, y, s: (x * 2, y + 1, f"got {s}"), - lambda x, y, s: (x + y, s)) + a, b = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, "foo"), + lambda x, y, s: Values(x * 2, y + 1, f"got {s}"), + lambda x, y, s: Values(x + y, s)) test[(a, b) == (13, "got foo")] # pipec: curry the functions before running the pipeline - a, b = pipec((1, 2), - lambda x: x + 1, # extra args passed through on the right - lambda x, y: (x * 2, y + 1)) + a, b = pipec(Values(1, 2), + lambda x: x + 1, # extra values passed through by curry (positionals on the right) + lambda x, y: Values(x * 2, y + 1)) test[(a, b) == (4, 3)] with test_raises[TypeError, "should error when the curry context exits with args remaining"]: - a, b = pipec((1, 2), + a, b = pipec(Values(1, 2), lambda x: x + 1, lambda x: x * 2) @@ -80,10 +94,10 @@ def runtests(): test[y | exitpipe == 84] # y is never modified by the pipe system # multi-arg version - f = lambda x, y: (2 * x, y + 1) - g = lambda x, y: (x + 1, 2 * y) + f = lambda x, y: Values(2 * x, y + 1) + g = lambda x, y: Values(x + 1, 2 * y) x = piped(2, 3) | f | g | exitpipe # --> (5, 8) - test[x == (5, 8)] + test[x == Values(5, 8)] # abuse multi-arg version for single-arg case test[piped(42) | double | inc | exitpipe == 85] @@ -91,9 +105,9 @@ def runtests(): with testset("lazy pipe (plan computations)"): # lazy pipe: compute later lst = [1] - def append_succ(l): - l.append(l[-1] + 1) - return l # important, handed to the next function in the pipe + def append_succ(lis): + lis.append(lis[-1] + 1) + return lis # important, handed to the next function in the pipe p = lazy_piped1(lst) | append_succ | append_succ # plan a computation test[lst == [1]] # nothing done yet p | exitpipe # run the computation @@ -113,24 +127,24 @@ def nextfibo(state): # multi-arg lazy pipe p1 = lazy_piped(2, 3) - p2 = p1 | (lambda x, y: (x + 1, 2 * y, "foo")) - p3 = p2 | (lambda x, y, s: (x * 2, y + 1, f"got {s}")) - p4 = p3 | (lambda x, y, s: (x + y, s)) + p2 = p1 | (lambda x, y: Values(x + 1, 2 * y, "foo")) + p3 = p2 | (lambda x, y, s: Values(x * 2, y + 1, f"got {s}")) + p4 = p3 | (lambda x, y, s: Values(x + y, s)) # nothing done yet, and all computations purely functional: - test[(p1 | exitpipe) == (2, 3)] - test[(p2 | exitpipe) == (3, 6, "foo")] # runs the chain up to p2 - test[(p3 | exitpipe) == (6, 7, "got foo")] # runs the chain up to p3 - test[(p4 | exitpipe) == (13, "got foo")] + test[(p1 | exitpipe) == Values(2, 3)] + test[(p2 | exitpipe) == Values(3, 6, "foo")] # runs the chain up to p2 + test[(p3 | exitpipe) == Values(6, 7, "got foo")] # runs the chain up to p3 + test[(p4 | exitpipe) == Values(13, "got foo")] # multi-arg lazy pipe as an unfold fibos = [] def nextfibo(a, b): # now two arguments fibos.append(a) - return (b, a + b) # two return values, still expressed as a tuple + return Values(a=b, b=(a + b)) # can return by name too p = lazy_piped(1, 1) for _ in range(10): p = p | nextfibo - p | exitpipe + test[p | exitpipe == Values(a=89, b=144)] # final state test[fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]] # abuse multi-arg version for single-arg case From fed6f1cde1ccd4a03525c610928282438b54e175 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:02:43 +0300 Subject: [PATCH 337/832] improve docstring --- unpythonic/lazyutil.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/unpythonic/lazyutil.py b/unpythonic/lazyutil.py index 75b654dd..2f026a43 100644 --- a/unpythonic/lazyutil.py +++ b/unpythonic/lazyutil.py @@ -98,7 +98,11 @@ def islazy(f): return hasattr(f, "_passthrough_lazy_args") or (hasattr(f, "__name__") and f.__name__ == "_let") def maybe_force_args(f, *thunks, **kwthunks): - """Internal. Helps calling strict functions from inside a ``with lazify`` block.""" + """Internal. Helps calling strict functions from inside a ``with lazify`` block. + + If `not islazy(f)`, forces the given args and kwargs, and then calls `f` with them. + If `islazy(f)`, calls `f` without forcing the args/kwargs. + """ if f is jump: # special case to avoid drastic performance hit in strict code target, *argthunks = thunks return jump(force1(target), *argthunks, **kwthunks) From b976f709f3404d0f0025a192b3e86c14c03f6d59 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:10:40 +0300 Subject: [PATCH 338/832] update doc --- doc/macros.md | 2 +- unpythonic/syntax/tailtools.py | 4 +--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 390c3acd..808a2ee8 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1260,7 +1260,7 @@ call_cc[f(...) if p else g(...)] **Assignment targets**: - - To destructure a multiple-values (from a tuple return value), use a tuple assignment target (comma-separated names, as usual). + - To destructure positional multiple-values (from a `Values` return value), use a tuple assignment target (comma-separated names, as usual). Destructuring *named* return values from a `call_cc` is currently not supported. - The last assignment target may be starred. It is transformed into the vararg (a.k.a. ``*args``, star-args) of the continuation function. (It will capture a whole tuple, or any excess items, as usual.) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 7d22a8a0..206cc421 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -372,9 +372,7 @@ def myfunc(a, b, cc): - To destructure positional multiple-values (from a `Values` return value), use a tuple assignment target (comma-separated names, as usual). - Destructuring named return values from a `call_cc` is currently not supported. - Instead, use a single assignment target to capture the whole `Values` object, - and then destructure it manually. + Destructuring *named* return values from a `call_cc` is currently not supported. - The last assignment target may be starred. It is transformed into the vararg (a.k.a. ``*args``) of the continuation function. From f649cd9c4a1935a4bf28ea3df9dc31f361c263f2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:18:43 +0300 Subject: [PATCH 339/832] update docstring --- unpythonic/it.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/unpythonic/it.py b/unpythonic/it.py index 6d707815..7fc16c4b 100644 --- a/unpythonic/it.py +++ b/unpythonic/it.py @@ -499,11 +499,16 @@ def flatten(iterable, pred=None): (or list), and return ``True`` if that tuple/list should be flattened. When ``pred`` returns False, that tuple/list is passed through as-is. - E.g. to flatten only those items that contain only tuples:: + E.g. to flatten only those items that contain only lists or tuples:: is_nested = lambda e: all(isinstance(x, (list, tuple)) for x in e) data = (((1, 2), (3, 4)), (5, 6)) assert tuple(flatten(data, is_nested)) == ((1, 2), (3, 4), (5, 6)) + + Even with a predicate, flattening is still performed recursively:: + + data = (((1, 2), ((3, 4), ((5, 6), (7, 8))), (9, 10))) + assert tuple(flatten(data, is_nested)) == ((1, 2), (3, 4), (5, 6), (7, 8), (9, 10)) """ return _flatten(iterable, pred, recursive=True) From 14ce1ca9718ad61dfaaf5e6e46d2b3a9c6b7b5dd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:20:33 +0300 Subject: [PATCH 340/832] update docstring --- unpythonic/it.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpythonic/it.py b/unpythonic/it.py index 7fc16c4b..80e4da94 100644 --- a/unpythonic/it.py +++ b/unpythonic/it.py @@ -505,7 +505,8 @@ def flatten(iterable, pred=None): data = (((1, 2), (3, 4)), (5, 6)) assert tuple(flatten(data, is_nested)) == ((1, 2), (3, 4), (5, 6)) - Even with a predicate, flattening is still performed recursively:: + Even with a predicate, flattening is still performed recursively + in any item that matches the predicate:: data = (((1, 2), ((3, 4), ((5, 6), (7, 8))), (9, 10))) assert tuple(flatten(data, is_nested)) == ((1, 2), (3, 4), (5, 6), (7, 8), (9, 10)) From 3b3807060932f059cfee4cf0c112caf621d77973 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:23:54 +0300 Subject: [PATCH 341/832] improve comments --- unpythonic/syntax/tests/test_conts.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 3f083345..e39907b8 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -375,6 +375,9 @@ def amb(lst, cc): return fail() first, *rest = tuple(lst) if rest: + # Note even the `lambda` below has an implicit `cc` parameter; + # hence we must name the current `cc` to something else to be + # able to use the value inside the `lambda`. ourcc = cc stack.append(lambda: amb(rest, cc=ourcc)) return first @@ -429,8 +432,9 @@ def pt_gen(maxn): count = 0 def pt(maxn): # This generates 1540 combinations, with several nested tail-calls each, - # so we really need TCO here. (Without TCO, nothing would return until - # the whole computation is done; it would blow the call stack very quickly.) + # so we really need TCO here. Without TCO, nothing would return until + # the whole computation is done; it would blow the call stack very quickly. + # With TCO, it's just a case of "lambda, the ultimate goto". z = call_cc[amb(range(1, maxn + 1))] y = call_cc[amb(range(1, z + 1))] x = call_cc[amb(range(1, y + 1))] From dcb10b79b9a16f5b730bea48991f6f7bc79a4aeb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 03:24:13 +0300 Subject: [PATCH 342/832] update TODO comment --- unpythonic/seq.py | 1 - 1 file changed, 1 deletion(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index e9ee21e0..01c72597 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -100,7 +100,6 @@ def lazy_begin0(*bodys): return out # TODO: check use of maybe_force_args and force1 in all function composition utilities -# TODO: finish the Values upgrade (grep: "multiple return values", "isinstance tuple") # TODO: test the new lazify support in piping constructs # TODO: test multiple-return-values support in all function composition utilities # TODO: expand tests of `continuations` to cases with named return values From d6dd85630ed00957942edc2b4e76e255b84fd6b9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 13:52:03 +0300 Subject: [PATCH 343/832] variable naming: avoid `l` --- doc/features.md | 6 +++--- unpythonic/seq.py | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/features.md b/doc/features.md index abfcbe04..b8dc31b7 100644 --- a/doc/features.md +++ b/doc/features.md @@ -952,9 +952,9 @@ Set up a pipe by calling ``piped`` for the initial value. Pipe into the sentinel from unpythonic import lazy_piped1, exitpipe lst = [1] -def append_succ(l): - l.append(l[-1] + 1) - return l # this return value is handed to the next function in the pipe +def append_succ(lis): + lis.append(lis[-1] + 1) + return lis # this return value is handed to the next function in the pipe p = lazy_piped1(lst) | append_succ | append_succ # plan a computation assert lst == [1] # nothing done yet p | exitpipe # run the computation diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 01c72597..acdfc1bd 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -241,9 +241,9 @@ def __or__(self, f): Examples:: lst = [1] - def append_succ(l): - l.append(l[-1] + 1) - return l # important, handed to the next function in the pipe + def append_succ(lis): + lis.append(lis[-1] + 1) + return lis # important, handed to the next function in the pipe p = lazy_piped1(lst) | append_succ | append_succ # plan a computation assert lst == [1] # nothing done yet p | exitpipe # run the computation From 75872adfab21381a5f68e7cb1ed670a5e4b9f3dd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 13:52:15 +0300 Subject: [PATCH 344/832] improve docstring --- unpythonic/seq.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index acdfc1bd..a85cb680 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -188,10 +188,10 @@ def __or__(self, f): Return a ``piped`` object, for chainability. As the only exception, if ``f`` is the sentinel ``exitpipe``, - return the current value (useful for exiting the pipe). + return the current value (thus exiting the pipe). - A new ``piped`` object is created at each step of piping; the "update" - is purely functional, nothing is overwritten. + A new ``piped`` object is created at each step of piping; + the "update" is purely functional, nothing is overwritten. Examples:: From a2caa1205ee46d08a341c53cf21adb16d98115e9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 14:04:13 +0300 Subject: [PATCH 345/832] update docstrings --- unpythonic/seq.py | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index a85cb680..e5a5fb9f 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -280,7 +280,8 @@ def pipe(values0, *bodys): The only restriction is that the call and return signatures must match: each function must take those positional/named arguments the previous one - returns. + returns. Use a `Values` object to denote multiple-return-values, and/or + named return values. At each step, if the output from a function is a `Values`, it is unpacked to the args and kwargs of the next function. Otherwise, we feed the output @@ -299,6 +300,9 @@ def pipe(values0, *bodys): a, b = pipe(Values(2, 3), lambda x, y: Values(x + 1, 2 * y), lambda x, y: Values(x * 2, y + 1)) + # If a `Values` object has only positional values, + # it can be unpacked like a tuple. Hence we don't + # see a `Values` wrapper here. assert (a, b) == (6, 7) a, b, c = pipe(Values(2, 3), @@ -306,6 +310,12 @@ def pipe(values0, *bodys): lambda x, y, s: Values(x * 2, y + 1, f"got {s}")) assert (a, b, c) == (6, 7, "got foo") + # Can bind arguments of the next step by name, too + a, b, c = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, s="foo"), + lambda x, y, s: Values(x * 2, y + 1, f"got {s}")) + assert (a, b, c) == (6, 7, "got foo") + a, b = pipe(Values(2, 3), lambda x, y: Values(x + 1, 2 * y, "foo"), lambda x, y, s: Values(x * 2, y + 1, f"got {s}"), @@ -351,7 +361,13 @@ def pipec(values0, *bodys): @passthrough_lazy_args class piped: - """Like piped1, but for any number of inputs/outputs at each step.""" + """Like piped1, but for any number of inputs/outputs at each step. + + The only restriction is that the call and return signatures must match: + each function must take those positional/named arguments the previous one + returns. Use a `Values` object to denote multiple-return-values, and/or + named return values. + """ def __init__(self, *xs, **kws): """Set up a pipe and load the initial values xs and kws into it. @@ -361,6 +377,10 @@ def __init__(self, *xs, **kws): def __or__(self, f): """Pipe the values through the function f. + If the data currently in the pipe is a `Values`, it is unpacked + to the args and kwargs of `f`. Otherwise, we feed the data to `f` + as a single positional argument. + Example:: f = lambda x, y: Values(2*x, y+1) @@ -388,6 +408,11 @@ def __repr__(self): # pragma: no cover class lazy_piped: """Like lazy_piped1, but for any number of inputs/outputs at each step. + The only restriction is that the call and return signatures must match: + each function must take those positional/named arguments the previous one + returns. Use a `Values` object to denote multiple-return-values, and/or + named return values. + Examples:: p1 = lazy_piped(2, 3) @@ -422,6 +447,10 @@ def __or__(self, f): When f is `exitpipe`, perform the planned computation. + When the computation is performed, when this `f` is reached, if the data + currently in the pipe is a `Values`, it is unpacked to the args and kwargs + of `f`. Otherwise, we feed the data to `f` as a single positional argument. + If the final return value is a `Values`, and contains only one positional return value, we unwrap it. Otherwise the `Values` object is returned as-is. """ From 3fd74daec1180badbfbec0b10c7b3a5e11e9e1e2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 14:23:37 +0300 Subject: [PATCH 346/832] Oops, return values are never implicitly lazy Forgetting parts of my own design here. --- unpythonic/fun.py | 5 +---- unpythonic/seq.py | 6 ------ 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 121d1499..1ff1ddc3 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -451,7 +451,6 @@ def curried(*args, **kwargs): now_kwargs = {k: v for k, v in kwargs.items() if k not in later_kwargs} now_result = maybe_force_args(f, *now_args, **now_kwargs) - now_result = force1(now_result) # just in case it's a `Lazy` # Inspect the return value(s). # - Inject the appropriate items to `later_args` and `later_kwargs`. @@ -730,7 +729,7 @@ def disjoined(*args, **kwargs): def _make_compose1(direction): # "left", "right" def compose1_two(f, g): # return lambda x: f(g(x)) - return lambda x: maybe_force_args(f, force1(maybe_force_args(g, x))) + return lambda x: maybe_force_args(f, maybe_force_args(g, x)) if direction == "right": compose1_two = flip(compose1_two) def compose1(fs): @@ -820,8 +819,6 @@ def composed(*args, **kwargs): bindings = {"curry_context": dyn.curry_context + [composed]} with dyn.let(**bindings): a = maybe_force_args(g, *args, **kwargs) - a = force1(a) # just in case it's a `Lazy` - # We could duck-test for an iterable, but this is more predictable. if isinstance(a, Values): return maybe_force_args(f, *a.rets, **a.kwrets) return maybe_force_args(f, a) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index e5a5fb9f..4c20d938 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -265,7 +265,6 @@ def nextfibo(state): if f is exitpipe: # compute now v = self._x for g in self._funcs: - v = force1(v) v = maybe_force_args(g, v) return v # just pass on the reference to the original x. @@ -334,12 +333,10 @@ def pipe(values0, *bodys): # (except the last one, since it exits the curry context). bindings = {"curry_context": dyn.curry_context + [update]} with dyn.let(**bindings): - xs = force1(xs) if isinstance(xs, Values): xs = maybe_force_args(update, *xs.rets, **xs.kwrets) else: xs = maybe_force_args(update, xs) - xs = force1(xs) if isinstance(xs, Values): return xs if xs.kwrets or len(xs.rets) > 1 else xs[0] return xs @@ -397,7 +394,6 @@ def __or__(self, f): return xs if xs.kwrets or len(xs.rets) > 1 else xs[0] cls = self.__class__ newxs = maybe_force_args(f, *xs.rets, **xs.kwrets) - newxs = force1(newxs) if isinstance(newxs, Values): return cls(*newxs.rets, **newxs.kwrets) return cls(newxs) @@ -458,12 +454,10 @@ def __or__(self, f): if f is exitpipe: # compute now vs = self._xs for g in self._funcs: - vs = force1(vs) if isinstance(vs, Values): vs = g(*vs.rets, **vs.kwrets) else: vs = g(vs) - vs = force1(vs) if isinstance(vs, Values): return vs if vs.kwrets or len(vs.rets) > 1 else vs[0] else: From 4d0bfd80da06371f54519bd39e27c1400292fa54 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 14:24:05 +0300 Subject: [PATCH 347/832] curry: use maybe_force_args to call leftmost positional return value Not sure if it's really needed since it's curried anyway, but this makes the code look more uniform (and hence more likely correct). --- unpythonic/fun.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 1ff1ddc3..2b17cd23 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -492,7 +492,7 @@ def curried(*args, **kwargs): if callable(leftmost): if not iscurried(leftmost): leftmost = curry(leftmost) - return leftmost(*later_args, **later_kwargs) + return maybe_force_args(leftmost, *later_args, **later_kwargs) # The first positional return value is not a callable. Pass the return value(s) through # to the curried procedure waiting in outerctx (e.g. in a curried compose chain). From 015034448e5443f8d5686dd4c9d542f3d744731b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 14:24:57 +0300 Subject: [PATCH 348/832] improve docstrings --- unpythonic/fun.py | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 2b17cd23..4c7e4d9b 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -726,18 +726,42 @@ def disjoined(*args, **kwargs): disjoined = passthrough_lazy_args(disjoined) return disjoined -def _make_compose1(direction): # "left", "right" +def _make_compose1(direction): + """Make a function that composes functions from an iterable. + + Return value is a function `compose1(fs)` -> `composed(x)`. + + `direction`: str, one of "left", "right". Which way to compose. + + For example, let `fs = (f1, f2, f3)`. + + If `direction == "left"`, `composed` computes f3(f2(f1(x))); + the functions apply leftmost first. + + If `direction == "right"`, `composed` computes f1(f2(f3(x))); + the functions apply rightmost first. + + Standard mathematical function composition notation f1 ∘ f2 ∘ f3 takes rightmost first, + but we refuse the temptation to guess. We provide only explicit `l` and `r` variants + of all the `compose1` utilities. + """ def compose1_two(f, g): # return lambda x: f(g(x)) return lambda x: maybe_force_args(f, maybe_force_args(g, x)) if direction == "right": compose1_two = flip(compose1_two) def compose1(fs): - # direction == "left" (leftmost is innermost): + """Compose one-argument functions from iterable `fs`. + + **CAUTION**: This is a closure. Which way to compose (left or right) + was chosen when this closure instance was created. Please use the + public API functions whose names explicitly state the direction. + """ + # If `direction == "left"` leftmost is innermost: # input: a b c # elt = b -> f, acc = a(x) -> g --> b(a(x)) # elt = c -> f, acc = b(a(x)) -> g --> c(b(a(x))) - # direction == "right" (rightmost is innermost): + # If `direction == "right"`, rightmost is innermost: # input: a b c # elt = b -> g, acc = a(x) -> f --> a(b(x)) # elt = c -> g, acc = a(b(x)) -> f --> a(b(c(x))) From bc2e984a360e189722a853137a68e7d5b9b19302 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 14:28:44 +0300 Subject: [PATCH 349/832] update TODO comments --- unpythonic/seq.py | 1 - 1 file changed, 1 deletion(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 4c20d938..9344e065 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,7 +99,6 @@ def lazy_begin0(*bodys): body() return out -# TODO: check use of maybe_force_args and force1 in all function composition utilities # TODO: test the new lazify support in piping constructs # TODO: test multiple-return-values support in all function composition utilities # TODO: expand tests of `continuations` to cases with named return values From 1a198b9cc7362bf207eb0f288819c2a180ca00d2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 22 May 2021 14:37:09 +0300 Subject: [PATCH 350/832] update TODO comments --- unpythonic/seq.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 9344e065..36ab81f3 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -100,6 +100,9 @@ def lazy_begin0(*bodys): return out # TODO: test the new lazify support in piping constructs +# TODO: test `Values` handling in `with lazify` +# - The `Values` container itself should always be eager (so it can be inspected without forcing the return value; important for symmetry with case of one positional return value) +# - Anything we place into it should get the regular treatment, because return values are never implicitly lazy # TODO: test multiple-return-values support in all function composition utilities # TODO: expand tests of `continuations` to cases with named return values # TODO: update code examples From de169d5f8ef704a51ac582fcb65211ed3a9cd021 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 May 2021 03:43:38 +0300 Subject: [PATCH 351/832] add sourcecode capture to lazy[] This works together with `Lazy`, which now takes an optional `sourcecode` argument. It is used as part of the repr; useful for debugging. --- unpythonic/lazyutil.py | 18 ++++++++++++++++-- unpythonic/syntax/lazify.py | 5 +++-- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/unpythonic/lazyutil.py b/unpythonic/lazyutil.py index 2f026a43..bfb553b7 100644 --- a/unpythonic/lazyutil.py +++ b/unpythonic/lazyutil.py @@ -34,11 +34,20 @@ def _init_module(): # called by unpythonic.__init__ when otherwise done class Lazy: """Delayed evaluation, with memoization. (A.k.a. *promise* in Racket.)""" - def __init__(self, thunk): - """`thunk`: 0-argument callable to be stored for delayed evaluation.""" + def __init__(self, thunk, *, sourcecode=None): + """Create a `Lazy` promise. + + `thunk`: 0-argument callable to be stored for delayed evaluation. + + `sourcecode`: str, optional, for use by the `lazy[]` macro. + + Source code of the thunk, if available. Used in the `repr`, + for debug purposes. + """ if not callable(thunk): raise TypeError(f"`thunk` must be a callable, got {type(thunk)} with value {repr(thunk)}") self.thunk = thunk + self.sourcecode = sourcecode self.value = _uninitialized self.thunk_returned_normally = _uninitialized @@ -62,6 +71,11 @@ def force(self): else: raise self.value + def __repr__(self): + if self.sourcecode: + return f'' + return f"" + def force1(x): """Force a ``Lazy`` promise. diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 7f7a8a04..2f3d50e8 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -7,10 +7,11 @@ Starred, keyword, List, Tuple, Dict, Set, Subscript, Load) from functools import partial -from mcpyrate.quotes import macros, q, a, h # noqa: F401 +from mcpyrate.quotes import macros, q, u, a, h # noqa: F401 from mcpyrate.astfixers import fix_ctx from mcpyrate.quotes import capture_as_macro, is_captured_value +from mcpyrate.unparser import unparse from mcpyrate.walkers import ASTTransformer from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, @@ -433,7 +434,7 @@ def doit(): # lazy: syntax transformer, lazify a single expression def _lazy(tree): - return q[h[Lazy](lambda: a[tree])] + return q[h[Lazy](lambda: a[tree], sourcecode=u[unparse(tree)])] # lazyrec: syntax transformer, recursively lazify elements in container literals # From f8c5ef4e8a3b8feda45cfde4a6a3e83c9394f89c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 May 2021 03:45:25 +0300 Subject: [PATCH 352/832] test pipes with `with lazify` --- unpythonic/syntax/tests/test_lazify.py | 120 +++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index c600fa6b..6cbd94f9 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -18,6 +18,7 @@ from ...it import flatten from ...llist import ll from ...misc import call, callwith +from ...seq import pipe1, piped1, lazy_piped1, pipe, pipec, piped, lazy_piped, exitpipe from ...tco import trampolined, jump from ...lazyutil import islazy, Lazy, force1, force # Lazy usually not needed in client code; for our tests only @@ -444,6 +445,125 @@ def h1(x): test[islazy(fact)] test[fact(5) == 120] + with testset("integration with pipes"): + # This is the testset from unpythonic/tests/test_seq.py, slightly modified. + with lazify: + double = lambda x: 2 * x + inc = lambda x: x + 1 + test[pipe1(42, double, inc) == 85] # 1-in-1-out + test[pipe1(42, inc, double) == 86] + test[pipe(42, double, inc) == 85] # n-in-m-out, supports also 1-in-1-out + test[pipe(42, inc, double) == 86] + + # 2-in-2-out + a, b = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y), + lambda x, y: Values(x * 2, y + 1)) + test[(a, b) == (6, 7)] + + # 2-in-2-out, pass intermediate result by name + a, b = pipe(Values(2, 3), + lambda x, y: Values(x=(x + 1), y=(2 * y)), + lambda x, y: Values(x * 2, y + 1)) + test[(a, b) == (6, 7)] + + # 2-in-2-out, also return final result by name + v = pipe(Values(2, 3), + lambda x, y: Values(x=(x + 1), y=(2 * y)), + lambda x, y: Values(a=(x * 2), b=(y + 1))) + test[v == Values(a=6, b=7)] + test[v["a"] == 6 and v["b"] == 7] # can access them via subscripting too + + # 2-in-eventually-3-out + a, b, c = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, "foo"), + lambda x, y, z: Values(x * 2, y + 1, f"got {z}")) + test[(a, b, c) == (6, 7, "got foo")] + + # 2-in-3-in-between-2-out + a, b = pipe(Values(2, 3), + lambda x, y: Values(x + 1, 2 * y, "foo"), + lambda x, y, s: Values(x * 2, y + 1, f"got {s}"), + lambda x, y, s: Values(x + y, s)) + test[(a, b) == (13, "got foo")] + + # pipec: curry the functions before running the pipeline + a, b = pipec(Values(1, 2), + lambda x: x + 1, # extra values passed through by curry (positionals on the right) + lambda x, y: Values(x * 2, y + 1)) + test[(a, b) == (4, 3)] + + with test_raises[TypeError, "should error when the curry context exits with args remaining"]: + a, b = pipec(Values(1, 2), + lambda x: x + 1, + lambda x: x * 2) + + # optional shell-like syntax + test[piped1(42) | double | inc | exitpipe == 85] + + y = piped1(42) | double + test[y | inc | exitpipe == 85] + test[y | exitpipe == 84] # y is never modified by the pipe system + + # multi-arg version + f = lambda x, y: Values(2 * x, y + 1) + g = lambda x, y: Values(x + 1, 2 * y) + x = piped(2, 3) | f | g | exitpipe # --> (5, 8) + test[x == Values(5, 8)] + + # abuse multi-arg version for single-arg case + test[piped(42) | double | inc | exitpipe == 85] + + with testset("integration with lazy pipes (plan computations)"): + # This is the testset from unpythonic/tests/test_seq.py, slightly modified. + with lazify: + # lazy pipe: compute later + lst = [1] + def append_succ(lis): + lis.append(lis[-1] + 1) + return lis # important, handed to the next function in the pipe + p = lazy_piped1(lst) | append_succ | append_succ # plan a computation + test[lst == [1]] # nothing done yet + p | exitpipe # run the computation + test[lst == [1, 2, 3]] # now the side effect has updated lst. + + # lazy pipe as an unfold + fibos = [] + def nextfibo(state): + a, b = state + fibos.append(a) # store result by side effect + return (b, a + b) # new state, handed to next function in the pipe + p = lazy_piped1((1, 1)) # load initial state into a lazy pipe + for _ in range(10): # set up pipeline + p = p | nextfibo + p | exitpipe + test[fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]] + + # multi-arg lazy pipe + p1 = lazy_piped(2, 3) + p2 = p1 | (lambda x, y: Values(x + 1, 2 * y, "foo")) + p3 = p2 | (lambda x, y, s: Values(x * 2, y + 1, f"got {s}")) + p4 = p3 | (lambda x, y, s: Values(x + y, s)) + # nothing done yet, and all computations purely functional: + test[(p1 | exitpipe) == Values(2, 3)] + test[(p2 | exitpipe) == Values(3, 6, "foo")] # runs the chain up to p2 + test[(p3 | exitpipe) == Values(6, 7, "got foo")] # runs the chain up to p3 + test[(p4 | exitpipe) == Values(13, "got foo")] + + # multi-arg lazy pipe as an unfold + fibos = [] + def nextfibo(a, b): # now two arguments + fibos.append(a) + return Values(a=b, b=(a + b)) # can return by name too + p = lazy_piped(1, 1) + for _ in range(10): + p = p | nextfibo + test[p | exitpipe == Values(a=89, b=144)] # final state + test[fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]] + + # abuse multi-arg version for single-arg case + test[lazy_piped(42) | double | inc | exitpipe == 85] + with testset("integration with TCO"): with lazify: @trampolined From 5571c8cebd8528a53478e38435324211e2d3e073 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 May 2021 03:45:39 +0300 Subject: [PATCH 353/832] fix bugs found in new tests --- unpythonic/seq.py | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 36ab81f3..200d9694 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -166,7 +166,10 @@ def pipe1(value0, *bodys): # def x(loop, update, acc): # return loop(update(acc)) # return x - x = value0 + + # This is forced when it is passed to an eager body (or when a lazy body uses it), + # but we force it here just for symmetry with the multi-arg version of `pipe`. + x = force1(value0) for update in bodys: update = force1(update) x = maybe_force_args(update, x) @@ -268,7 +271,12 @@ def nextfibo(state): v = self._x for g in self._funcs: v = maybe_force_args(g, v) - return v + # In `unpythonic`, return values are never implicitly lazy. + # The final result here is a return value. + # + # It is legal to pipe the initial value immediately to `exitpipe`; + # in that case, in a `with lazify` block, it will be a promise. + return force(v) # just pass on the reference to the original x. cls = self.__class__ return cls(x=self._x, _funcs=self._funcs + (force1(f),)) @@ -323,7 +331,10 @@ def pipe(values0, *bodys): lambda x, y, s: Values(x + y, s)) assert (a, b) == (13, "got foo") """ - xs = values0 + # We must force `values0` to analyze it, because we treat `Values` objects separately. + # Otherwise, in a `with lazify` block, the lazified `Values` object will get passed as + # one argument to the first body - not what we want. + xs = force1(values0) n = len(bodys) for k, update in enumerate(bodys): islast = (k == n - 1) @@ -457,13 +468,20 @@ def __or__(self, f): vs = self._xs for g in self._funcs: if isinstance(vs, Values): - vs = g(*vs.rets, **vs.kwrets) + vs = maybe_force_args(g, *vs.rets, **vs.kwrets) else: - vs = g(vs) + vs = maybe_force_args(g, vs) if isinstance(vs, Values): - return vs if vs.kwrets or len(vs.rets) > 1 else vs[0] + ret = vs if vs.kwrets or len(vs.rets) > 1 else vs[0] else: - return vs + ret = vs + # In `unpythonic`, return values are never implicitly lazy. + # The final result here is a return value. + # + # It is legal to pipe the initial value immediately to `exitpipe`; + # in that case, in a `with lazify` block, it will be a promise + # (or a `Values` of several promises). + return force(ret) # just pass on the references to the original xs. cls = self.__class__ return cls(*self._xs.rets, _funcs=self._funcs + (force1(f),), **self._xs.kwrets) From 93bece70378c08f2eda0edfca5c98ff6e81ebf88 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 May 2021 03:45:45 +0300 Subject: [PATCH 354/832] update TODO comments --- unpythonic/seq.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 200d9694..4a79dd56 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,13 +99,12 @@ def lazy_begin0(*bodys): body() return out -# TODO: test the new lazify support in piping constructs # TODO: test `Values` handling in `with lazify` # - The `Values` container itself should always be eager (so it can be inspected without forcing the return value; important for symmetry with case of one positional return value) # - Anything we place into it should get the regular treatment, because return values are never implicitly lazy -# TODO: test multiple-return-values support in all function composition utilities +# TODO: test multiple-return-values support in all function composition utilities (`curry`, `compose` family, `pipe` family) # TODO: expand tests of `continuations` to cases with named return values -# TODO: update code examples +# TODO: update code examples in docs # sequence one-input, one-output functions @passthrough_lazy_args From 16c4a8eaff9f4f2f0a29bc7d77657449d00c6d2e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:04:55 +0300 Subject: [PATCH 355/832] add tests for `Values` handling in `with lazify` --- unpythonic/seq.py | 3 --- unpythonic/syntax/tests/test_lazify.py | 31 ++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 4a79dd56..b5944117 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,9 +99,6 @@ def lazy_begin0(*bodys): body() return out -# TODO: test `Values` handling in `with lazify` -# - The `Values` container itself should always be eager (so it can be inspected without forcing the return value; important for symmetry with case of one positional return value) -# - Anything we place into it should get the regular treatment, because return values are never implicitly lazy # TODO: test multiple-return-values support in all function composition utilities (`curry`, `compose` family, `pipe` family) # TODO: expand tests of `continuations` to cases with named return values # TODO: update code examples in docs diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 6cbd94f9..d4569ad7 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -373,6 +373,37 @@ def f(a, b): # letrec injects lambdas into its bindings, so test it too. test[letrec[[c << 42, d << e] in f(c, d)] == 42] + # In `unpythonic`, return values are never implicitly lazy. + # At the minimum, you can always inspect whether it is an object or a `Values` instance, + # representing multiple and/or named return values. + with testset("interaction with Values (multiple and named return values)"): + with lazify: + def multireturn1(a, b): + # As usual, the mention of `a` and `b` inside the function body forces the promises. + # It doesn't matter whether the mention occurs in a `Values(...)` call. + return Values(a, b) + test[isinstance(multireturn1(2, 3), Values)] + test[multireturn1(2, 3) == Values(2, 3)] + + def multireturn2(a, b): + # Assignment to a temporary doesn't matter; `lazify` detects the `Values(...)` call anywhere. + tmp = Values(a, b) + return tmp + test[isinstance(multireturn2(2, 3), Values)] + test[multireturn2(2, 3) == Values(2, 3)] + + def namedreturn1(x, y): + # Named return values can be given as named arguments. + return Values(x=x, y=y) + test[isinstance(namedreturn1(2, 3), Values)] + test[namedreturn1(2, 3) == Values(x=2, y=3)] + + def namedreturn2(x, y): + tmp = Values(x=x, y=y) + return tmp + test[isinstance(namedreturn2(2, 3), Values)] + test[namedreturn2(2, 3) == Values(x=2, y=3)] + # various higher-order functions, mostly from unpythonic.fun with testset("interaction with higher-order functions"): with lazify: From 4e9fe0ca5bab1aa34c7d42c148c57c23306f607e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:50:40 +0300 Subject: [PATCH 356/832] Values does not need to be marked lazy ...because return values are never implicitly lazy in `unpythonic`. --- unpythonic/collections.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/collections.py b/unpythonic/collections.py index ea324c87..b6cf44e0 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -22,7 +22,6 @@ from .env import env from .dynassign import _Dyn -from .lazyutil import passthrough_lazy_args from .llist import cons, Nil from .misc import getattrrec from .regutil import register_decorator @@ -906,7 +905,6 @@ def _canonize_slice(s, length=None, wrap=None): # convert negatives, inject def # ----------------------------------------------------------------------------- -@passthrough_lazy_args class Values: """Structured multiple-return-values. From f713a3f42a84082cfaf057a0fc987fb7472d985b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:51:03 +0300 Subject: [PATCH 357/832] improve comment --- unpythonic/lazyutil.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/lazyutil.py b/unpythonic/lazyutil.py index bfb553b7..159f51cf 100644 --- a/unpythonic/lazyutil.py +++ b/unpythonic/lazyutil.py @@ -117,7 +117,7 @@ def maybe_force_args(f, *thunks, **kwthunks): If `not islazy(f)`, forces the given args and kwargs, and then calls `f` with them. If `islazy(f)`, calls `f` without forcing the args/kwargs. """ - if f is jump: # special case to avoid drastic performance hit in strict code + if f is jump: # special case to avoid drastic performance hit in TCO'd strict code target, *argthunks = thunks return jump(force1(target), *argthunks, **kwthunks) if islazy(f): From dd5dd188cfa04782267aaedc74efdf002902eaab Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:51:11 +0300 Subject: [PATCH 358/832] optimize: don't autocurry `Values(...)` calls --- unpythonic/syntax/autocurry.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 5eb775a7..2dca5984 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -85,7 +85,10 @@ def transform(self, tree): return tree hascurry = self.state.hascurry - if type(tree) is Call: + # Curry all calls; except as a small optimization, skip `Values(...)`, + # which accepts any args and kwargs, so currying it does not make sense. + # (It represents multiple-return-values in `unpythonic`.) + if type(tree) is Call and not isx(tree.func, "Values"): if has_curry(tree): # detect decorated lambda with manual curry # the lambda inside the curry(...) is the next Lambda node we will descend into. hascurry = True From 276e867d9ccfd9d386de46cb63fcd0811d88a391 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:51:24 +0300 Subject: [PATCH 359/832] use `unparse(tree, debug=True)` for `lazy[]` sourcecode arg This shows hygienic captures in a much cleaner format, useful for debugging. --- unpythonic/syntax/lazify.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 2f3d50e8..78c533c2 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -434,7 +434,7 @@ def doit(): # lazy: syntax transformer, lazify a single expression def _lazy(tree): - return q[h[Lazy](lambda: a[tree], sourcecode=u[unparse(tree)])] + return q[h[Lazy](lambda: a[tree], sourcecode=u[f"lazy[{unparse(tree, debug=True)}]"])] # lazyrec: syntax transformer, recursively lazify elements in container literals # From d637c3a5fc9595b1850646cb88ec9685853e803a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:52:12 +0300 Subject: [PATCH 360/832] add comment --- unpythonic/syntax/tailtools.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 206cc421..8a8fca90 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -751,6 +751,21 @@ def cc(*rets, **kwrets): if cc1 is not None: @passthrough_lazy_args def cc(return_value): + # Return values are never implicitly lazy in `unpythonic`, + # so why we need to `force1` here requires a comment. + # + # In general, we should treat these `cc` functions as lazy, + # so they won't force their args. Those args here are a return value, + # but due to `continuations`, it's not just a return, but a call + # into the `cc` function. + # + # Thus, returning a `Values` from a continuation-enabled function, + # that `Values` ends up here (or in the other branch, with no `cc1`). + # Because it's *technically* an argument for a lazy function, it gets + # a `lazy[]` wrapper added by `with lazify`. + # + # To determine whether we have one or multiple return values, we must + # force that wrapper promise, without touching anything inside. return_value = force1(return_value) if isinstance(return_value, Values): return jump(cc1, cc=cc2, *return_value.rets, **return_value.kwrets) From f1b0ee4af1885fa6dc3afc64fe4b4d6ad87b6bad Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 May 2021 00:55:37 +0300 Subject: [PATCH 361/832] update TODOs --- unpythonic/seq.py | 1 + 1 file changed, 1 insertion(+) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index b5944117..9717f514 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,6 +99,7 @@ def lazy_begin0(*bodys): body() return out +# TODO: move `call`, `callwith`, `Values` into a `funcutils.py`? # TODO: test multiple-return-values support in all function composition utilities (`curry`, `compose` family, `pipe` family) # TODO: expand tests of `continuations` to cases with named return values # TODO: update code examples in docs From 24ad34bbccf7b38610066bc4ef632e879b4e38e9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 00:36:26 +0300 Subject: [PATCH 362/832] update docstring --- unpythonic/misc.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/unpythonic/misc.py b/unpythonic/misc.py index 991cf3ce..19d01dfb 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -206,9 +206,6 @@ def pack(*args): In other words, the inverse of tuple unpacking, as a function. E.g. ``pack(a, b, c)`` is the same as ``(a, b, c)``. - Or, if we semantically consider a tuple as a representation for multiple - return values, this is the identity function, returning its args. - We provide this because the default constructor `tuple(...)` requires an iterable, and there are use cases where it is useful to be able to say *pack these args into a tuple*. From 1bae38831c1ddd2f59a42a9b3d8289c10259b4ff Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:13:25 +0300 Subject: [PATCH 363/832] improve lazify support in some function composition utilities --- unpythonic/fun.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 4c7e4d9b..f94bb469 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -687,6 +687,7 @@ def andf(*fs): # Racket: conjoin assert andf(lambda x: isinstance(x, int), lambda x: x % 2 == 0)(42) is True assert andf(lambda x: isinstance(x, int), lambda x: x % 2 == 0)(43) is False """ + @passthrough_lazy_args def conjoined(*args, **kwargs): b = True for f in fs: @@ -694,8 +695,6 @@ def conjoined(*args, **kwargs): if not b: return False return b - if all(islazy(f) for f in fs): - conjoined = passthrough_lazy_args(conjoined) return conjoined def orf(*fs): # Racket: disjoin @@ -715,6 +714,7 @@ def orf(*fs): # Racket: disjoin assert orf(isstr, iseven)("foo") is True assert orf(isstr, iseven)(None) is False # neither condition holds """ + @passthrough_lazy_args def disjoined(*args, **kwargs): b = False for f in fs: @@ -722,8 +722,6 @@ def disjoined(*args, **kwargs): if b: return b return False - if all(islazy(f) for f in fs): - disjoined = passthrough_lazy_args(disjoined) return disjoined def _make_compose1(direction): @@ -769,8 +767,7 @@ def compose1(fs): # - if fs is empty, we output None # - if fs contains only one item, we output it as-is composed = reducel(compose1_two, fs) # op(elt, acc) - if all(islazy(f) for f in fs): - composed = passthrough_lazy_args(composed) + composed = passthrough_lazy_args(composed) return composed return compose1 @@ -858,8 +855,7 @@ def compose(fs): """ fs = force(fs) composed = reducel(compose_two, fs) # op(elt, acc) - if all(islazy(f) for f in fs): - composed = passthrough_lazy_args(composed) + composed = passthrough_lazy_args(composed) return composed return compose From 6cd3b07203c6ccc1ec67601a5e5e7d3fc9b61046 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:14:16 +0300 Subject: [PATCH 364/832] update comment --- unpythonic/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 8e441fbb..a1b55249 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -28,7 +28,7 @@ from .it import * # noqa: F401, F403 from .let import * # no guarantees on evaluation order (before Python 3.6), nice syntax # noqa: F401, F403 -# guaranteed evaluation order, clunky syntax +# code generation target API for macros from .lispylet import (let as ordered_let, letrec as ordered_letrec, # noqa: F401 dlet as ordered_dlet, dletrec as ordered_dletrec, blet as ordered_blet, bletrec as ordered_bletrec) From 6cb8ea9e9384a269f0e9d953f260bc1a9fd7beae Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:16:58 +0300 Subject: [PATCH 365/832] update comment --- unpythonic/collections.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/collections.py b/unpythonic/collections.py index b6cf44e0..eb0503e1 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -131,7 +131,7 @@ def doit(x): elif isinstance(x, MappingView): return {doit(elt) for elt in x} # env and dyn provide the Mapping API, but shouldn't get the general Mapping treatment here. - # (This is important for the curry and lazify macros.) + # (This is important for the autocurry and lazify macros.) elif isinstance(x, Mapping) and not isinstance(x, (env, _Dyn)): ctor = type(x) return ctor({k: doit(v) for k, v in x.items()}) From 5eff93dce18a485faafa10cc9759b6ca822b5f41 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:19:40 +0300 Subject: [PATCH 366/832] add comment --- unpythonic/collections.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/unpythonic/collections.py b/unpythonic/collections.py index eb0503e1..da548cf7 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -20,6 +20,11 @@ from operator import lt, le, ge, gt import threading +# Some of these are used only to detect (and perhaps mogrify) our own cat food in `mogrify`. +# +# Still, importing these has the rather unpleasant consequence of creating dependency loops +# (circular imports), because many other modules in `unpythonic` use definitions from the +# `unpythonic.collections` module. from .env import env from .dynassign import _Dyn from .llist import cons, Nil From 3fc067141d4f5c2f03d21dd9aa2f2a386925dceb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:24:26 +0300 Subject: [PATCH 367/832] call, callwith, Values, valuify -> new module unpythonic.funutil --- unpythonic/__init__.py | 7 +- unpythonic/collections.py | 209 +---------- unpythonic/dialects/tests/test_pytkell.py | 2 +- unpythonic/fun.py | 7 +- unpythonic/funutil.py | 406 ++++++++++++++++++++++ unpythonic/let.py | 4 +- unpythonic/lispylet.py | 4 +- unpythonic/misc.py | 183 +--------- unpythonic/seq.py | 2 +- unpythonic/syntax/letdo.py | 2 +- unpythonic/syntax/prefix.py | 2 +- unpythonic/syntax/tailtools.py | 2 +- unpythonic/syntax/tests/test_conts.py | 4 +- unpythonic/syntax/tests/test_dbg.py | 2 +- unpythonic/syntax/tests/test_lazify.py | 4 +- unpythonic/syntax/tests/test_tco.py | 2 +- unpythonic/tests/test_fploop.py | 5 +- unpythonic/tests/test_fun.py | 2 +- unpythonic/tests/test_funutil.py | 99 ++++++ unpythonic/tests/test_gmemo.py | 3 +- unpythonic/tests/test_it.py | 2 +- unpythonic/tests/test_let.py | 2 +- unpythonic/tests/test_misc.py | 87 +---- unpythonic/tests/test_seq.py | 2 +- 24 files changed, 543 insertions(+), 501 deletions(-) create mode 100644 unpythonic/funutil.py create mode 100644 unpythonic/tests/test_funutil.py diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index a1b55249..a1e936f9 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -44,8 +44,13 @@ from .tco import * # noqa: F401, F403 from .typecheck import * # noqa: F401, F403 -# HACK: break dependency loop +# HACK: break dependency loops for circular imports from .lazyutil import _init_module _init_module() del _init_module from .lazyutil import Lazy, force1, force # noqa: F401 + +from .funutil import _init_module +_init_module() +del _init_module +from .funutil import * # noqa: F401, F403 diff --git a/unpythonic/collections.py b/unpythonic/collections.py index da548cf7..9d544168 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -5,8 +5,7 @@ "frozendict", "roview", "view", "ShadowedSequence", "mogrify", "get_abcs", "in_slice", "index_in_slice", - "SequenceView", "MutableSequenceView", # ABCs - "Values", "valuify"] + "SequenceView", "MutableSequenceView"] # ABCs from functools import wraps from itertools import repeat @@ -27,9 +26,9 @@ # `unpythonic.collections` module. from .env import env from .dynassign import _Dyn +from .funutil import Values from .llist import cons, Nil from .misc import getattrrec -from .regutil import register_decorator def get_abcs(cls): """Return a set of the collections.abc superclasses of cls (virtuals too).""" @@ -907,207 +906,3 @@ def _canonize_slice(s, length=None, wrap=None): # convert negatives, inject def stop = -1 # yes, really -1 to have index 0 inside the slice return start, stop, step - -# ----------------------------------------------------------------------------- - -class Values: - """Structured multiple-return-values. - - That is, return multiple values positionally and by name. This completes - the symmetry between passing function arguments and returning values - from a function: Python itself allows passing arguments by name, but has - no concept of returning values by name. This class adds that concept. - - Having a `Values` type separate from `tuple` also helps with semantic - accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value now - means just that - one value that is a `tuple`. It is different from a - `Values` that contains several positional return values (that are meant - to be treated separately). - - **When to use**: - - Most of the time, returning a tuple to denote multiple-return-values - and unpacking it is just fine, and that is exactly what `unpythonic` - does internally in many places. - - But the distinction is critically important for function composition, - so that positional return values can be automatically mapped into - positional arguments to the next function in the chain, and named - return values into named arguments. - - Accordingly, various parts of `unpythonic` that deal with function - composition use the `Values` abstraction; particularly `curry`, and - the `compose` and `pipe` families. - - **Behavior**: - - `Values` is a duck-type with some features of both sequences and mappings, - but not the full `collections.abc` API of either. - - Each operation that obviously and without ambiguity makes sense only - for the positional or named part, accesses that part. - - The only exception is `__getitem__` (subscripting), which makes sense - for both parts, unambiguously, because the key types differ. If the index - expression is an `int` or a `slice`, it is an index/slice for the - positional part. If it is an `str`, it is a key for the named part. - - If you need to explicitly access either part (and its full API), - use the `rets` and `kwrets` attributes. The names are in analogy - with `args` and `kwargs`. - - `rets` is a `tuple`, and `kwrets` is a `frozendict`. - - `Values` objects can be compared for equality. Two `Values` objects - are equal if both their `rets` and `kwrets` (respectively) are. - - Examples:: - - def f(): - return Values(1, 2, 3) - result = f() - assert isinstance(result, Values) - assert result.rets == (1, 2, 3) - assert not result.kwrets - assert result[0] == 1 - assert result[:-1] == (1, 2) - a, b, c = result # if no kwrets, can be unpacked like a tuple - a, b, c = f() - - def g(): - return Values(x=3) # named return value - result = g() - assert isinstance(result, Values) - assert not result.rets - assert result.kwrets == {"x": 3} # actually a `frozendict` - assert "x" in result # `in` looks in the named part - assert result["x"] == 3 - assert result.get("x", None) == 3 - assert result.get("y", None) == None - assert tuple(results.keys()) == ("x",) # also `values()`, `items()` - - def h(): - return Values(1, 2, x=3) - result = h() - assert isinstance(result, Values) - assert result.rets == (1, 2) - assert result.kwrets == {"x": 3} - a, b = result.rets # positionals can always be unpacked explicitly - assert result[0] == 1 - assert "x" in result - assert result["x"] == 3 - - def silly_but_legal(): - return Values(42) - result = silly_but_legal() - assert result.rets[0] == 42 - assert result.ret == 42 # shorthand for single-value case - - The last example is silly, but legal, because it is preferable to just omit - the `Values` if it is known that there is only one return value. (This also - applies when that value is a `tuple`, when the intent is to return it as a - single `tuple`, in contexts where this distinction matters.) - """ - def __init__(self, *rets, **kwrets): - """Create a `Values` object. - - `rets`: positional return values - `kwrets`: named return values - """ - self.rets = rets - self.kwrets = frozendict(kwrets) - - # Shorthand for one-value case - def _ret(self): - return self.rets[0] - ret = property(fget=_ret, doc="Shorthand for `self.rets[0]`. Read-only.") - - # Iterable - def __iter__(self): - """Values is iterable when there are no `kwrets`; this then iterates over `rets`. - - This is meant to minimize impact on existing code that receives a `tuple` - as a pythonic multiple-return-values idiom. Changing the `return` to - return a `Values` instead requires no changes at the receiving end - (unless you change the sending end to return some named values; - if you do, then it *should* yell, to avoid silently discarding - those named values). - - Note that you can iterate over `rets` or `kwrets` to explicitly state - which you mean; that always works. - """ - if self.kwrets: - raise ValueError(f"Named values present, cannot iterate over all values. Got: {self.kwrets}") - return iter(self.rets) - - # Sequence (no full support: no `__len__`, `__reversed__`, `index`, `count`) - def __getitem__(self, idx): - """Subscripting. - - Indexing by an `int` or `slice` indexes the positional part. - Indexing by an `str` indexes the named part. - - Indexing by any other type raises `TypeError`. - """ - # multi-headed hydra - if isinstance(idx, (int, slice)): - return self.rets[idx] - elif isinstance(idx, str): - return self.kwrets[idx] - raise TypeError(f"Expected either int, slice or str subscript, got {type(idx)} with value {repr(idx)}") - - # Container - def __contains__(self, k): - """The `in` operator, looks in the named part.""" - return k in self.kwrets - - # Mapping (no full support: no `__len__`) - def items(self): - """Items of the named part.""" - return self.kwrets.items() - def keys(self): - """Keys of the named part.""" - return self.kwrets.keys() - def values(self): - """Values of the named part.""" - return self.kwrets.values() - def get(self, k, default=None): - """Dict-like `get` for the named part.""" - return self[k] if k in self else default - - # comparison - def __eq__(self, other): - """Equality comparison. - - Two `Values` objects are equal if both their `rets` and `kwrets` - (respectively) are. - """ - if not isinstance(other, Values): - return False - return other.rets == self.rets and other.kwrets == self.kwrets - def __ne__(self, other): - """Inequality comparison.""" - return not (self == other) - - # no `__len__`, because we have two candidates - - # pretty-printing - def __repr__(self): # pragma: no cover - """Pretty-printing. Eval-able if the contents are.""" - rets_list = [repr(x) for x in self.rets] - rets_str = ", ".join(rets_list) - kwrets_list = [f"{name}={repr(value)}" for name, value in self.kwrets.items()] - kwrets_str = ", ".join(kwrets_list) - sep = ", " if self.rets and self.kwrets else "" - return f"Values({rets_str}{sep}{kwrets_str})" - -@register_decorator(priority=30) -def valuify(f): - """Decorator. If `f` returns `tuple` (exactly, no subclass), convert into `Values`, else pass through.""" - @wraps(f) - def valuified(*args, **kwargs): - result = f(*args, **kwargs) - if type(result) is tuple: # yes, exactly tuple - result = Values(*result) - return result - return valuified diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index e4bc69e5..94c9a77a 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -8,7 +8,7 @@ from ...test.fixtures import session, testset from ...syntax import macros, continuations, call_cc, tco # noqa: F401, F811 -from ...collections import Values +from ...funutil import Values from ...misc import timer from types import FunctionType diff --git a/unpythonic/fun.py b/unpythonic/fun.py index f94bb469..89815076 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -28,6 +28,7 @@ _get_argument_type_mismatches, _raise_multiple_dispatch_error, _list_multimethods, _extract_self_or_cls) from .dynassign import dyn, make_dynvar +from .funutil import Values from .regutil import register_decorator from .symbol import sym @@ -282,8 +283,6 @@ def f(x, y): return named outputs. See: https://github.com/Technologicat/unpythonic/issues/32 """ - from .collections import Values # circular import - f = force(f) # lazify support: we need the value of f # trivial case first: interaction with call_ec and other replace-def-with-value decorators if not callable(f): @@ -622,7 +621,6 @@ def identity(*args, **kwargs): assert identity(42) == 42 assert identity() is None """ - from .collections import Values # circular import if not args and not kwargs: return None return Values(*args, **kwargs) if kwargs or len(args) > 1 else args[0] @@ -649,7 +647,6 @@ def const(*args, **kwargs): c = const() assert c("anything") is None """ - from .collections import Values # circular import if not args and not kwargs: ret = None else: @@ -830,7 +827,6 @@ def compose_two(f, g): (f ∘ g)(...) ≡ f(g(...)) """ - from .collections import Values # circular import def composed(*args, **kwargs): bindings = {} if iscurried(f): @@ -927,7 +923,6 @@ def tokth(k, f): Especially useful in multi-arg compose chains. See ``unpythonic.test.test_fun`` for examples. """ - from .collections import Values # circular import def apply_f_to_kth_arg(*args, **kwargs): n = len(args) if not n: diff --git a/unpythonic/funutil.py b/unpythonic/funutil.py new file mode 100644 index 00000000..ccdf05c8 --- /dev/null +++ b/unpythonic/funutil.py @@ -0,0 +1,406 @@ +# -*- coding: utf-8 -*- +"""Function call and return value related utilities.""" + +__all__ = ["call", "callwith", + "Values", "valuify"] + +from functools import wraps + +from .lazyutil import passthrough_lazy_args, islazy, maybe_force_args, force +from .regutil import register_decorator +from .symbol import sym + +# HACK: break dependency loop llist -> fun -> funutil -> collections -> llist +_init_done = False +frozendict = sym("frozendict") # doesn't matter what the value is, will be overwritten later +def _init_module(): # called by unpythonic.__init__ when otherwise done + global frozendict, _init_done + from .collections import frozendict + _init_done = True + +# Only the single-argument form (just f) of the "call" decorator is supported by unpythonic.syntax.util.sort_lambda_decorators. +# +# This is as it should be; if given any arguments beside f, the call doesn't conform +# to the decorator API, but is a normal function call. See "callwith" if you need to +# pass arguments and then call f from a decorator position. +@register_decorator(priority=80) +@passthrough_lazy_args +def call(f, *args, **kwargs): + """Call the function f. + + **When used as a decorator**: + + Run the function immediately, then overwrite the definition by its + return value. + + Useful for making lispy not-quite-functions where the def just delimits + a block of code that runs immediately (think call-with-something in Lisps, + but without the something). + + The function will be called with no arguments. If you need to pass + arguments when using ``call`` as a decorator, see ``callwith``. + + **When called normally**: + + ``call(f, *a, **kw)`` is the same as ``f(*a, **kw)``. + + *Why ever use call() normally?* + + - Readability and aesthetics in cases like ``makef(dostuffwith(args))()``, + where ``makef`` is a function factory, and we want to immediately + call its result. + + Rewriting this as ``call(makef(dostuffwith(args)))`` relocates the + odd one out from the mass of parentheses at the end. (A real FP example + would likely have more levels of nesting.) + + - Notational uniformity with ``curry(f, *args, **kwargs)`` for cases + without currying. See ``unpythonic.fun.curry``. + + - For fans of S-expressions. Write Python almost like Lisp! + + Name inspired by "call-with-something", but since here we're calling + without any specific thing, it's just "call". + + Examples:: + + @call + def result(): # this block of code runs immediately + return "hello" + print(result) # "hello" + + # if the return value is of no interest: + @call + def _(): + ... # code with cheeky side effects goes here + + @call + def x(): + a = 2 # many temporaries that help readability... + b = 3 # ...of this calculation, but would just pollute locals... + c = 5 # ...after the block exits + return a * b * c + + @call + def _(): + for x in range(10): + for y in range(10): + if x * y == 42: + return # "multi-break" out of both loops! + ... + + Note that in the multi-break case, ``x`` and ``y`` are no longer in scope + outside the block, since the block is a function. + """ +# return f(*args, **kwargs) + return maybe_force_args(force(f), *args, **kwargs) # support unpythonic.syntax.lazify + +@register_decorator(priority=80) +@passthrough_lazy_args +def callwith(*args, **kwargs): + """Freeze arguments, choose function later. + + **Used as decorator**, this is like ``@call``, but with arguments:: + + @callwith(3) + def result(x): + return x**2 + assert result == 9 + + **Called normally**, this creates a function to apply the given arguments + to a callable to be specified later:: + + def myadd(a, b): + return a + b + def mymul(a, b): + return a * b + apply23 = callwith(2, 3) + assert apply23(myadd) == 5 + assert apply23(mymul) == 6 + + When called normally, the two-step application is mandatory. The first step + stores the given arguments. It returns a function ``f(callable)``. When + ``f`` is called, it calls its ``callable`` argument, passing in the arguments + stored in the first step. + + In other words, ``callwith`` is similar to ``functools.partial``, but without + specializing to any particular function. The function to be called is + given later, in the second step. + + Hence, ``callwith(2, 3)(myadd)`` means "make a function that passes in + two positional arguments, with values ``2`` and ``3``. Then call this + function for the callable ``myadd``". + + But if we instead write``callwith(2, 3, myadd)``, it means "make a function + that passes in three positional arguments, with values ``2``, ``3`` and + ``myadd`` - not what we want in the above example. + + Curry obviously does not help; it will happily pass in all arguments + in one go. If you want to specialize some arguments now and some later, + use ``partial``:: + + from functools import partial + + p1 = partial(callwith, 2) + p2 = partial(p1, 3) + p3 = partial(p2, 4) + apply234 = p3() # actually call callwith, get the function + def add3(a, b, c): + return a + b + c + def mul3(a, b, c): + return a * b * c + assert apply234(add3) == 9 + assert apply234(mul3) == 24 + + If the code above feels weird, it should. Arguments are gathered first, + and the function to which they will be passed is chosen in the last step. + + A pythonic alternative to the above examples is:: + + a = [2, 3] + def myadd(a, b): + return a + b + def mymul(a, b): + return a * b + assert myadd(*a) == 5 + assert mymul(*a) == 6 + + a = [2] + a += [3] + a += [4] + def add3(a, b, c): + return a + b + c + def mul3(a, b, c): + return a * b * c + assert add3(*a) == 9 + assert mul3(*a) == 24 + + Another use case of ``callwith`` is ``map``, if we want to vary the function + instead of the data:: + + m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) + assert tuple(m) == (6, 9, 3**(1/2)) + + The pythonic alternative here is to use the comprehension notation, + which can already do this:: + + m = (f(3) for f in [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) + assert tuple(m) == (6, 9, 3**(1/2)) + + Inspiration: + + *Function application with $* in + http://learnyouahaskell.com/higher-order-functions + """ + def applyfrozenargsto(f): + return maybe_force_args(force(f), *args, **kwargs) + return applyfrozenargsto + + +class Values: + """Structured multiple-return-values. + + That is, return multiple values positionally and by name. This completes + the symmetry between passing function arguments and returning values + from a function: Python itself allows passing arguments by name, but has + no concept of returning values by name. This class adds that concept. + + Having a `Values` type separate from `tuple` also helps with semantic + accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value now + means just that - one value that is a `tuple`. It is different from a + `Values` that contains several positional return values (that are meant + to be treated separately e.g. by a function composition utility). + + **When to use**: + + Most of the time, returning a tuple to denote multiple-return-values + and unpacking it is just fine, and that is exactly what `unpythonic` + does internally in many places. + + But the distinction is critically important in function composition, + so that positional return values can be automatically mapped into + positional arguments to the next function in the chain, and named + return values into named arguments. + + Accordingly, various parts of `unpythonic` that deal with function + composition use the `Values` abstraction; particularly `curry`, and + the `compose` and `pipe` families. + + **Behavior**: + + `Values` is a duck-type with some features of both sequences and mappings, + but not the full `collections.abc` API of either. + + Each operation that obviously and without ambiguity makes sense only + for the positional or named part, accesses that part. + + The only exception is `__getitem__` (subscripting), which makes sense + for both parts, unambiguously, because the key types differ. If the index + expression is an `int` or a `slice`, it is an index/slice for the + positional part. If it is an `str`, it is a key for the named part. + + If you need to explicitly access either part (and its full API), + use the `rets` and `kwrets` attributes. The names are in analogy + with `args` and `kwargs`. + + `rets` is a `tuple`, and `kwrets` is an `unpythonic.collections.frozendict`. + + `Values` objects can be compared for equality. Two `Values` objects + are equal if both their `rets` and `kwrets` (respectively) are. + + Examples:: + + def f(): + return Values(1, 2, 3) + result = f() + assert isinstance(result, Values) + assert result.rets == (1, 2, 3) + assert not result.kwrets + assert result[0] == 1 + assert result[:-1] == (1, 2) + a, b, c = result # if no kwrets, can be unpacked like a tuple + a, b, c = f() + + def g(): + return Values(x=3) # named return value + result = g() + assert isinstance(result, Values) + assert not result.rets + assert result.kwrets == {"x": 3} # actually a `frozendict` + assert "x" in result # `in` looks in the named part + assert result["x"] == 3 + assert result.get("x", None) == 3 + assert result.get("y", None) == None + assert tuple(results.keys()) == ("x",) # also `values()`, `items()` + + def h(): + return Values(1, 2, x=3) + result = h() + assert isinstance(result, Values) + assert result.rets == (1, 2) + assert result.kwrets == {"x": 3} + a, b = result.rets # positionals can always be unpacked explicitly + assert result[0] == 1 + assert "x" in result + assert result["x"] == 3 + + def silly_but_legal(): + return Values(42) + result = silly_but_legal() + assert result.rets[0] == 42 + assert result.ret == 42 # shorthand for single-value case + + The last example is silly, but legal, because it is preferable to just omit + the `Values` if it is known that there is only one return value. (This also + applies when that value is a `tuple`, when the intent is to return it as a + single `tuple`, in contexts where this distinction matters.) + """ + def __init__(self, *rets, **kwrets): + """Create a `Values` object. + + `rets`: positional return values + `kwrets`: named return values + """ + self.rets = rets + self.kwrets = frozendict(kwrets) + + # Shorthand for one-value case + def _ret(self): + return self.rets[0] + ret = property(fget=_ret, doc="Shorthand for `self.rets[0]`. Read-only.") + + # Iterable + def __iter__(self): + """Values is iterable when there are no `kwrets`; this then iterates over `rets`. + + This is meant to minimize impact on existing code that receives a `tuple` + as a pythonic multiple-return-values idiom. Changing the `return` to + return a `Values` instead requires no changes at the receiving end + (unless you change the sending end to return some named values; + if you do, then it *should* yell, to avoid silently discarding + those named values). + + Note that you can iterate over `rets` or `kwrets` to explicitly state + which you mean; that always works. + """ + if self.kwrets: + raise ValueError(f"Named values present, cannot iterate over all values. Got: {self.kwrets}") + return iter(self.rets) + + # Sequence (no full support: no `__len__`, `__reversed__`, `index`, `count`) + def __getitem__(self, idx): + """Subscripting. + + Indexing by an `int` or `slice` indexes the positional part. + Indexing by an `str` indexes the named part. + + Indexing by any other type raises `TypeError`. + """ + # multi-headed hydra + if isinstance(idx, (int, slice)): + return self.rets[idx] + elif isinstance(idx, str): + return self.kwrets[idx] + raise TypeError(f"Expected either int, slice or str subscript, got {type(idx)} with value {repr(idx)}") + + # Container + def __contains__(self, k): + """The `in` operator, looks in the named part.""" + return k in self.kwrets + + # Mapping (no full support: no `__len__`) + def items(self): + """Items of the named part.""" + return self.kwrets.items() + def keys(self): + """Keys of the named part.""" + return self.kwrets.keys() + def values(self): + """Values of the named part.""" + return self.kwrets.values() + def get(self, k, default=None): + """Dict-like `get` for the named part.""" + return self[k] if k in self else default + + # comparison + def __eq__(self, other): + """Equality comparison. + + Two `Values` objects are equal if both their `rets` and `kwrets` + (respectively) are. + """ + if not isinstance(other, Values): + return False + return other.rets == self.rets and other.kwrets == self.kwrets + def __ne__(self, other): + """Inequality comparison.""" + return not (self == other) + + # no `__len__`, because we have two candidates + + # pretty-printing + def __repr__(self): # pragma: no cover + """Pretty-printing. Eval-able if the contents are.""" + rets_list = [repr(x) for x in self.rets] + rets_str = ", ".join(rets_list) + kwrets_list = [f"{name}={repr(value)}" for name, value in self.kwrets.items()] + kwrets_str = ", ".join(kwrets_list) + sep = ", " if self.rets and self.kwrets else "" + return f"Values({rets_str}{sep}{kwrets_str})" + + +@register_decorator(priority=30) +def valuify(f): + """Decorator. Convert the pythonic tuple-as-multiple-return-values idiom into `Values`. + + If `f` returns `tuple` (exactly, no subclass), convert into `Values`, else pass through. + """ + @wraps(f) + def valuified(*args, **kwargs): + result = f(*args, **kwargs) + if type(result) is tuple: # yes, exactly tuple + result = Values(*result) + return result + if islazy(f): + valuified = passthrough_lazy_args(valuified) + return valuified diff --git a/unpythonic/let.py b/unpythonic/let.py index 3a17d50d..e457a46e 100644 --- a/unpythonic/let.py +++ b/unpythonic/let.py @@ -5,9 +5,9 @@ from functools import wraps -from .misc import call -from .env import env as _envcls from .arity import arity_includes, UnknownArity +from .env import env as _envcls +from .funutil import call def let(body, **bindings): """``let`` expression. diff --git a/unpythonic/lispylet.py b/unpythonic/lispylet.py index 42bb8f50..da9c1516 100644 --- a/unpythonic/lispylet.py +++ b/unpythonic/lispylet.py @@ -5,9 +5,9 @@ from functools import wraps -from .misc import call -from .env import env as _envcls from .arity import arity_includes, UnknownArity +from .env import env as _envcls +from .funutil import call def let(bindings, body): """``let`` expression. diff --git a/unpythonic/misc.py b/unpythonic/misc.py index 19d01dfb..757db587 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -1,8 +1,7 @@ # -*- coding: utf-8 -*- """Miscellaneous constructs.""" -__all__ = ["call", "callwith", - "pack", +__all__ = ["pack", "namelambda", "timer", "getattrrec", "setattrrec", @@ -21,184 +20,6 @@ from types import CodeType, FunctionType, LambdaType from .regutil import register_decorator -from .lazyutil import passthrough_lazy_args, maybe_force_args, force - -# Only the single-argument form (just f) of the "call" decorator is supported by unpythonic.syntax.util.sort_lambda_decorators. -# -# This is as it should be; if given any arguments beside f, the call doesn't conform -# to the decorator API, but is a normal function call. See "callwith" if you need to -# pass arguments and then call f from a decorator position. -@register_decorator(priority=80) -@passthrough_lazy_args -def call(f, *args, **kwargs): - """Call the function f. - - **When used as a decorator**: - - Run the function immediately, then overwrite the definition by its - return value. - - Useful for making lispy not-quite-functions where the def just delimits - a block of code that runs immediately (think call-with-something in Lisps). - - The function will be called with no arguments. If you need to pass - arguments when using ``call`` as a decorator, see ``callwith``. - - **When called normally**: - - ``call(f, *a, **kw)`` is the same as ``f(*a, **kw)``. - - *Why ever use call() normally?* - - - Readability and aesthetics in cases like ``makef(dostuffwith(args))()``, - where ``makef`` is a function factory, and we want to immediately - call its result. - - Rewriting this as ``call(makef(dostuffwith(args)))`` relocates the - odd one out from the mass of parentheses at the end. (A real FP example - would likely have more levels of nesting.) - - - Notational uniformity with ``curry(f, *args, **kwargs)`` for cases - without currying. See ``unpythonic.fun.curry``. - - - For fans of S-expressions. Write Python almost like Lisp! - - Name inspired by "call-with-something", but since here we're calling - without any specific thing, it's just "call". - - Examples:: - - @call - def result(): # this block of code runs immediately - return "hello" - print(result) # "hello" - - # if the return value is of no interest: - @call - def _(): - ... # code with cheeky side effects goes here - - @call - def x(): - a = 2 # many temporaries that help readability... - b = 3 # ...of this calculation, but would just pollute locals... - c = 5 # ...after the block exits - return a * b * c - - @call - def _(): - for x in range(10): - for y in range(10): - if x * y == 42: - return # "multi-break" out of both loops! - ... - - Note that in the multi-break case, ``x`` and ``y`` are no longer in scope - outside the block, since the block is a function. - """ -# return f(*args, **kwargs) - return maybe_force_args(force(f), *args, **kwargs) # support unpythonic.syntax.lazify - -@register_decorator(priority=80) -@passthrough_lazy_args -def callwith(*args, **kwargs): - """Freeze arguments, choose function later. - - **Used as decorator**, this is like ``@call``, but with arguments:: - - @callwith(3) - def result(x): - return x**2 - assert result == 9 - - **Called normally**, this creates a function to apply the given arguments - to a callable to be specified later:: - - def myadd(a, b): - return a + b - def mymul(a, b): - return a * b - apply23 = callwith(2, 3) - assert apply23(myadd) == 5 - assert apply23(mymul) == 6 - - When called normally, the two-step application is mandatory. The first step - stores the given arguments. It returns a function ``f(callable)``. When - ``f`` is called, it calls its ``callable`` argument, passing in the arguments - stored in the first step. - - In other words, ``callwith`` is similar to ``functools.partial``, but without - specializing to any particular function. The function to be called is - given later, in the second step. - - Hence, ``callwith(2, 3)(myadd)`` means "make a function that passes in - two positional arguments, with values ``2`` and ``3``. Then call this - function for the callable ``myadd``". - - But if we instead write``callwith(2, 3, myadd)``, it means "make a function - that passes in three positional arguments, with values ``2``, ``3`` and - ``myadd`` - not what we want in the above example. - - Curry obviously does not help; it will happily pass in all arguments - in one go. If you want to specialize some arguments now and some later, - use ``partial``:: - - from functools import partial - - p1 = partial(callwith, 2) - p2 = partial(p1, 3) - p3 = partial(p2, 4) - apply234 = p3() # actually call callwith, get the function - def add3(a, b, c): - return a + b + c - def mul3(a, b, c): - return a * b * c - assert apply234(add3) == 9 - assert apply234(mul3) == 24 - - If the code above feels weird, it should. Arguments are gathered first, - and the function to which they will be passed is chosen in the last step. - - A pythonic alternative to the above examples is:: - - a = [2, 3] - def myadd(a, b): - return a + b - def mymul(a, b): - return a * b - assert myadd(*a) == 5 - assert mymul(*a) == 6 - - a = [2] - a += [3] - a += [4] - def add3(a, b, c): - return a + b + c - def mul3(a, b, c): - return a * b * c - assert add3(*a) == 9 - assert mul3(*a) == 24 - - Another use case of ``callwith`` is ``map``, if we want to vary the function - instead of the data:: - - m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) - assert tuple(m) == (6, 9, 3**(1/2)) - - The pythonic alternative here is to use the comprehension notation, - which can already do this:: - - m = (f(3) for f in [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) - assert tuple(m) == (6, 9, 3**(1/2)) - - Inspiration: - - *Function application with $* in - http://learnyouahaskell.com/higher-order-functions - """ - def applyfrozenargsto(f): - return maybe_force_args(force(f), *args, **kwargs) - return applyfrozenargsto def pack(*args): """Multi-argument constructor for tuples. @@ -359,6 +180,7 @@ def setattrrec(object, name, value): o = getattr(o, name) setattr(o, name, value) +# TODO: move `Popper` to `unpythonic.it`? class Popper: """Pop-while iterator. @@ -427,6 +249,7 @@ def __next__(self): return self._pop() raise StopIteration +# TODO: move `CountingIterator` to `unpythonic.it`? class CountingIterator: """Iterator that counts how many elements it has yielded. diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 9717f514..873c2eca 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -10,10 +10,10 @@ from collections import namedtuple from .arity import arity_includes, UnknownArity -from .collections import Values from .dynassign import dyn from .env import env from .fun import curry, iscurried +from .funutil import Values from .lazyutil import force1, force, maybe_force_args, passthrough_lazy_args from .symbol import sym diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index b4a5c3e8..f435cb41 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -325,7 +325,7 @@ def result(): Because names inside a ``def`` have mutually recursive scope, an almost equivalent pure Python solution (no macros) is:: - from unpythonic.misc import call + from unpythonic import call @call def result(): diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 31ffb2a7..7f9a31f0 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -62,7 +62,7 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 - How to pass named args:: - from unpythonic.misc import call + from unpythonic import call with prefix: (f, kw(myarg=3)) # ``kw(...)`` (syntax, not really a function!) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 8a8fca90..b6ca5b82 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -34,9 +34,9 @@ suggest_decorator_index, UnpythonicASTMarker, ExpandedContinuationsMarker) -from ..collections import Values from ..dynassign import dyn from ..fun import identity +from ..funutil import Values from ..it import uniqify from ..lazyutil import force1, passthrough_lazy_args from ..tco import trampolined, jump diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index e39907b8..227326eb 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -6,11 +6,11 @@ from ...syntax import macros, continuations, call_cc, multilambda, autoreturn, autocurry, let # noqa: F401, F811 -from ...collections import Values from ...ec import call_ec from ...fploop import looped -from ...tco import trampolined, jump from ...fun import withself +from ...funutil import Values +from ...tco import trampolined, jump def runtests(): with testset("basic usage"): diff --git a/unpythonic/syntax/tests/test_dbg.py b/unpythonic/syntax/tests/test_dbg.py index be1e5869..1e81d9f4 100644 --- a/unpythonic/syntax/tests/test_dbg.py +++ b/unpythonic/syntax/tests/test_dbg.py @@ -9,7 +9,7 @@ from ...syntax import dbgprint_block from ...dynassign import dyn -from ...misc import call +from ...funutil import call def runtests(): # some usage examples diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index d4569ad7..d4f26c7f 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -10,14 +10,14 @@ autocurry, continuations, call_cc) -from ...collections import frozendict, Values +from ...collections import frozendict from ...ec import call_ec from ...excutil import raisef from ...fun import (curry, memoize, flip, rotate, apply, notf, andf, orf, tokth, withself) +from ...funutil import call, callwith, Values from ...it import flatten from ...llist import ll -from ...misc import call, callwith from ...seq import pipe1, piped1, lazy_piped1, pipe, pipec, piped, lazy_piped, exitpipe from ...tco import trampolined, jump diff --git a/unpythonic/syntax/tests/test_tco.py b/unpythonic/syntax/tests/test_tco.py index 02a1a106..49f95957 100644 --- a/unpythonic/syntax/tests/test_tco.py +++ b/unpythonic/syntax/tests/test_tco.py @@ -7,10 +7,10 @@ from ...syntax import (macros, tco, autoreturn, autocurry, do, let, letseq, dletrec, # noqa: F401, F811 quicklambda, f, continuations, call_cc) -from ...collections import Values from ...ec import call_ec from ...fploop import looped_over from ...fun import withself, curry +from ...funutil import Values def runtests(): # - any explicit return statement in a function body is TCO'd diff --git a/unpythonic/tests/test_fploop.py b/unpythonic/tests/test_fploop.py index 5dd2b3f5..ac1ca062 100644 --- a/unpythonic/tests/test_fploop.py +++ b/unpythonic/tests/test_fploop.py @@ -6,10 +6,11 @@ from ..fploop import looped, looped_over, breakably_looped, breakably_looped_over from ..tco import trampolined, jump +from ..ec import catch, throw +from ..funutil import call from ..let import let +from ..misc import timer from ..seq import begin -from ..misc import call, timer -from ..ec import catch, throw def runtests(): with testset("basic usage"): diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index fbfd76af..5b2ae968 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -6,7 +6,6 @@ from collections import Counter import sys -from ..collections import Values from ..dispatch import generic from ..fun import (memoize, partial, curry, apply, identity, const, @@ -16,6 +15,7 @@ composelc, composerc, to1st, to2nd, tokth, tolast, to, withself) +from ..funutil import Values from ..dynassign import dyn diff --git a/unpythonic/tests/test_funutil.py b/unpythonic/tests/test_funutil.py new file mode 100644 index 00000000..067e622d --- /dev/null +++ b/unpythonic/tests/test_funutil.py @@ -0,0 +1,99 @@ +# -*- coding: utf-8 -*- + +from ..syntax import macros, test, the # noqa: F401 +from ..test.fixtures import session, testset + +from operator import add +from functools import partial + +# `Values` is tested where function composition utilities that use it are; the class itself is trivial. +from ..funutil import call, callwith + +def runtests(): + with testset("@call (def as code block)"): + # def as a code block (function overwritten by return value) + @call + def result(): + return "hello" + test[result == "hello"] + + # use case 1: make temporaries fall out of scope + @call + def x(): + a = 2 # many temporaries that help readability... + b = 3 # ...of this calculation, but would just pollute locals... + c = 5 # ...after the block exits + return a * b * c + test[x == 30] + + # use case 2: multi-break out of nested loops + @call + def result(): + for x in range(10): + for y in range(10): + if x * y == 42: + return (x, y) + ... # more code here # pragma: no cover + test[result == (6, 7)] + + # can also be used normally + test[the[call(add, 2, 3)] == the[add(2, 3)]] + + with testset("@callwith (argument freezer), and pythonic solutions to avoid it"): + # to pass arguments when used as decorator, use @callwith instead + @callwith(3) + def result(x): + return x**2 + test[result == 9] + + # specialize for given arguments, choose function later + apply23 = callwith(2, 3) + def myadd(a, b): + return a + b + def mymul(a, b): + return a * b + test[apply23(myadd) == 5] + test[apply23(mymul) == 6] + + # callwith is not essential; we can do the same pythonically like this: + a = [2, 3] + test[myadd(*a) == 5] + test[mymul(*a) == 6] + + # build up the argument list as we go + # - note curry does not help, must use partial; this is because curry + # will happily call "callwith" (and thus terminate the gathering step) + # as soon as it gets at least one argument. + p1 = partial(callwith, 2) + p2 = partial(p1, 3) + p3 = partial(p2, 4) + apply234 = p3() # terminate gathering step by actually calling callwith + def add3(a, b, c): + return a + b + c + def mul3(a, b, c): + return a * b * c + test[apply234(add3) == 9] + test[apply234(mul3) == 24] + + # pythonic solution: + a = [2] + a += [3] + a += [4] + test[add3(*a) == 9] + test[mul3(*a) == 24] + + # callwith in map, if we want to vary the function instead of the data + m = map(callwith(3), [lambda x: 2 * x, + lambda x: x**2, + lambda x: x**(1 / 2)]) + test[tuple(m) == (6, 9, 3**(1 / 2))] + + # pythonic solution - use comprehension notation: + m = (f(3) for f in [lambda x: 2 * x, + lambda x: x**2, + lambda x: x**(1 / 2)]) + test[tuple(m) == (6, 9, 3**(1 / 2))] + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() diff --git a/unpythonic/tests/test_gmemo.py b/unpythonic/tests/test_gmemo.py index 63c1a9ea..a0e05627 100644 --- a/unpythonic/tests/test_gmemo.py +++ b/unpythonic/tests/test_gmemo.py @@ -10,7 +10,8 @@ from ..it import take, drop, last from ..fold import prod -from ..misc import call, timer +from ..funutil import call +from ..misc import timer def runtests(): with testset("multiple instances, interleaved"): diff --git a/unpythonic/tests/test_it.py b/unpythonic/tests/test_it.py index 1617ba46..ffde8b12 100644 --- a/unpythonic/tests/test_it.py +++ b/unpythonic/tests/test_it.py @@ -30,8 +30,8 @@ subset, powerset, allsame) -from ..collections import Values from ..fun import composel, identity, curry +from ..funutil import Values from ..gmemo import imemoize, gmemoize from ..mathseq import s from ..misc import Popper diff --git a/unpythonic/tests/test_let.py b/unpythonic/tests/test_let.py index 496f95d3..142c6d99 100644 --- a/unpythonic/tests/test_let.py +++ b/unpythonic/tests/test_let.py @@ -6,7 +6,7 @@ from ..let import let, letrec, dlet, dletrec, blet, bletrec from ..env import env as _envcls -from ..misc import call +from ..funutil import call from ..seq import begin def runtests(): diff --git a/unpythonic/tests/test_misc.py b/unpythonic/tests/test_misc.py index 9b70bd39..3002c21c 100644 --- a/unpythonic/tests/test_misc.py +++ b/unpythonic/tests/test_misc.py @@ -1,15 +1,12 @@ # -*- coding: utf-8 -*- -from ..syntax import macros, test, test_raises, error, warn, the # noqa: F401 +from ..syntax import macros, test, test_raises, the # noqa: F401 from ..test.fixtures import session, testset -from operator import add -from functools import partial from collections import deque from queue import Queue -from ..misc import (call, callwith, - pack, +from ..misc import (pack, namelambda, timer, getattrrec, setattrrec, @@ -20,86 +17,6 @@ from ..fun import withself def runtests(): - with testset("@call (def as code block)"): - # def as a code block (function overwritten by return value) - @call - def result(): - return "hello" - test[result == "hello"] - - # use case 1: make temporaries fall out of scope - @call - def x(): - a = 2 # many temporaries that help readability... - b = 3 # ...of this calculation, but would just pollute locals... - c = 5 # ...after the block exits - return a * b * c - test[x == 30] - - # use case 2: multi-break out of nested loops - @call - def result(): - for x in range(10): - for y in range(10): - if x * y == 42: - return (x, y) - ... # more code here # pragma: no cover - test[result == (6, 7)] - - # can also be used normally - test[the[call(add, 2, 3)] == the[add(2, 3)]] - - with testset("@callwith (argument freezer), and pythonic solutions to avoid it"): - # to pass arguments when used as decorator, use @callwith instead - @callwith(3) - def result(x): - return x**2 - test[result == 9] - - # specialize for given arguments, choose function later - apply23 = callwith(2, 3) - def myadd(a, b): - return a + b - def mymul(a, b): - return a * b - test[apply23(myadd) == 5] - test[apply23(mymul) == 6] - - # callwith is not essential; we can do the same pythonically like this: - a = [2, 3] - test[myadd(*a) == 5] - test[mymul(*a) == 6] - - # build up the argument list as we go - # - note curry does not help, must use partial; this is because curry - # will happily call "callwith" (and thus terminate the gathering step) - # as soon as it gets at least one argument. - p1 = partial(callwith, 2) - p2 = partial(p1, 3) - p3 = partial(p2, 4) - apply234 = p3() # terminate gathering step by actually calling callwith - def add3(a, b, c): - return a + b + c - def mul3(a, b, c): - return a * b * c - test[apply234(add3) == 9] - test[apply234(mul3) == 24] - - # pythonic solution: - a = [2] - a += [3] - a += [4] - test[add3(*a) == 9] - test[mul3(*a) == 24] - - # callwith in map, if we want to vary the function instead of the data - m = map(callwith(3), [lambda x: 2 * x, lambda x: x**2, lambda x: x**(1 / 2)]) - test[tuple(m) == (6, 9, 3**(1 / 2))] - - # pythonic solution - use comprehension notation: - m = (f(3) for f in [lambda x: 2 * x, lambda x: x**2, lambda x: x**(1 / 2)]) - test[tuple(m) == (6, 9, 3**(1 / 2))] - with testset("pack"): myzip = lambda lol: map(pack, *lol) lol = ((1, 2), (3, 4), (5, 6)) diff --git a/unpythonic/tests/test_seq.py b/unpythonic/tests/test_seq.py index dbc05c5c..03ea273a 100644 --- a/unpythonic/tests/test_seq.py +++ b/unpythonic/tests/test_seq.py @@ -3,7 +3,7 @@ from ..syntax import macros, test, test_raises, fail # noqa: F401 from ..test.fixtures import session, testset -from ..collections import Values +from ..funutil import Values from ..seq import (begin, begin0, lazy_begin, lazy_begin0, pipe1, pipe, pipec, piped1, piped, exitpipe, From 88799c4d8fa1a7025b55fcd3c183cc22f11f4d09 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:26:08 +0300 Subject: [PATCH 368/832] improve comments --- unpythonic/__init__.py | 6 +++++- unpythonic/lazyutil.py | 1 - 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index a1e936f9..d3ed076f 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -28,7 +28,7 @@ from .it import * # noqa: F401, F403 from .let import * # no guarantees on evaluation order (before Python 3.6), nice syntax # noqa: F401, F403 -# code generation target API for macros +# As of 0.15.0, lispylet is nowadays primarily a code generation target API for macros. from .lispylet import (let as ordered_let, letrec as ordered_letrec, # noqa: F401 dlet as ordered_dlet, dletrec as ordered_dletrec, blet as ordered_blet, bletrec as ordered_bletrec) @@ -44,10 +44,14 @@ from .tco import * # noqa: F401, F403 from .typecheck import * # noqa: F401, F403 +# -------------------------------------------------------------------------------- # HACK: break dependency loops for circular imports + from .lazyutil import _init_module _init_module() del _init_module +# We're slightly selective here, because user code likely doesn't need `islazy`, `passthrough_lazy_args`, +# or `maybe_force_args`, although strictly speaking those functions are part of the public API. from .lazyutil import Lazy, force1, force # noqa: F401 from .funutil import _init_module diff --git a/unpythonic/lazyutil.py b/unpythonic/lazyutil.py index 159f51cf..7058798c 100644 --- a/unpythonic/lazyutil.py +++ b/unpythonic/lazyutil.py @@ -13,7 +13,6 @@ from .symbol import sym # HACK: break dependency loop llist -> fun -> lazyutil -> collections -> llist -#from .collections import mogrify _init_done = False jump = sym("jump") # doesn't matter what the value is, will be overwritten later def _init_module(): # called by unpythonic.__init__ when otherwise done From 2596aa3682f9678a08ae20065fa40b763fa7c6b2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:26:34 +0300 Subject: [PATCH 369/832] update TODOs --- unpythonic/seq.py | 1 - 1 file changed, 1 deletion(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 873c2eca..e707cf8c 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,7 +99,6 @@ def lazy_begin0(*bodys): body() return out -# TODO: move `call`, `callwith`, `Values` into a `funcutils.py`? # TODO: test multiple-return-values support in all function composition utilities (`curry`, `compose` family, `pipe` family) # TODO: expand tests of `continuations` to cases with named return values # TODO: update code examples in docs From d3e3712506b1c87a8d9fbb9f5beca9c9fcae6640 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:30:44 +0300 Subject: [PATCH 370/832] update changelog --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b0bf990..075a418a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -174,8 +174,9 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - This change fixes a `flake8` [E741](https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes) warning, and the new name for the parameter is more descriptive. - **Miscellaneous.** - - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - The functions `raisef`, `tryf`, `equip_with_traceback`, and `async_raise` now live in `unpythonic.excutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + - The functions `call` and `callwith` now live in `unpythonic.funutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - Remove the internal utility class `unpythonic.syntax.util.ASTMarker`. We now have `mcpyrate.markers.ASTMarker`, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time. - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From 3b7c517ac14e5a3697bc6369e9395a76479b9594 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:33:26 +0300 Subject: [PATCH 371/832] update changelog note on upgraded curry --- CHANGELOG.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 075a418a..551e82d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -114,9 +114,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Miscellaneous.** - Resolve issue [#61](https://github.com/Technologicat/unpythonic/issues/61): `curry` now supports kwargs properly. - We now analyze parameter bindings like Python itself does, so it should no longer matter whether arguments are passed by position or by name. - - Positional passthrough works as before. - - Now any remaining named arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result. - - However, passing named arguments to an outer curry context is not supported, because the clean solution for that requires support for named return values (see issue [#32](https://github.com/Technologicat/unpythonic/issues/32)). + - Positional passthrough works as before. Named passthrough added. + - Any remaining arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result. - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). From ee71d415dfd5cd37b4e337697e9cd04a35ba0ead Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:41:51 +0300 Subject: [PATCH 372/832] improve curry explanation --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 551e82d6..d04f0436 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -115,7 +115,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Resolve issue [#61](https://github.com/Technologicat/unpythonic/issues/61): `curry` now supports kwargs properly. - We now analyze parameter bindings like Python itself does, so it should no longer matter whether arguments are passed by position or by name. - Positional passthrough works as before. Named passthrough added. - - Any remaining arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result. + - Any remaining arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result (if any), and then outward on the curry context stack as a `Values`. Since `curry` in this role is essentially a function-composition utility, the receiving curried function instance unpacks the `Values` into args and kwargs. + - If any extra arguments (positional or named) remain when the top-level curry context exits, then by default, `TypeError` is raised. To override, use `with dyn.let(curry_context=["whatever"])`, just like before. Then you'll get a `Values` object. - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). From 089a26322846276d64468bbdda5944d75034fcd0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:45:57 +0300 Subject: [PATCH 373/832] improve docstrings/comments --- unpythonic/fun.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 89815076..14f20c36 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -86,7 +86,10 @@ def memoized(*args, **kwargs): # because there are other places in the stdlib, particularly `inspect._signature_get_partial` # (as of Python 3.8), that expect the standard semantics. def partial(func, *args, **kwargs): - """Wrapper over `functools.partial` that type-checks the arguments against the type annotations on `func`. + """Type-checking `functools.partial`. + + This is a wrapper that type-checks the arguments against the type annotations + on `func`, and if the type check passes, calls `functools.partial`. Arguments can be passed by position or by name; we compute their bindings to function parameters like Python itself does. @@ -97,10 +100,10 @@ def partial(func, *args, **kwargs): Trying to pass an argument of a type that does not match the corresponding parameter's type specification raises `TypeError` immediately. - Any parameter that does not have a type annotation will not be type-checked. + Any parameter that does not have a type annotation will be ignored in the type check. Note the check still occurs at run time, but at the use site of `partial`, - when the partially applied function is constructed. This makes it fail-faster + when the partially applied function is constructed. This makes it fail-fast-er than an `isinstance` check inside the function. To conveniently make regular calls of the function type-check arguments, too, @@ -108,7 +111,7 @@ def partial(func, *args, **kwargs): """ # HACK: As of Python 3.8, `typing.get_type_hints` does not know about `functools.partial` objects, # HACK: but those objects have `args` and `keywords` attributes, so we can extract what we need. - # TODO: Remove this hack if `typing.get_type_hints` gets support for `functools.partial` at some point. + # TODO: Maybe remove this hack if `typing.get_type_hints` gets support for `functools.partial` at some point. if isinstance(func, functools_partial): thecallable = func.func collected_args = func.args + args From 1f10e769276594077e3abec8bdd6968643e6332b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:54:42 +0300 Subject: [PATCH 374/832] improve docstrings --- unpythonic/fun.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 14f20c36..e1c0c70b 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -623,6 +623,16 @@ def identity(*args, **kwargs): assert identity(1, 2, 3) == Values(1, 2, 3) assert identity(42) == 42 assert identity() is None + + **CAUTION**: Not lazy. In code using `with lazify`, all arguments + to `identity` will be forced. This is due to two reasons: + + 1. `identity` is the default continuation in `with continuations`, + producing the final return value in a continuation-enabled + computation. + + 2. `identity` just returns its arguments. Return values are + never implicitly lazy in `unpythonic`. """ if not args and not kwargs: return None @@ -649,6 +659,11 @@ def const(*args, **kwargs): c = const() assert c("anything") is None + + **CAUTION**: Not lazy. In code using `with lazify`, all arguments + to `const` will be forced. This is because the function returned + by `const` just returns the arguments that were supplied to `const`; + return values are never implicitly lazy in `unpythonic`. """ if not args and not kwargs: ret = None From 23c78827fd9c12ea8d111f73202713bf008c1fd3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:56:55 +0300 Subject: [PATCH 375/832] update comment --- unpythonic/tests/test_fun.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 5b2ae968..99427861 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -64,7 +64,8 @@ def memotuple(*args): test[memotuple((1, 2, 3)) is memotuple((1, 2, 3))] test[memotuple((1, 2, 3)) is not memotuple((1, 2))] - # "memoize lambda": classic evaluate-at-most-once thunk + # "memoize lambda": classic evaluate-at-most-once thunk. + # See also the `lazy[]` macro. thunk = memoize(lambda: print("hi from thunk")) thunk() thunk() From e36501f8089fb65189fdae5cf7c2b7e572bc1387 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 01:58:22 +0300 Subject: [PATCH 376/832] oops, add forgotten test cases --- unpythonic/tests/test_fun.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 99427861..799cf794 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -138,6 +138,9 @@ def t(): with testset("partial (type-checking wrapper)"): def nottypedfunc(x): return "ok" + test[returns_normally(partial(nottypedfunc, 42))] + test[returns_normally(partial(nottypedfunc, "abc"))] + def typedfunc(x: int): return "ok" test[returns_normally(partial(typedfunc, 42))] From 9e582c8e182de5e57a6aae5572c820c8df4cf466 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:02:24 +0300 Subject: [PATCH 377/832] add some useful the[] --- unpythonic/tests/test_fun.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 799cf794..ce26e79f 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -318,7 +318,7 @@ def f2(a, b): return a + b f1_then_f2_a = composelc(f1, f2) f1_then_f2_b = composerc(f2, f1) - test[f1_then_f2_a(2, 3) == f1_then_f2_b(2, 3) == 13] + test[the[f1_then_f2_a(2, 3)] == the[f1_then_f2_b(2, 3)] == 13] def f3(a, b): return Values(a, b) From 41c7238f2fe829ffd16b9042ffc48e775d63dbad Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:06:40 +0300 Subject: [PATCH 378/832] add `Values` tests for `composel`, remove done TODO. --- unpythonic/seq.py | 1 - unpythonic/tests/test_fun.py | 11 +++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index e707cf8c..daa83ccb 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,7 +99,6 @@ def lazy_begin0(*bodys): body() return out -# TODO: test multiple-return-values support in all function composition utilities (`curry`, `compose` family, `pipe` family) # TODO: expand tests of `continuations` to cases with named return values # TODO: update code examples in docs diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index ce26e79f..4eaa7650 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -311,6 +311,17 @@ def mymul_typed(y: int): test[inc2_then_double(3) == 10] test[double_then_inc2(3) == 8] + with testset("compose with multiple-return-values, named return values"): + f = lambda x, y: Values(2 * x, 3 * y) + g = lambda x, y: Values(x + 2, y + 3) + f_then_g = composel(f, g) + test[f_then_g(1, 2) == Values(4, 9)] + + f = lambda x, y: Values(x=2 * x, y=3 * y) + g = lambda x, y: Values(x=x + 2, y=y + 3) + f_then_g = composel(f, g) + test[f_then_g(1, 2) == Values(x=4, y=9)] + with testset("curry in compose chain"): def f1(a, b): return Values(2 * a, 3 * b) From 1498646d243047717517cfb3df67bfb474e7f5e3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:08:50 +0300 Subject: [PATCH 379/832] update comment --- unpythonic/syntax/tailtools.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index b6ca5b82..fd5d58ec 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -975,8 +975,7 @@ def make_continuation(owner, callcc, contbody): else: posargdefaults = [] - # Name the continuation: f_cont, f_cont1, f_cont2, ... - # if multiple call_cc[]s in the same function body. + # Name the continuation: f_cont_UUID if owner: # TODO: robustness: use regexes, strip suf and any numbers at the end, until no match. # return prefix of s before the first occurrence of suf. From e818fd83d34017dc3a22f5d5b70e14eeba605e16 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:37:21 +0300 Subject: [PATCH 380/832] update comments --- unpythonic/syntax/tailtools.py | 9 +++++++-- unpythonic/syntax/tests/test_conts.py | 7 +++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index fd5d58ec..981903dd 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -911,9 +911,13 @@ def split_at_callcc(body): before.append(stmt) if not after: return before, None, [] + # TODO: To support named return values (`kwrets` in a `Values` object) from the `call_cc`'d function, + # TODO: we need to change the syntax to something that allows us to specify which names are meant to + # TODO: capture the positional return values, and which ones the named return values. Doing so will + # TODO: likely break IDE support, because there's no standard name-binding construct we could abuse here. def analyze_callcc(stmt): starget = None # "starget" = starred target, becomes the vararg for the cont - def maybe_starred(expr): # return expr.id or set starget + def maybe_starred(expr): # return [expr.id] or set starget nonlocal starget if type(expr) is Name: return [expr.id] @@ -930,7 +934,8 @@ def maybe_starred(expr): # return expr.id or set starget target = stmt.targets[0] if type(target) in (Tuple, List): rest, last = target.elts[:-1], target.elts[-1] - # TODO: limitation due to Python's vararg syntax - the "*args" must be after positional args. + # TODO: limitation due to Python's vararg syntax - the "*args" must be after positional args + # TODO: in a function definition (we're going to define the cont using these). if any(type(x) is Starred for x in rest): raise SyntaxError("in call_cc[], only the last assignment target may be starred") # pragma: no cover if not all(type(x) is Name for x in rest): diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 227326eb..7852b952 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -34,6 +34,9 @@ def baz(): # The cc arg must be declared as the last one that has no default value, # or declared as by-name-only. It's always passed by name. + # + # If the function is going to be used as a target for `call_cc[]`, + # multiple return values must be packed into a `Values`. def f(a, b, cc): return Values(2 * a, 3 * b) test[f(3, 4) == Values(6, 12)] @@ -41,11 +44,15 @@ def f(a, b, cc): test[x == 6 and y == 12] def g(a, b): + # `f` packs its multiple return values into a `Values`, + # so we can use an unpacking assignment to extract them. x, y = call_cc[f(a, b)] return x, y fail["This line should not be reached."] # pragma: no cover test[g(3, 4) == (6, 12)] + # Unpacking into a star-target (as the last target) sends any + # remaining positional return values there, as a tuple. xs, *a = call_cc[f(1, 2)] test[xs == 2 and a == (6,)] From 0112df90b14d87172458a555bc0e95a924537fb3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:48:28 +0300 Subject: [PATCH 381/832] add test for named return values for `with continuations` --- unpythonic/seq.py | 1 - unpythonic/syntax/tests/test_conts.py | 15 +++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index daa83ccb..1148e82c 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,7 +99,6 @@ def lazy_begin0(*bodys): body() return out -# TODO: expand tests of `continuations` to cases with named return values # TODO: update code examples in docs # sequence one-input, one-output functions diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 7852b952..34238797 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -236,6 +236,21 @@ def doit(): test[k('again') == ['the call returned', 'again']] test[k('thrice', '!') == ['the call returned', 'thrice', '!']] + with testset("integration with named return values"): + # Named return values aren't supported as assignment targets in a `call_cc[]` + # due to syntactic limitations. But they can be used elsewhere in continuation-enabled code. + with continuations: + def f1(x, y): + return Values(x=x, y=y) # named return values + def f2(*, x, y): # note keyword-only parameters + return x, y # one return value, a tuple (for multiple-return-values, use `Values(...)`) + # Think through carefully what this does: call `f1`, chain to `f2` as the continuation. + # The continuation is set here by explicitly providing a value for the implicit `cc` parameter. + # + # The named return values from `f1` are then unpacked, by the continuation machinery, + # into the kwargs of `f2`. Then `f2` takes those, and returns a tuple. + test[f1(2, 3, cc=f2) == (2, 3)] + with testset("top level call_cc"): # A top-level "call_cc" is also allowed. # From c2567dcf5f2d9c7cfc835aedc9e1f8af63591e77 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:49:01 +0300 Subject: [PATCH 382/832] `call` now lives in `funutil` --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 808a2ee8..61f7a347 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1256,7 +1256,7 @@ call_cc[f(...) if p else g(...)] *NOTE*: ``*xs`` may need to be written as ``*xs,`` in order to explicitly make the LHS into a tuple. The variant without the comma seems to work when run from a ``.py`` file with the `macropython` bootstrapper from [`mcpyrate`](https://pypi.org/project/mcpyrate/), but fails in code run interactively in the `mcpyrate` REPL. -*NOTE*: ``f()`` and ``g()`` must be **literal function calls**. Sneaky trickery (such as calling indirectly via ``unpythonic.misc.call`` or ``unpythonic.fun.curry``) is not supported. (The ``prefix`` and ``curry`` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the ``call_cc[]`` needs to patch the ``cc=...`` kwarg of the call being made. +*NOTE*: ``f()`` and ``g()`` must be **literal function calls**. Sneaky trickery (such as calling indirectly via ``unpythonic.funutil.call`` or ``unpythonic.fun.curry``) is not supported. (The ``prefix`` and ``curry`` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the ``call_cc[]`` needs to patch the ``cc=...`` kwarg of the call being made. **Assignment targets**: From bcaa4a377eba8276db369e6e104400150b771b66 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 02:55:59 +0300 Subject: [PATCH 383/832] wording --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index b8dc31b7..0f81088e 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1176,7 +1176,7 @@ it means the following. Let ``m1`` and ``m2`` be the minimum and maximum positio - If ``n < m1``, partially apply ``f`` to the given arguments, yielding a new function with smaller ``m1``, ``m2``. Then curry the result and return it. - Internally we stack ``functools.partial`` applications, but there will be only one ``curried`` wrapper no matter how many invocations are used to build up arguments before ``f`` eventually gets called. -As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the presence of kwargs, `@generic` functions, and `Values` multiple-return-values, is: +As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the presence of kwargs, `@generic` functions, and `Values` multiple-return-values (and named return values), is: - If `f` is **not** `@generic` or `@typed`: - Compute parameter bindings of the args and kwargs collected so far, against the call signature of `f`. From 9e01cb1b1c2fe680dda3bbd25cf38ea9f099fde0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 03:00:15 +0300 Subject: [PATCH 384/832] remove last TODO marker, now tracked in issue #32 --- unpythonic/seq.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 1148e82c..d2b79cb2 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -99,8 +99,6 @@ def lazy_begin0(*bodys): body() return out -# TODO: update code examples in docs - # sequence one-input, one-output functions @passthrough_lazy_args def pipe1(value0, *bodys): From d45e49b592e5deef57f792deb79788bf32631437 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 May 2021 03:08:10 +0300 Subject: [PATCH 385/832] update doc for `pipe` --- doc/features.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 0f81088e..470d71ae 100644 --- a/doc/features.md +++ b/doc/features.md @@ -915,6 +915,8 @@ This way, any assignments made in the ``do`` (which occur only after ``do`` gets ### ``pipe``, ``piped``, ``lazy_piped``: sequence functions +**Changed in v0.15.0.** Multiple return values and named return values, for passing on to the next function in the pipe, as well as in the final return value from the pipe, are now represented as a `Values`. + Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/). A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It's just function composition, but with an emphasis on data flow, which helps improve readability: ```python @@ -969,11 +971,15 @@ from unpythonic import lazy_piped, exitpipe fibos = [] def nextfibo(a, b): # multiple arguments allowed fibos.append(a) # store result by side effect - return (b, a + b) # new state, handed to next function in the pipe + # New state, handed to next function in the pipe. + # As of v0.15.0, use `Values(...)` to represent multiple return values. + # Positional args will be passed positionally, named ones by name. + return Values(a=b, b=a + b) p = lazy_piped(1, 1) # load initial state for _ in range(10): # set up pipeline p = p | nextfibo p | exitpipe +assert (p | exitpipe) == Values(a=89, b=144) # final state assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] ``` From ac569ebd61dc75c0b6684e039878ed952e6f0b82 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 03:31:58 +0300 Subject: [PATCH 386/832] add numerical utilities triangular, partition_int_triangular --- CHANGELOG.md | 2 + unpythonic/__init__.py | 6 +- unpythonic/it.py | 101 +------------------- unpythonic/mathseq.py | 28 +++++- unpythonic/numutil.py | 158 ++++++++++++++++++++++++++++++- unpythonic/tests/test_it.py | 48 ++-------- unpythonic/tests/test_mathseq.py | 10 +- unpythonic/tests/test_numutil.py | 51 +++++++++- 8 files changed, 256 insertions(+), 148 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d04f0436..5e16e20f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -96,6 +96,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Add `unpythonic.excutil.reraise_in` (expr form), `unpythonic.excutil.reraise` (block form): conveniently remap library exception types to application exception types. Idea from [Alexis King (2016): Four months with Haskell](https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/). - Add variants of the above for the conditions-and-restarts system: `unpythonic.conditions.resignal_in`, `unpythonic.conditions.resignal`. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an `error` remains an `error` even if re-signaling changes its type. - Add `resolve_bindings_partial`, useful for analyzing partial application. + - Add `triangular`, to generate the triangular numbers (1, 3, 6, 10, ...). + - Add `partition_int_triangular` to answer a timeless question concerning stackable plushies. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) - Python 3.8 and 3.9 support added. diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index d3ed076f..5d4a8709 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -36,7 +36,6 @@ from .llist import * # noqa: F401, F403 from .mathseq import * # noqa: F401, F403 from .misc import * # noqa: F401, F403 -from .numutil import * # noqa: F401, F403 from .seq import * # noqa: F401, F403 from .singleton import * # noqa: F401, F403 from .slicing import * # noqa: F401, F403 @@ -58,3 +57,8 @@ _init_module() del _init_module from .funutil import * # noqa: F401, F403 + +from .numutil import _init_module +_init_module() +del _init_module +from .numutil import * # noqa: F401, F403 diff --git a/unpythonic/it.py b/unpythonic/it.py index 80e4da94..a83bd825 100644 --- a/unpythonic/it.py +++ b/unpythonic/it.py @@ -23,10 +23,9 @@ "flatten", "flatten1", "flatten_in", "iterate", "iterate1", "partition", - "partition_int", "inn", "iindex", "find", "window", "chunked", - "within", "fixpoint", + "within", "interleave", "subset", "powerset", "allsame"] @@ -594,7 +593,7 @@ def partition(pred, iterable): It will eventually run out of memory storing all the odd numbers "to be read later".) - Not to be confused with `unpythonic.it.partition_int`, which partitions + Not to be confused with `unpythonic.numutil.partition_int`, which partitions a (small) positive integer to smaller integers, in all possible ways, such that those integers sum to the original one. """ @@ -602,63 +601,6 @@ def partition(pred, iterable): t1, t2 = tee(iterable) return filterfalse(pred, t1), filter(pred, t2) -def partition_int(n, lower=1, upper=None): - """Yield all ordered sequences of smaller positive integers that sum to `n`. - - `n` must be an integer >= 1. - - `lower` is an optional lower limit for each member of the sum. Each member - of the sum must be `>= lower`. - - (Most of the splits are a ravioli consisting mostly of ones, so it is much - faster to not generate such splits than to filter them out from the result. - The default value `lower=1` generates everything.) - - `upper` is, similarly, an optional upper limit; each member of the sum - must be `<= upper`. The default `None` means no upper limit (effectively, - in that case `upper=n`). - - It must hold that `1 <= lower <= upper <= n`. - - Not to be confused with `unpythonic.it.partition`, which partitions an - iterable based on a predicate. - - **CAUTION**: The number of possible partitions grows very quickly with `n`, - so in practice this is only useful for small numbers, or with a lower limit - that is not too much smaller than `n / 2`. A possible use case for this - function is to determine the number of letters to allocate for each - component of an anagram that may consist of several words. - - See: - https://en.wikipedia.org/wiki/Partition_(number_theory) - """ - # sanity check the preconditions, fail-fast - if not isinstance(n, int): - raise TypeError(f"n must be integer; got {type(n)} with value {repr(n)}") - if not isinstance(lower, int): - raise TypeError(f"lower must be integer; got {type(lower)} with value {repr(lower)}") - if upper is not None and not isinstance(upper, int): - raise TypeError(f"upper must be integer; got {type(upper)} with value {repr(upper)}") - upper = upper if upper is not None else n - if n < 1: - raise ValueError(f"n must be positive; got {n}") - if lower < 1 or upper < 1 or lower > n or upper > n or lower > upper: - raise ValueError(f"it must hold that 1 <= lower <= upper <= n; got lower={lower}, upper={upper}") - - def _partition(n): - for k in range(min(n, upper), lower - 1, -1): - m = n - k - if m == 0: - yield (k,) - else: - out = [] - for item in _partition(m): - out.append((k,) + item) - for term in out: - yield term - - return _partition(n) # instantiate the generator - def inn(x, iterable): """Contains-check (``x in iterable``) with automatic termination. @@ -839,42 +781,6 @@ def within(tol, iterable): yield b return -def fixpoint(f, x0, tol=0): - """Compute the (arithmetic) fixed point of f, starting from the initial guess x0. - - (Not to be confused with the logical fixed point with respect to the - definedness ordering.) - - The fixed point must be attractive for this to work. See the Banach - fixed point theorem. - https://en.wikipedia.org/wiki/Banach_fixed-point_theorem - - If the fixed point is attractive, and the values are represented in - floating point (hence finite precision), the computation should - eventually converge down to the last bit (barring roundoff or - catastrophic cancellation in the final few steps). Hence the default tol - of zero. - - CAUTION: an arbitrary function from ℝ to ℝ **does not** necessarily - have a fixed point. Limit cycles and chaotic behavior of `f` will cause - non-termination. Keep in mind the classic example: - https://en.wikipedia.org/wiki/Logistic_map - - Examples:: - from math import cos, sqrt - from unpythonic import fixpoint, ulp - c = fixpoint(cos, x0=1) - - # Actually "Newton's" algorithm for the square root was already known to the - # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) - def sqrt_newton(n): - def sqrt_iter(x): # has an attractive fixed point at sqrt(n) - return (x + n / x) / 2 - return fixpoint(sqrt_iter, x0=n / 2) - assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) - """ - return last(within(tol, iterate1(f, x0))) - def interleave(*iterables): """Interleave items from several iterables. Generator. @@ -1003,7 +909,8 @@ def total_num_items(ld): def allsame(iterable): """Return whether all elements of an iterable are the same. - The test uses `!=` to compare. + The test uses `!=` to compare, and short-circuits at the + first item that is different. If `iterable` is empty, the return value is `True` (like for `all`). diff --git a/unpythonic/mathseq.py b/unpythonic/mathseq.py index 8d2e041c..d92aca13 100644 --- a/unpythonic/mathseq.py +++ b/unpythonic/mathseq.py @@ -413,7 +413,7 @@ def arith(): return imathify(arith() if n is infty else take(n, arith())) elif seqtype == "geom": if isinstance(k, _symExpr) or abs(k) >= 1: - def geoimathify(): + def geom(): j = 0 while True: yield x0 * (k**j) @@ -425,12 +425,12 @@ def geoimathify(): # Note that 1/(1/3) --> 3.0 even for floats, so we don't actually # need to modify the detection algorithm to account for this. kinv = 1 / k - def geoimathify(): + def geom(): j = 0 while True: yield x0 / (kinv**j) j += 1 - return imathify(geoimathify() if n is infty else take(n, geoimathify())) + return imathify(geom() if n is infty else take(n, geom())) else: # seqtype == "power": if isinstance(k, _symExpr) or abs(k) >= 1: def power(): @@ -889,6 +889,28 @@ def fibos(): a, b = b, a + b return imathify(fibos()) +def triangular(): + """Return the triangular numbers 1, 3, 6, 10, ... as a lazy sequence. + + Etymology:: + + x + x x + x x x + x x x x + ... + """ + # We could just use Gauss's result n * (n + 1) / 2 (which can be proved by induction), + # but this algorithm is trivially correct. + def _triangular(): + s = 1 # running total + r = 2 # places in the next row of the triangle + while True: + yield s + s += r + r += 1 + return imathify(_triangular()) + # See test_gmemo.py for history. This is an FP-ized sieve of Eratosthenes. # # This version wins in speed for moderate n (1e5) on typical architectures where diff --git a/unpythonic/numutil.py b/unpythonic/numutil.py index f573f262..72df5982 100644 --- a/unpythonic/numutil.py +++ b/unpythonic/numutil.py @@ -1,11 +1,25 @@ # -*- coding: utf-8 -*- """Low-level utilities for numerics.""" -__all__ = ["almosteq", "ulp"] +__all__ = ["almosteq", "ulp", + "fixpoint", + "partition_int", "partition_int_triangular"] +from itertools import takewhile from math import floor, log2 import sys +from .it import iterate1, last, within +from .symbol import sym + +# HACK: break dependency loop mathseq -> numutil -> mathseq +_init_done = False +triangular = sym("triangular") # doesn't matter what the value is, will be overwritten later +def _init_module(): # called by unpythonic.__init__ when otherwise done + global triangular, _init_done + from .mathseq import triangular + _init_done = True + class _NoSuchType: pass @@ -65,3 +79,145 @@ def ulp(x): # Unit in the Last Place # m_min = abs. value represented by a mantissa of 1.0, with the same exponent as x has m_min = 2**floor(log2(abs(x))) return m_min * eps + + +def fixpoint(f, x0, tol=0): + """Compute the (arithmetic) fixed point of f, starting from the initial guess x0. + + (Not to be confused with the logical fixed point with respect to the + definedness ordering.) + + The fixed point must be attractive for this to work. See the Banach + fixed point theorem. + https://en.wikipedia.org/wiki/Banach_fixed-point_theorem + + If the fixed point is attractive, and the values are represented in + floating point (hence finite precision), the computation should + eventually converge down to the last bit (barring roundoff or + catastrophic cancellation in the final few steps). Hence the default tol + of zero. + + CAUTION: an arbitrary function from ℝ to ℝ **does not** necessarily + have a fixed point. Limit cycles and chaotic behavior of `f` will cause + non-termination. Keep in mind the classic example: + https://en.wikipedia.org/wiki/Logistic_map + + Examples:: + from math import cos, sqrt + from unpythonic import fixpoint, ulp + c = fixpoint(cos, x0=1) + + # Actually "Newton's" algorithm for the square root was already known to the + # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) + def sqrt_newton(n): + def sqrt_iter(x): # has an attractive fixed point at sqrt(n) + return (x + n / x) / 2 + return fixpoint(sqrt_iter, x0=n / 2) + assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) + """ + return last(within(tol, iterate1(f, x0))) + + +def partition_int(n, lower=1, upper=None): + """Yield all ordered sequences of smaller positive integers that sum to `n`. + + `n` must be an integer >= 1. + + `lower` is an optional lower limit for each member of the sum. Each member + of the sum must be `>= lower`. + + (Most of the splits are a ravioli consisting mostly of ones, so it is much + faster to not generate such splits than to filter them out from the result. + The default value `lower=1` generates everything.) + + `upper` is, similarly, an optional upper limit; each member of the sum + must be `<= upper`. The default `None` means no upper limit (effectively, + in that case `upper=n`). + + It must hold that `1 <= lower <= upper <= n`. + + Not to be confused with `unpythonic.it.partition`, which partitions an + iterable based on a predicate. + + **CAUTION**: The number of possible partitions grows very quickly with `n`, + so in practice this is only useful for small numbers, or with a lower limit + that is not too much smaller than `n / 2`. A possible use case for this + function is to determine the number of letters to allocate for each + component of an anagram that may consist of several words. + + See: + https://en.wikipedia.org/wiki/Partition_(number_theory) + """ + # sanity check the preconditions, fail-fast + if not isinstance(n, int): + raise TypeError(f"n must be integer; got {type(n)} with value {repr(n)}") + if not isinstance(lower, int): + raise TypeError(f"lower must be integer; got {type(lower)} with value {repr(lower)}") + if upper is not None and not isinstance(upper, int): + raise TypeError(f"upper must be integer; got {type(upper)} with value {repr(upper)}") + upper = upper if upper is not None else n + if n < 1: + raise ValueError(f"n must be positive; got {n}") + if lower < 1 or upper < 1 or lower > n or upper > n or lower > upper: + raise ValueError(f"it must hold that 1 <= lower <= upper <= n; got lower={lower}, upper={upper}") + + return _partition_int(n, range(min(n, upper), lower - 1, -1)) # instantiate the generator + +def partition_int_triangular(n, lower=1, upper=None): + """Like `partition_int`, but allow only triangular numbers in the result. + + Triangular numbers are 1, 3, 6, 10, ... + + This function answers the timeless question: if I have `n` stackable plushies, + what are the possible stack configurations? Example:: + + configurations = partition_int_triangular(78, lower=10) + print(frozenset(tuple(sorted(c)) for c in configurations)) + + Result:: + + frozenset({(10, 10, 10, 10, 10, 28), + (10, 10, 15, 15, 28), + (15, 21, 21, 21), + (21, 21, 36), + (78,)}) + + Here `lower` sets the minimum number of plushies to allocate for one stack. + """ + if not isinstance(n, int): + raise TypeError(f"n must be integer; got {type(n)} with value {repr(n)}") + if not isinstance(lower, int): + raise TypeError(f"lower must be integer; got {type(lower)} with value {repr(lower)}") + if upper is not None and not isinstance(upper, int): + raise TypeError(f"upper must be integer; got {type(upper)} with value {repr(upper)}") + upper = upper if upper is not None else n + if n < 1: + raise ValueError(f"n must be positive; got {n}") + if lower < 1 or upper < 1 or lower > n or upper > n or lower > upper: + raise ValueError(f"it must hold that 1 <= lower <= upper <= n; got lower={lower}, upper={upper}") + + triangulars_upto_n = takewhile(lambda m: m <= n, + triangular()) + return _partition_int(n, filter(lambda m: lower <= m <= upper, + triangulars_upto_n)) + +def _partition_int(n, components): + """Implementation for `partition_int`, `partition_triangular`. + + `n`: integer to partition. + `components`: iterable of ints; numbers that are allowed to appear + in the partitioning result. Each number `m` must + satisfy `1 <= m <= n`. + """ + # TODO: Check contracts on input? This is an internal function for now, so no validation. + components = tuple(components) + for k in components: + m = n - k + if m == 0: + yield (k,) + else: + out = [] + for item in _partition_int(m, (x for x in components if x <= m)): + out.append((k,) + item) + for term in out: + yield term diff --git a/unpythonic/tests/test_it.py b/unpythonic/tests/test_it.py index ffde8b12..7b7f0fce 100644 --- a/unpythonic/tests/test_it.py +++ b/unpythonic/tests/test_it.py @@ -7,7 +7,7 @@ from itertools import tee, count, takewhile from operator import add, itemgetter from collections import deque -from math import cos, sqrt +from math import cos from ..it import (map, mapr, rmap, zipr, rzip, map_longest, mapr_longest, rmap_longest, @@ -22,10 +22,9 @@ flatten, flatten1, flatten_in, iterate1, iterate, partition, - partition_int, inn, iindex, find, window, chunked, - within, fixpoint, + within, interleave, subset, powerset, allsame) @@ -35,7 +34,6 @@ from ..gmemo import imemoize, gmemoize from ..mathseq import s from ..misc import Popper -from ..numutil import ulp def runtests(): with testset("mapping and zipping"): @@ -343,7 +341,9 @@ def primes(): S = {"cat", "lynx", "lion", "tiger"} # unordered test[all(subset(tuple(s), S) for s in powerset(S))] - # repeated function application + # Repeated function application. + # If you want to compute arithmetic fixpoints (like we do here for testing), + # see `unpythonic.numutil.fixpoint`. with testset("iterate1, iterate"): test[last(take(100, iterate1(cos, 1.0))) == 0.7390851332151607] @@ -373,47 +373,13 @@ def g2(): yield 4 test[tuple(within(0, g2())) == (1, 2, 3, 4, 4)] - # Arithmetic fixed points. - with testset("fixpoint (arithmetic fixed points)"): - c = fixpoint(cos, x0=1) - test[the[c] == the[cos(c)]] # 0.7390851332151607 - - # Actually "Newton's" algorithm for the square root was already known to the - # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) - def sqrt_newton(n): - def sqrt_iter(x): # has an attractive fixed point at sqrt(n) - return (x + n / x) / 2 - return fixpoint(sqrt_iter, x0=n / 2) - # different algorithm, so not necessarily equal down to the last bit - # (caused by the fixpoint update becoming smaller than the ulp, so it - # stops there, even if the limit is still one ulp away). - test[abs(the[sqrt_newton(2)] - the[sqrt(2)]) <= the[ulp(1.414)]] - # partition: split an iterable according to a predicate with testset("partition"): iseven = lambda item: item % 2 == 0 test[[tuple(it) for it in partition(iseven, range(10))] == [(1, 3, 5, 7, 9), (0, 2, 4, 6, 8)]] - # partition_int: split a small positive integer, in all possible ways, into smaller integers that sum to it - with testset("partition_int"): - test[tuple(partition_int(4)) == ((4,), (3, 1), (2, 2), (2, 1, 1), (1, 3), (1, 2, 1), (1, 1, 2), (1, 1, 1, 1))] - test[tuple(partition_int(5, lower=2)) == ((5,), (3, 2), (2, 3))] - test[tuple(partition_int(5, lower=2, upper=3)) == ((3, 2), (2, 3))] - test[tuple(partition_int(10, lower=3, upper=5)) == ((5, 5), (4, 3, 3), (3, 4, 3), (3, 3, 4))] - test[all(sum(terms) == 10 for terms in partition_int(10))] - test[all(sum(terms) == 10 for terms in partition_int(10, lower=3))] - test[all(sum(terms) == 10 for terms in partition_int(10, lower=3, upper=5))] - - test_raises[TypeError, partition_int("not a number")] - test_raises[TypeError, partition_int(4, lower="not a number")] - test_raises[TypeError, partition_int(4, upper="not a number")] - test_raises[ValueError, partition_int(-3)] - test_raises[ValueError, partition_int(4, lower=-1)] - test_raises[ValueError, partition_int(4, lower=5)] - test_raises[ValueError, partition_int(4, upper=-1)] - test_raises[ValueError, partition_int(4, upper=5)] - test_raises[ValueError, partition_int(4, lower=3, upper=2)] - + # Test whether all items of an iterable are equal. + # (Short-circuits at the first item that is different.) with testset("allsame"): test[allsame(())] test[allsame((1,))] diff --git a/unpythonic/tests/test_mathseq.py b/unpythonic/tests/test_mathseq.py index e04f7305..1e5232a9 100644 --- a/unpythonic/tests/test_mathseq.py +++ b/unpythonic/tests/test_mathseq.py @@ -3,12 +3,12 @@ from ..syntax import macros, test, test_raises, error, the # noqa: F401 from ..test.fixtures import session, testset -from operator import mul +from operator import add, mul from math import exp, trunc, floor, ceil from ..mathseq import (s, imathify, gmathify, sadd, smul, spow, cauchyprod, - primes, fibonacci, + primes, fibonacci, triangular, sign, log) from ..it import take, last from ..fold import scanl @@ -359,10 +359,14 @@ def runtests(): with testset("some special sequences"): test[tuple(take(10, primes())) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29)] test[tuple(take(10, fibonacci())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55)] + test[tuple(take(10, triangular())) == (1, 3, 6, 10, 15, 21, 28, 36, 45, 55)] test[tuple(take(10, primes(optimize="speed"))) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29)] test[tuple(take(10, primes(optimize="memory"))) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29)] - test_raises[ValueError, primes(optimize="fun")] # only "speed" and "memory" modes exist + test_raises[ValueError, primes(optimize="fun")] # unfortunately only "speed" and "memory" modes exist + + triangulars = imemoize(scanl(add, 1, s(2, 3, ...))) + test[tuple(take(10, triangulars)) == tuple(take(10, triangular()))] factorials = imemoize(scanl(mul, 1, s(1, 2, ...))) # 0!, 1!, 2!, ... test[last(take(6, factorials())) == 120] diff --git a/unpythonic/tests/test_numutil.py b/unpythonic/tests/test_numutil.py index ae2f808d..34e9d32d 100644 --- a/unpythonic/tests/test_numutil.py +++ b/unpythonic/tests/test_numutil.py @@ -1,11 +1,12 @@ # -*- coding: utf-8 -*- -from ..syntax import macros, test, test_raises, error # noqa: F401 +from ..syntax import macros, test, test_raises, error, the # noqa: F401 from ..test.fixtures import session, testset +from math import cos, sqrt import sys -from ..numutil import almosteq, ulp +from ..numutil import almosteq, fixpoint, partition_int, partition_int_triangular, ulp def runtests(): with testset("ulp (unit in the last place; float utility)"): @@ -39,6 +40,52 @@ def runtests(): test[almosteq(1.0, mpf(1.0 + ulp(1.0)))] test[almosteq(mpf(1.0), 1.0 + ulp(1.0))] + # Arithmetic fixed points. + with testset("fixpoint (arithmetic fixed points)"): + c = fixpoint(cos, x0=1) + test[the[c] == the[cos(c)]] # 0.7390851332151607 + + # Actually "Newton's" algorithm for the square root was already known to the + # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) + def sqrt_newton(n): + def sqrt_iter(x): # has an attractive fixed point at sqrt(n) + return (x + n / x) / 2 + return fixpoint(sqrt_iter, x0=n / 2) + # different algorithm, so not necessarily equal down to the last bit + # (caused by the fixpoint update becoming smaller than the ulp, so it + # stops there, even if the limit is still one ulp away). + test[abs(the[sqrt_newton(2)] - the[sqrt(2)]) <= the[ulp(1.414)]] + + # partition_int: split a small positive integer, in all possible ways, into smaller integers that sum to it + with testset("partition_int"): + test[tuple(partition_int(4)) == ((4,), (3, 1), (2, 2), (2, 1, 1), (1, 3), (1, 2, 1), (1, 1, 2), (1, 1, 1, 1))] + test[tuple(partition_int(5, lower=2)) == ((5,), (3, 2), (2, 3))] + test[tuple(partition_int(5, lower=2, upper=3)) == ((3, 2), (2, 3))] + test[tuple(partition_int(10, lower=3, upper=5)) == ((5, 5), (4, 3, 3), (3, 4, 3), (3, 3, 4))] + test[all(sum(terms) == 10 for terms in partition_int(10))] + test[all(sum(terms) == 10 for terms in partition_int(10, lower=3))] + test[all(sum(terms) == 10 for terms in partition_int(10, lower=3, upper=5))] + + test_raises[TypeError, partition_int("not a number")] + test_raises[TypeError, partition_int(4, lower="not a number")] + test_raises[TypeError, partition_int(4, upper="not a number")] + test_raises[ValueError, partition_int(-3)] + test_raises[ValueError, partition_int(4, lower=-1)] + test_raises[ValueError, partition_int(4, lower=5)] + test_raises[ValueError, partition_int(4, upper=-1)] + test_raises[ValueError, partition_int(4, upper=5)] + test_raises[ValueError, partition_int(4, lower=3, upper=2)] + + # partition_int_triangular: like partition_int, but in the output, allow triangular numbers only. + # Triangular numbers are 1, 3, 6, 10, ... + with testset("partition_int_triangular"): + test[frozenset(tuple(sorted(c)) for c in partition_int_triangular(78, lower=10)) == + frozenset({(10, 10, 10, 10, 10, 28), + (10, 10, 15, 15, 28), + (15, 21, 21, 21), + (21, 21, 36), + (78,)})] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 5cb615323a792b60d06efd0fc497f70cc03da08c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 03:32:33 +0300 Subject: [PATCH 387/832] update changelog --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e16e20f..d541cdf2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -166,6 +166,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `f[]` now respects nesting: an invocation of `f[]` will not descend into another nested `f[]`. - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import, by internally querying the expander to determine the name(s) the macro `f` is currently bound to. + - For the benefit of code using the `with lazify` macro, laziness is now better respected by the `compose` family, `andf` and `orf`. The utilities themselves are marked lazy, and arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain. - Rename the `curry` macro to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. - Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) - Change parameter ordering of `unpythonic.it.window` to make it curry-friendly. Usage is now `window(n, iterable)`. @@ -178,7 +179,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - **Miscellaneous.** - The functions `raisef`, `tryf`, `equip_with_traceback`, and `async_raise` now live in `unpythonic.excutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - The functions `call` and `callwith` now live in `unpythonic.funutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - - The functions `almosteq` and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. + - The functions `almosteq`, `fixpoint`, `partition_int`, and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - Remove the internal utility class `unpythonic.syntax.util.ASTMarker`. We now have `mcpyrate.markers.ASTMarker`, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time. - Rename contribution guidelines to `CONTRIBUTING.md`, which is the modern standard name. Old name was `HACKING.md`, which was correct, but nowadays obscure. - Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life. From e2e3ba877ddb684246862afc05828c7a0b0c3f92 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 03:32:48 +0300 Subject: [PATCH 388/832] update docs on multiple-return-values handling --- doc/features.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/doc/features.md b/doc/features.md index 470d71ae..137dbeeb 100644 --- a/doc/features.md +++ b/doc/features.md @@ -915,7 +915,11 @@ This way, any assignments made in the ``do`` (which occur only after ``do`` gets ### ``pipe``, ``piped``, ``lazy_piped``: sequence functions -**Changed in v0.15.0.** Multiple return values and named return values, for passing on to the next function in the pipe, as well as in the final return value from the pipe, are now represented as a `Values`. +**Changed in v0.15.0.** *Multiple return values and named return values, for unpacking to the args and kwargs of the next function in the pipe, as well as in the final return value from the pipe, are now represented as a `Values`.* + +*The variants `pipe` and `pipec` now expect a `Values` initial value if you want to unpack it into the args and kwargs of the first function in the pipe. Otherwise, the initial value is sent as a single positional argument (notably tuples too).* + +*The variants `piped` and `lazy_piped` pack the initial arguments automatically into a `Values`.* Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/). A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It's just function composition, but with an emphasis on data flow, which helps improve readability: @@ -974,7 +978,7 @@ def nextfibo(a, b): # multiple arguments allowed # New state, handed to next function in the pipe. # As of v0.15.0, use `Values(...)` to represent multiple return values. # Positional args will be passed positionally, named ones by name. - return Values(a=b, b=a + b) + return Values(a=b, b=(a + b)) p = lazy_piped(1, 1) # load initial state for _ in range(10): # set up pipeline p = p | nextfibo @@ -985,7 +989,7 @@ assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with ``1``. The use case is one-argument functions that return one value (which may also be a tuple). -In the n-to-m versions, when a function returns a tuple, it is unpacked to the argument list of the next function in the pipe. At ``exitpipe`` time, the tuple wrapper (if any) around the final result is discarded if it contains only one item. (This allows the n-to-m versions to work also with a single value, as long as it is not a tuple.) The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as there are as many "slots" on both sides of each individual connection). +In the n-to-m versions, when a function returns a `Values`, it is unpacked to the args and kwargs of the next function in the pipe. At ``exitpipe`` time, the `Values` wrapper (if any) around the final result is discarded if it contains only one positional value. The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as the args/kwargs of each output `Values` can be accepted as input by the next function in the pipe). ## Batteries @@ -1005,7 +1009,7 @@ Things missing from the standard library. - **Changed in v0.15.0.** `unpythonic`'s multiple-dispatch system (`@generic`, `@typed`) is supported. `curry` looks for an exact match first, then a match with extra args/kwargs, and finally a partial match. If there is still no match, this implies that at least one parameter would get a binding that fails the type check. In such a case `TypeError` regarding failed multiple dispatch is raised. - **Changed in v0.15.0.** If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`. - Passthrough for args/kwargs that are incompatible with the target function's call signature (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket). - - Here *incompatible* means too many positional args, or named args that have no corresponding parameter. (Note that if the function has a `**kwargs` parameter, then all named args are considered compatible, because it absorbs anything.) + - Here *incompatible* means too many positional args, or named args that have no corresponding parameter. Note that if the function has a `**kwargs` parameter, then all named args are considered compatible, because it absorbs anything. - Multiple return values (both positional and named) are denoted using `Values` (which see). A standard return value is considered to consist of one positional return value only. - Positional args are passed through **on the right**. Any positional return values of the curried function are prepended, on the left. - If the first positional return value of an intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining args and kwargs, after merging the rest of the return values into the args and kwargs. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). @@ -1017,15 +1021,17 @@ Things missing from the standard library. - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python you have (and whether CPython or PyPy3). - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values packed into a tuple are unpacked to the argument list of the next function in the chain. - - `composelc`, `composerc`: curry each function before composing them. Useful with passthrough. - - An implicit top-level curry context is inserted around all the functions except the one that is applied last. - - `composel1`, `composer1`: 1-in-1-out chains (faster; also useful for a single value that is a tuple). + - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* + - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values, or named return values, packed into a `Values`, are unpacked to the args and kwargs of the next function in the chain. + - `composelc`, `composerc`: curry each function before composing them. This comboes well with the passthrough of extra args/kwargs in `curry`. + - An implicit top-level curry context is inserted around all the functions except the one that is applied last, to allow passthrough to the top level while applying the composed function. + - `composel1`, `composer1`: 1-in-1-out chains (faster). - suffix `i` to use with an iterable that contains the functions (`composeli`, `composeri`, `composelci`, `composerci`, `composel1i`, `composer1i`) - `withself`: essentially, the Y combinator trick as a decorator. Allows a lambda to refer to itself. - The ``self`` argument is declared explicitly, but passed implicitly (as the first positional argument), just like the ``self`` argument of a method. - `apply`: the lispy approach to starargs. Mainly useful with the ``prefix`` [macro](macros.md). - `andf`, `orf`, `notf`: compose predicates (like Racket's `conjoin`, `disjoin`, `negate`). + - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, `andf` and `orf` are now marked lazy. Arguments will be forced only when a lazy predicate in the chain actually uses them, or when an eager (not lazy) predicate is encountered in the chain.* - `flip`: reverse the order of positional arguments. - `rotate`: a cousin of `flip`. Permute the order of positional arguments in a cycle. - `to1st`, `to2nd`, `tokth`, `tolast`, `to` to help inserting 1-in-1-out functions into m-in-n-out compose chains. (Currying can eliminate the need for these.) From f5d1b3ee2602dba727032aa6df09639eb07270fa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 04:07:19 +0300 Subject: [PATCH 389/832] add tests for Values --- unpythonic/tests/test_funutil.py | 48 ++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/unpythonic/tests/test_funutil.py b/unpythonic/tests/test_funutil.py index 067e622d..c4cc4e2a 100644 --- a/unpythonic/tests/test_funutil.py +++ b/unpythonic/tests/test_funutil.py @@ -6,8 +6,8 @@ from operator import add from functools import partial -# `Values` is tested where function composition utilities that use it are; the class itself is trivial. -from ..funutil import call, callwith +# `Values` is also tested where function composition utilities that use it are. +from ..funutil import call, callwith, Values def runtests(): with testset("@call (def as code block)"): @@ -94,6 +94,50 @@ def mul3(a, b, c): lambda x: x**(1 / 2)]) test[tuple(m) == (6, 9, 3**(1 / 2))] + # The `Values` abstraction is used by various parts of `unpythonic` that + # deal with function composition; particularly `curry`, the `compose` and + # `pipe` families, and the `with continuations` macro. + with testset("Values (multiple-return-values, named return values)"): + def f(): + return Values(1, 2, 3) + result = f() + test[isinstance(result, Values)] + test[result.rets == (1, 2, 3)] + test[not result.kwrets] + test[result[0] == 1] + test[result[:-1] == (1, 2)] + a, b, c = result # if no kwrets, can be unpacked like a tuple + a, b, c = f() + + def g(): + return Values(x=3) # named return value + result = g() + test[isinstance(result, Values)] + test[not result.rets] + test[result.kwrets == {"x": 3}] # actually a `frozendict` + test["x" in result] # `in` looks in the named part + test[result["x"] == 3] + test[result.get("x", None) == 3] + test[result.get("y", None) is None] + test[tuple(result.keys()) == ("x",)] # also `values()`, `items()` + + def h(): + return Values(1, 2, x=3) + result = h() + test[isinstance(result, Values)] + test[result.rets == (1, 2)] + test[result.kwrets == {"x": 3}] + a, b = result.rets # positionals can always be unpacked explicitly + test[result[0] == 1] + test["x" in result] + test[result["x"] == 3] + + def silly_but_legal(): + return Values(42) + result = silly_but_legal() + test[result.rets[0] == 42] + test[result.ret == 42] # shorthand for single-value case + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 0f710675b68547cc8c2daa0145f26277d9958ebd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 04:07:29 +0300 Subject: [PATCH 390/832] fix example --- unpythonic/funutil.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/funutil.py b/unpythonic/funutil.py index ccdf05c8..68519e81 100644 --- a/unpythonic/funutil.py +++ b/unpythonic/funutil.py @@ -224,7 +224,7 @@ class Values: Accordingly, various parts of `unpythonic` that deal with function composition use the `Values` abstraction; particularly `curry`, and - the `compose` and `pipe` families. + the `compose` and `pipe` families, and the `with continuations` macro. **Behavior**: @@ -270,8 +270,8 @@ def g(): assert "x" in result # `in` looks in the named part assert result["x"] == 3 assert result.get("x", None) == 3 - assert result.get("y", None) == None - assert tuple(results.keys()) == ("x",) # also `values()`, `items()` + assert result.get("y", None) is None + assert tuple(result.keys()) == ("x",) # also `values()`, `items()` def h(): return Values(1, 2, x=3) From 318174a764b3cc62c49154dd30b0817a6f7461db Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 04:07:36 +0300 Subject: [PATCH 391/832] update docs --- doc/features.md | 220 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 178 insertions(+), 42 deletions(-) diff --git a/doc/features.md b/doc/features.md index 137dbeeb..1994e790 100644 --- a/doc/features.md +++ b/doc/features.md @@ -67,9 +67,15 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``async_raise``: inject an exception to another thread](#async_raise-inject-an-exception-to-another-thread) *(CPython only)* - [`reraise_in`, `reraise`: automatically convert exception types](#reraise_in-reraise-automatically-convert-exception-types) -[**Other**](#other) +[**Function call and return value tools**](#function-call-and-return-value-tools) - [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. - [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) +- [`Values`: multiple and named return values](#values-multiple-and-named-return-values) + +[**Numerical tools**](#numerical-tools) + - `almosteq`, `fixpoint`, `partition_int`, `partition_int_triangular`, `ulp`. + +[**Other**](#other) - [``callsite_filename``](#callsite-filename) - [``safeissubclass``](#safeissubclass), convenience function. - [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) @@ -78,7 +84,6 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) - [``arities``, ``kwargs``, ``resolve_bindings``: Function signature inspection utilities](#arities-kwargs-resolve_bindings-function-signature-inspection-utilities) - [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) -- [``ulp``: unit in last place](#ulp-unit-in-last-place) For many examples, see [the unit tests](unpythonic/tests/), the docstrings of the individual features, and this guide. @@ -1448,7 +1453,6 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - Can be useful for the occasional abuse of `collections.deque` as an *alist* [[1]](https://en.wikipedia.org/wiki/Association_list) [[2]](http://www.gigamonkeys.com/book/beyond-lists-other-uses-for-cons-cells.html). Use `.appendleft(...)` to add new items, and then this `find` to get the currently active association. - `running_minmax`, `minmax`: Extract both min and max in one pass over an iterable. The `running_` variant is a scan and returns a generator; the just-give-me-the-final-result variant is a fold. **Added in v0.14.2.** - *math-related*: - - `fixpoint`: arithmetic fixed-point finder (not to be confused with `fix`). **Added in v0.14.2.** - `within`: yield items from iterable until successive iterates are close enough. Useful with [Cauchy sequences](https://en.wikipedia.org/wiki/Cauchy_sequence). **Added in v0.14.2.** - `prod`: like the builtin `sum`, but compute the product. Oddly missing from the standard library. - `iterate1`, `iterate`: return an infinite generator that yields `x`, `f(x)`, `f(f(x))`, ... @@ -1464,7 +1468,6 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `slurp`: extract all items from a `queue.Queue` (until it is empty) to a list, returning that list. **Added in v0.14.2.** - `subset`: test whether an iterable is a subset of another. **Added in v0.14.3.** - `powerset`: yield the power set (set of all subsets) of an iterable. Works also for potentially infinite iterables, if only a finite prefix is ever requested. (But beware, both runtime and memory usage are exponential in the input size.) **Added in v0.14.2.** - - `partition_int`: split a small positive integer, in all possible ways, into smaller integers that sum to it. Useful e.g. for determining how many letters the components of an anagram may have. **Added in v0.14.2.** - `allsame`: test whether all elements of an iterable are the same. Sometimes useful in writing testing code. **Added in v0.14.3.** Examples: @@ -1994,10 +1997,10 @@ We provide the [Cauchy product](https://en.wikipedia.org/wiki/Cauchy_product), a We also provide ``gmathify``, a decorator to mathify a gfunc, so that it will ``imathify()`` the generator instances it makes. Combo with ``imemoize`` for great justice, e.g. ``a = gmathify(imemoize(myiterable))``, and then ``a()`` to instantiate a memoized-and-mathified copy. -Finally, we provide ready-made generators that yield some common sequences (currently, the Fibonacci numbers and the prime numbers). The prime generator is an FP-ized sieve of Eratosthenes. +Finally, we provide ready-made generators that yield some common sequences (currently, the Fibonacci numbers, the triangular numbers, and the prime numbers). The prime generator is an FP-ized sieve of Eratosthenes. ```python -from unpythonic import s, imathify, cauchyprod, take, last, fibonacci, primes +from unpythonic import s, imathify, cauchyprod, take, last, fibonacci, triangular, primes assert tuple(take(10, s(1, ...))) == (1,)*10 assert tuple(take(10, s(1, 2, ...))) == tuple(range(1, 11)) @@ -2025,6 +2028,7 @@ assert tuple(take(3, cauchyprod(s(1, 3, 5, ...), s(2, 4, 6, ...)))) == (2, 10, 2 assert tuple(take(10, primes())) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) assert tuple(take(10, fibonacci())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) +assert tuple(take(10, triangular())) == (1, 3, 6, 10, 15, 21, 28, 36, 45, 55) ``` A math iterable (i.e. one that has infix math support) is an instance of the class ``imathify``: @@ -3468,9 +3472,7 @@ Full details in docstrings. If you use the conditions-and-restarts system, see also `resignal_in`, `resignal`, which perform the same job for conditions. The new signal is sent using the same error handling protocol as the original signal, so e.g. an `error` will remain an `error` even if re-signaling changes its type. -## Other - -Stuff that didn't fit elsewhere. +## Function call and return value tools ### ``def`` as a code block: ``@call`` @@ -3661,10 +3663,177 @@ assert tuple(m) == (6, 9, 3**(1/2)) Inspired by *Function application with $* in [LYAH: Higher Order Functions](http://learnyouahaskell.com/higher-order-functions). +### `Values`: multiple and named return values + +**Added in v0.15.0.** + +`Values` is a structured multiple-return-values type. We also provide `valuify`, a decorator that converts the pythonic tuple-as-multiple-return-values idiom into `Values`. + +With `Values`, you can return multiple values positionally and by name. This completes the symmetry between passing function arguments and returning values from a function: Python itself allows passing arguments by name, but has no concept of returning values by name. This class adds that concept. + +Having a `Values` type separate from `tuple` also helps with semantic accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value now means just that - one value that is a `tuple`. It is different from a `Values` that contains several positional return values (that are meant to be treated separately e.g. by a function composition utility). + +#### When to use `Values` + +Most of the time, returning a tuple to denote multiple-return-values and unpacking it is just fine, and that is exactly what `unpythonic` does internally in many places. + +But the distinction is critically important in function composition, so that positional return values can be automatically mapped into positional arguments to the next function in the chain, and named return values into named arguments. + +Accordingly, various parts of `unpythonic` that deal with function composition use the `Values` abstraction; particularly `curry`, and the `compose` and `pipe` families, and the `with continuations` macro. + +#### Behavior + +`Values` is a duck-type with some features of both sequences and mappings, but not the full `collections.abc` API of either. + +Each operation that obviously and without ambiguity makes sense only for the positional or named part, accesses that part. + +The only exception is `__getitem__` (subscripting), which makes sense for both parts, unambiguously, because the key types differ. If the index expression is an `int` or a `slice`, it is an index/slice for the positional part. If it is an `str`, it is a key for the named part. + +If you need to explicitly access either part (and its full API), use the `rets` and `kwrets` attributes. The names are in analogy with `args` and `kwargs`. + +`rets` is a `tuple`, and `kwrets` is an `unpythonic.collections.frozendict`. + +`Values` objects can be compared for equality. Two `Values` objects are equal if both their `rets` and `kwrets` (respectively) are. + +Examples: + +```python +def f(): + return Values(1, 2, 3) +result = f() +assert isinstance(result, Values) +assert result.rets == (1, 2, 3) +assert not result.kwrets +assert result[0] == 1 +assert result[:-1] == (1, 2) +a, b, c = result # if no kwrets, can be unpacked like a tuple +a, b, c = f() + +def g(): + return Values(x=3) # named return value +result = g() +assert isinstance(result, Values) +assert not result.rets +assert result.kwrets == {"x": 3} # actually a `frozendict` +assert "x" in result # `in` looks in the named part +assert result["x"] == 3 +assert result.get("x", None) == 3 +assert result.get("y", None) is None +assert tuple(result.keys()) == ("x",) # also `values()`, `items()` + +def h(): + return Values(1, 2, x=3) +result = h() +assert isinstance(result, Values) +assert result.rets == (1, 2) +assert result.kwrets == {"x": 3} +a, b = result.rets # positionals can always be unpacked explicitly +assert result[0] == 1 +assert "x" in result +assert result["x"] == 3 + +def silly_but_legal(): + return Values(42) +result = silly_but_legal() +assert result.rets[0] == 42 +assert result.ret == 42 # shorthand for single-value case +``` + +The last example is silly, but legal, because it is preferable to just omit the `Values` if it is known that there is only one return value. (This also applies when that value is a `tuple`, when the intent is to return it as a single `tuple`, in contexts where this distinction matters.) + + +## Numerical tools + +Overview: + +- `almosteq`: test floating-point numbers for near-equality. Reverts to exact equality for non-floating-point types. + +- `fixpoint`: arithmetic fixed-point finder (not to be confused with `fix`). **Added in v0.14.2.** + +- `partition_int`: [partition](https://en.wikipedia.org/wiki/Partition_(number_theory)) a small positive integer, i.e., split it in all possible ways, into smaller integers that sum to it. Useful e.g. for determining how many letters the components of an anagram may have. **Added in v0.14.2.** + + Not to be confused with `unpythonic.partition`, which partitions an iterable based on a predicate. + +- `partition_int_triangular`: like `partition_int`, but accept only triangular numbers (1, 3, 6, 10, ...) as components of the partition. This function answers a timeless question: if I have `n` stackable plushies, what are the possible stack configurations? **Added in v0.15.0.** + +- ``ulp``: unit in last place. The numerical value of the least-significant bit of a floating-point number at a given point on the real line. **Added in v0.14.2.** + +We provide more detailed documentation on some of these below. For the rest, see the docstrings and [the unit tests](../unpythonic/tests/test_numutil.py) for discussion and examples. + + +### `fixpoint`: arithmetic fixed-point finder + +**Added in v0.14.2.** + +Compute the (arithmetic) fixed point of a function, starting from a given initial guess. The fixed point must be attractive for this to work. See the [Banach fixed point theorem](https://en.wikipedia.org/wiki/Banach_fixed-point_theorem). + +(Not to be confused with the logical fixed point with respect to the definedness ordering, which is what Haskell's `fix` function relates to.) + +If the fixed point is attractive, and the values are represented in floating point (hence finite precision), the computation should eventually converge down to the last bit (barring roundoff or catastrophic cancellation in the final few steps). Hence the default tolerance is zero; but a desired tolerance can be passed as an argument. + +**CAUTION**: an arbitrary function from ℝ to ℝ **does not** necessarily have a fixed point. Limit cycles and chaotic behavior of the function will cause non-termination. Keep in mind the classic example, [the logistic map](https://en.wikipedia.org/wiki/Logistic_map). + +Examples: + +```python +from math import cos, sqrt +from unpythonic import fixpoint, ulp + +c = fixpoint(cos, x0=1) + +# Actually "Newton's" algorithm for the square root was already known to the +# ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) +def sqrt_newton(n): + def sqrt_iter(x): # has an attractive fixed point at sqrt(n) + return (x + n / x) / 2 + return fixpoint(sqrt_iter, x0=n / 2) +assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) +``` + + +### ``ulp``: unit in last place + +**Added in v0.14.2.** + +Given a floating point number `x`, return the value of the *unit in the last place* (the "least significant bit"). This is the local size of a "tick", i.e. the difference between `x` and the next larger float. At `x = 1.0`, this is the [machine epsilon](https://en.wikipedia.org/wiki/Machine_epsilon), by definition of the machine epsilon. + +The float format is [IEEE-754](https://en.wikipedia.org/wiki/IEEE_754), i.e. standard Python `float`. + +This is just a small convenience function that is for some reason missing from the `math` standard library. + +```python +from unpythonic import ulp + +# in IEEE-754, exponent changes at integer powers of two +print([ulp(x) for x in (0.25, 0.5, 1.0, 2.0, 4.0)]) +# --> [5.551115123125783e-17, +# 1.1102230246251565e-16, +# 2.220446049250313e-16, # x = 1.0, so this is sys.float_info.epsilon +# 4.440892098500626e-16, +# 8.881784197001252e-16] +print(ulp(1e10)) +# --> 1.9073486328125e-06 +print(ulp(1e100)) +# --> 1.942668892225729e+84 +print(ulp(2**52)) +# --> 1.0 # yes, exactly 1 +``` + +When `x` is a round number in base-10, the ULP is not, because the usual kind of floats use base-2. + +For more reading, see [David Goldberg (1991): What every computer scientist should know about floating-point arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), or for a [tl;dr](http://catplanet.org/tldr-cat-meme/) version, [the floating point guide](https://floating-point-gui.de/). + + +## Other + +Stuff that didn't fit elsewhere. + ### ``callsite_filename`` **Added in v0.14.3**. +**Changed in v0.15.0.** *This utility now ignores `unpythonic`'s call helpers, and gives the filename from the deepest stack frame that does not match one of our helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`).* + Return the filename from which this function is being called. Useful as a building block for debug utilities and similar. @@ -3902,36 +4071,3 @@ The input container must support either ``popleft()`` or ``pop(0)``. This is ful Per-iteration efficiency is O(1) for ``collections.deque``, and O(n) for a ``list``. Named after [Karl Popper](https://en.wikipedia.org/wiki/Karl_Popper). - - -### ``ulp``: unit in last place - -**Added in v0.14.2.** - -Given a floating point number `x`, return the value of the *unit in the last place* (the "least significant bit"). This is the local size of a "tick", i.e. the difference between `x` and the next larger float. At `x = 1.0`, this is the [machine epsilon](https://en.wikipedia.org/wiki/Machine_epsilon), by definition of the machine epsilon. - -The float format is [IEEE-754](https://en.wikipedia.org/wiki/IEEE_754), i.e. standard Python `float`. - -This is just a small convenience function that is for some reason missing from the `math` standard library. - -```python -from unpythonic import ulp - -# in IEEE-754, exponent changes at integer powers of two -print([ulp(x) for x in (0.25, 0.5, 1.0, 2.0, 4.0)]) -# --> [5.551115123125783e-17, -# 1.1102230246251565e-16, -# 2.220446049250313e-16, # x = 1.0, so this is sys.float_info.epsilon -# 4.440892098500626e-16, -# 8.881784197001252e-16] -print(ulp(1e10)) -# --> 1.9073486328125e-06 -print(ulp(1e100)) -# --> 1.942668892225729e+84 -print(ulp(2**52)) -# --> 1.0 # yes, exactly 1 -``` - -When `x` is a round number in base-10, the ULP is not, because the usual kind of floats use base-2. - -For more reading, see [David Goldberg (1991): What every computer scientist should know about floating-point arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), or for a [tl;dr](http://catplanet.org/tldr-cat-meme/) version, [the floating point guide](https://floating-point-gui.de/). From 3cfbc8954ae0fb05fd5f2a9d046afd5fb7aaf68e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 04:09:11 +0300 Subject: [PATCH 392/832] fix borked test --- unpythonic/tests/test_mathseq.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/tests/test_mathseq.py b/unpythonic/tests/test_mathseq.py index 1e5232a9..c9328759 100644 --- a/unpythonic/tests/test_mathseq.py +++ b/unpythonic/tests/test_mathseq.py @@ -366,7 +366,7 @@ def runtests(): test_raises[ValueError, primes(optimize="fun")] # unfortunately only "speed" and "memory" modes exist triangulars = imemoize(scanl(add, 1, s(2, 3, ...))) - test[tuple(take(10, triangulars)) == tuple(take(10, triangular()))] + test[tuple(take(10, triangulars())) == tuple(take(10, triangular()))] factorials = imemoize(scanl(mul, 1, s(1, 2, ...))) # 0!, 1!, 2!, ... test[last(take(6, factorials())) == 120] From a211141e44633d634ca741b4e774ed8d6a7c361b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 04:16:07 +0300 Subject: [PATCH 393/832] add tests for valuify --- unpythonic/tests/test_funutil.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/unpythonic/tests/test_funutil.py b/unpythonic/tests/test_funutil.py index c4cc4e2a..16fe240f 100644 --- a/unpythonic/tests/test_funutil.py +++ b/unpythonic/tests/test_funutil.py @@ -7,7 +7,7 @@ from functools import partial # `Values` is also tested where function composition utilities that use it are. -from ..funutil import call, callwith, Values +from ..funutil import call, callwith, Values, valuify def runtests(): with testset("@call (def as code block)"): @@ -138,6 +138,13 @@ def silly_but_legal(): test[result.rets[0] == 42] test[result.ret == 42] # shorthand for single-value case + with testset("valuify (convert tuple as multiple-return-values into Values)"): + @valuify + def f(x, y, z): + return x, y, z + test[isinstance(f(1, 2, 3), Values)] + test[f(1, 2, 3) == Values(1, 2, 3)] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 6e426fe834ce2f2e9e302ce925cebf79f98c9d0b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 04:40:41 +0300 Subject: [PATCH 394/832] document `valuify` --- doc/features.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 1994e790..3073d9ce 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3667,7 +3667,7 @@ Inspired by *Function application with $* in [LYAH: Higher Order Functions](http **Added in v0.15.0.** -`Values` is a structured multiple-return-values type. We also provide `valuify`, a decorator that converts the pythonic tuple-as-multiple-return-values idiom into `Values`. +`Values` is a structured multiple-return-values type. With `Values`, you can return multiple values positionally and by name. This completes the symmetry between passing function arguments and returning values from a function: Python itself allows passing arguments by name, but has no concept of returning values by name. This class adds that concept. @@ -3742,6 +3742,24 @@ assert result.ret == 42 # shorthand for single-value case The last example is silly, but legal, because it is preferable to just omit the `Values` if it is known that there is only one return value. (This also applies when that value is a `tuple`, when the intent is to return it as a single `tuple`, in contexts where this distinction matters.) +### `valuify` + +We also provide `valuify`, a decorator that converts the pythonic tuple-as-multiple-return-values idiom into `Values`, for compatibility with our function composition utilities. + +It converts a `tuple` return value, exactly; no subclasses. + +Demonstrating just the conversion: + +```python +@valuify +def f(x, y, z): + return x, y, z + +assert isinstance(f(1, 2, 3), Values) +assert f(1, 2, 3) == Values(1, 2, 3) +``` + + ## Numerical tools Overview: From 9dca2bbb8ae0d383cd8287e4303f9c80c8b05133 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 13:45:21 +0300 Subject: [PATCH 395/832] advertise passthrough feature of curry in README --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c80649a0..57a6cb99 100644 --- a/README.md +++ b/README.md @@ -170,8 +170,10 @@ assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, We bind arguments to parameters like Python itself does, so it does not matter whether arguments are passed by position or by name during currying. We support `@generic` multiple-dispatch functions. +We also feature a Haskell-inspired passthrough system: any args and kwargs that are not accepted by the call signature will be passed through. This is useful when a curried function returns a new function, which is then the target for the passthrough. See the docs for details. + ```python -from unpythonic import curry, generic +from unpythonic import curry, generic, foldr, composerc, cons, nil, ll @curry def f(x, y): @@ -216,6 +218,11 @@ assert g(1.0)(2.0) == "float" assert g("cat") == "str" assert g(s="cat") == "str" + +# simple example of passthrough +mymap = lambda f: curry(foldr, composerc(cons, f), nil) +myadd = lambda a, b: a + b +assert curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9) ```
Multiple-dispatch generic functions, like in CLOS or Julia. From a7c7280ebed769f60e523fc3b61e2162ca4e7997 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 14:21:16 +0300 Subject: [PATCH 396/832] link valuify from the TOC --- doc/features.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/features.md b/doc/features.md index 3073d9ce..47bcff39 100644 --- a/doc/features.md +++ b/doc/features.md @@ -71,6 +71,7 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. - [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) - [`Values`: multiple and named return values](#values-multiple-and-named-return-values) + - [`valuify`](#valuify): convert pythonic multiple-return-values idiom of `tuple` into `Values`. [**Numerical tools**](#numerical-tools) - `almosteq`, `fixpoint`, `partition_int`, `partition_int_triangular`, `ulp`. From 70cd2b5efef1c202d2ab12afdaa9d47c151b5c2b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 14:21:25 +0300 Subject: [PATCH 397/832] document numutil module --- doc/features.md | 66 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 54 insertions(+), 12 deletions(-) diff --git a/doc/features.md b/doc/features.md index 47bcff39..e3982d2c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -74,7 +74,10 @@ The exception are the features marked **[M]**, which are primarily intended as a - [`valuify`](#valuify): convert pythonic multiple-return-values idiom of `tuple` into `Values`. [**Numerical tools**](#numerical-tools) - - `almosteq`, `fixpoint`, `partition_int`, `partition_int_triangular`, `ulp`. + - [`almosteq`: floating-point almost-equality](#almosteq-floating-point-almost-equality) + - [`fixpoint`: arithmetic fixed-point finder](#fixpoint-arithmetic-fixed-point-finder) + - [`partition_int`, `partition_int_triangular`: partition integers](#partition_int-partition_int_triangular-partition-integers) + - [``ulp``: unit in last place](#ulp-unit-in-last-place) [**Other**](#other) - [``callsite_filename``](#callsite-filename) @@ -3763,21 +3766,26 @@ assert f(1, 2, 3) == Values(1, 2, 3) ## Numerical tools -Overview: +We briefly introduce the functions below. More details and examples can be found in the docstrings and [the unit tests](../unpythonic/tests/test_numutil.py**. -- `almosteq`: test floating-point numbers for near-equality. Reverts to exact equality for non-floating-point types. +**CAUTION** for anyone new to numerics: -- `fixpoint`: arithmetic fixed-point finder (not to be confused with `fix`). **Added in v0.14.2.** +When working with floating-point numbers, keep in mind that they are, very roughly speaking, a finite-precision logarithmic representation of [ℝ](https://en.wikipedia.org/wiki/Real_line). They are, necessarily, actually a subset of [ℚ](https://en.wikipedia.org/wiki/Rational_number), that's not even [dense](https://en.wikipedia.org/wiki/Dense_set). The spacing between adjacent floats depends on where you are on the real line; see `ulp` below. -- `partition_int`: [partition](https://en.wikipedia.org/wiki/Partition_(number_theory)) a small positive integer, i.e., split it in all possible ways, into smaller integers that sum to it. Useful e.g. for determining how many letters the components of an anagram may have. **Added in v0.14.2.** +For finer points concerning the behavior of floating-point numbers, see [David Goldberg (1991): What every computer scientist should know about floating-point arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), or for a [tl;dr](http://catplanet.org/tldr-cat-meme/) version, [the floating point guide](https://floating-point-gui.de/). - Not to be confused with `unpythonic.partition`, which partitions an iterable based on a predicate. +Or you could look at [my lecture slides from 2018](https://github.com/Technologicat/python-3-scicomp-intro/tree/master/lecture_slides); particularly, [lecture 7](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/lecture_slides/lectures_tut_2018_7.pdf) covers the floating-point representation. It collects the most important details, and some more links to further reading. -- `partition_int_triangular`: like `partition_int`, but accept only triangular numbers (1, 3, 6, 10, ...) as components of the partition. This function answers a timeless question: if I have `n` stackable plushies, what are the possible stack configurations? **Added in v0.15.0.** -- ``ulp``: unit in last place. The numerical value of the least-significant bit of a floating-point number at a given point on the real line. **Added in v0.14.2.** +### `almosteq`: floating-point almost-equality -We provide more detailed documentation on some of these below. For the rest, see the docstrings and [the unit tests](../unpythonic/tests/test_numutil.py) for discussion and examples. +Test floating-point numbers for near-equality. Beside the built-in `float`, we support also the arbitrary-precision software-implemented floating-point type `mpf` from `SymPy`'s `mpmath` package. + +Anything else, for example `SymPy` expressions, strings, and containers (regardless of content), is tested for exact equality. + +For ``mpmath.mpf``, we just delegate to ``mpmath.almosteq``, with the given tolerance. + +For ``float``, we use the strategy suggested in [the floating point guide](https://floating-point-gui.de/errors/comparison/), because naive absolute and relative comparisons against a tolerance fail in commonly encountered situations. ### `fixpoint`: arithmetic fixed-point finder @@ -3810,11 +3818,47 @@ assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) ``` +### `partition_int`, `partition_int_triangular`: partition integers + +**Added in v0.14.2.** + +**Changed in v0.15.0.** *Added `partition_int_triangular`.* + +The `partition_int` function [partitions](https://en.wikipedia.org/wiki/Partition_(number_theory)) a small positive integer, i.e., splits it in all possible ways, into smaller integers that sum to it. This is useful e.g. to determine the number of letters to allocate for each component of an anagram that may consist of several words. + +The `partition_int_triangular` function is like `partition_int`, but accepts only triangular numbers (1, 3, 6, 10, ...) as components of the partition. This function answers a timeless question: if I have `n` stackable plushies, what are the possible stack configurations? + +(These are not to be confused with `unpythonic.partition`, which partitions an iterable based on a predicate.) + +Examples: + +```python +from unpythonic import partition_int, partition_int_triangular + +assert tuple(partition_int(4)) == ((4,), (3, 1), (2, 2), (2, 1, 1), (1, 3), (1, 2, 1), (1, 1, 2), (1, 1, 1, 1)) +assert tuple(partition_int(5, lower=2)) == ((5,), (3, 2), (2, 3)) +assert tuple(partition_int(5, lower=2, upper=3)) == ((3, 2), (2, 3)) + +assert (frozenset(tuple(sorted(c)) for c in partition_int_triangular(78, lower=10)) == + frozenset({(10, 10, 10, 10, 10, 28), + (10, 10, 15, 15, 28), + (15, 21, 21, 21), + (21, 21, 36), + (78,)})) +``` + +As the first example demonstrates, most of the splits are a ravioli consisting mostly of ones. It is much faster to not generate such splits than to filter them out from the result. Use the `lower` parameter to set the smallest acceptable value for one component of the split; the default value `lower=1` generates all splits. Similarly, the `upper` parameter sets the largest acceptable value for one component of the split. The default `upper=None` sets no upper limit. + +In `partition_int_triangular`, the `lower` and `upper` parameters work exactly the same. The only difference to `partition_int` is that each component of the split must be a triangular number. + +**CAUTION**: The number of possible partitions grows very quickly with `n`, so in practice these functions are only useful for small numbers, or with a lower limit that is not too much smaller than `n / 2`. + + ### ``ulp``: unit in last place **Added in v0.14.2.** -Given a floating point number `x`, return the value of the *unit in the last place* (the "least significant bit"). This is the local size of a "tick", i.e. the difference between `x` and the next larger float. At `x = 1.0`, this is the [machine epsilon](https://en.wikipedia.org/wiki/Machine_epsilon), by definition of the machine epsilon. +Given a floating point number `x`, return the value of the *unit in the last place* (the "least significant bit"). This is the local size of a "tick", i.e. the difference between `x` and the *next larger* float. At `x = 1.0`, this is the [machine epsilon](https://en.wikipedia.org/wiki/Machine_epsilon), by definition of the machine epsilon. The float format is [IEEE-754](https://en.wikipedia.org/wiki/IEEE_754), i.e. standard Python `float`. @@ -3840,8 +3884,6 @@ print(ulp(2**52)) When `x` is a round number in base-10, the ULP is not, because the usual kind of floats use base-2. -For more reading, see [David Goldberg (1991): What every computer scientist should know about floating-point arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), or for a [tl;dr](http://catplanet.org/tldr-cat-meme/) version, [the floating point guide](https://floating-point-gui.de/). - ## Other From e6859fea07418a7da4ffa2bd8a2016738cad0dd2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 May 2021 14:37:30 +0300 Subject: [PATCH 398/832] Wording for other libraries that provide some feature of unpythonic The point of having these features in `unpythonic` is integration, and a consistent API. So if you need only one specific language-extension feature, then a library that concentrates on that particular feature is likely a good choice. If you need the kitchen sink, too, then it's better to use our implementation, since our implementations of the various features are designed to work together. In some cases (e.g. the condition system), our implementation may offer extra features not present in the original library that inspired it. In other cases (e.g. multiple dispatch), the *other* implementation may be better (e.g. runs much faster). --- doc/features.md | 6 ++++-- doc/troubleshooting.md | 10 ++++++++++ unpythonic/typecheck.py | 3 ++- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/doc/features.md b/doc/features.md index e3982d2c..f3dc9d59 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3048,7 +3048,7 @@ Conditions are one of the killer features of Common Lisp, so if you're new to co For Python, conditions were first implemented in [python-cl-conditions](https://github.com/svetlyak40wt/python-cl-conditions/) by Alexander Artemenko (2016). -What we provide here is essentially a rewrite, based on studying that implementation. The main reasons for the rewrite are to give the condition system an API consistent with the style of `unpythonic`, to drop any and all historical baggage without needing to consider backward compatibility, and to allow interaction with (and customization taking into account) the other parts of `unpythonic`. If you specifically need a condition system, not a kitchen-sink language extension, then by all means go for `python-cl-conditions`! +What we provide here is essentially a rewrite, based on studying that implementation. The main reasons for the rewrite are to give the condition system an API consistent with the style of `unpythonic`, to drop any and all historical baggage without needing to consider backward compatibility, and to allow interaction with (and customization taking into account) the other parts of `unpythonic`. The core idea can be expressed in fewer than 100 lines of Python; ours is (as of v0.14.2) 151 lines, not counting docstrings, comments, or blank lines. The main reason our module is over 700 lines are the docstrings. @@ -3193,6 +3193,8 @@ The machinery itself is also missing some advanced features, such as matching th **CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If a new multimethod is added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the multiple-dispatch table is global state! +If you need multiple dispatch, but not the other features of `unpythonic`, see the [multipledispatch](https://github.com/mrocklin/multipledispatch) library, which likely runs faster. + #### ``typed``: add run-time type checks with type annotation syntax @@ -3293,7 +3295,7 @@ See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. **CAUTION**: The `isoftype` function is one big hack. In Python 3.6 through 3.9, there is no consistent way to handle a type specification at run time. We must access some private attributes of the ``typing`` meta-utilities, because that seems to be the only way to get what we need to do this. -For a similar tool for run-time type-checking, see also the [`typeguard`](https://github.com/agronholm/typeguard) library. +If you need a run-time type checker, but not the other features of `unpythonic`, see the [`typeguard`](https://github.com/agronholm/typeguard) library. ## Exception tools diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index 1693ea59..15112ad7 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -19,6 +19,7 @@ - [Cannot import the name `macros`?](#cannot-import-the-name-macros) - [But I did run my program with `macropython`?](#but-i-did-run-my-program-with-macropython) - [I'm hacking a macro inside a module in `unpythonic.syntax`, and my changes don't take?](#im-hacking-a-macro-inside-a-module-in-unpythonicsyntax-and-my-changes-dont-take) + - [Both `unpythonic` and library `x` provide language-extension feature `y`. Which is better?](#both-unpythonic-and-library-x-provide-language-extension-feature-y-which-is-better) @@ -81,3 +82,12 @@ I might modify the `mcpyrate` analyzer in the future, but doing so will make the For now, we just note that this issue mainly concerns developers of large macro packages (such as `unpythonic.syntax`) that need to split - for factoring reasons - their macro definitions into separate modules, while presenting all macros to the user in one interface module. This issue does not affect the development of macro-using programs, or any programs where macros are imported from their original definition site (like they always were with MacroPy). Try clearing the bytecode cache in `unpythonic/`; this will force a recompile. + + +### Both `unpythonic` and library `x` provide language-extension feature `y`. Which is better? + +The point of having these features in `unpythonic` is integration, and a consistent API. So if you need only one specific language-extension feature, then a library that concentrates on that particular feature is likely a good choice. If you need the kitchen sink, too, then it's better to use our implementation, since our implementations of the various features are designed to work together. + +In some cases (e.g. the condition system), our implementation may offer extra features not present in the original library that inspired it. + +In other cases (e.g. multiple dispatch), the *other* implementation may be better (e.g. runs much faster). diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 089cfd63..1e5758bc 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -8,7 +8,8 @@ We currently provide `isoftype` (cf. `isinstance`), but no `issubtype` (cf. `issubclass`). -If you need a run-time type checker for serious general use, consider `typeguard`: +If you need a run-time type checker, but not the other features of `unpythonic`, +see `typeguard`: https://github.com/agronholm/typeguard """ From 4628f19c789b4e6c0611d4f7b42c3fba545bad3f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 May 2021 00:20:32 +0300 Subject: [PATCH 399/832] presentation order --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index f3dc9d59..14512e3c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1013,6 +1013,7 @@ Things missing from the standard library. - This is essentially because ``self`` is an argument, and custom classes have a default ``__hash__``. - Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. - For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)). + - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `curry`, with some extra features: - **Changed in v0.15.0.** `curry` supports both positional and named arguments, and binds arguments to function parameters like Python itself does. The call triggers when all parameters are bound, regardless of whether they were passed by position or by name, and at which step of the currying process they were passed. - **Changed in v0.15.0.** `unpythonic`'s multiple-dispatch system (`@generic`, `@typed`) is supported. `curry` looks for an exact match first, then a match with extra args/kwargs, and finally a partial match. If there is still no match, this implies that at least one parameter would get a binding that fails the type check. In such a case `TypeError` regarding failed multiple dispatch is raised. @@ -1028,7 +1029,6 @@ Things missing from the standard library. - Can be used both as a decorator and as a regular function. - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python you have (and whether CPython or PyPy3). - - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values, or named return values, packed into a `Values`, are unpacked to the args and kwargs of the next function in the chain. From 37ef52ab3ffbde56dd259524eaef8dcb3c62590c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 May 2021 00:20:57 +0300 Subject: [PATCH 400/832] improve docs --- doc/features.md | 16 ++++++++++++++-- doc/macros.md | 11 +++++++---- 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/doc/features.md b/doc/features.md index 14512e3c..0e64f02b 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1031,7 +1031,7 @@ Things missing from the standard library. - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python you have (and whether CPython or PyPy3). - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* - - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values, or named return values, packed into a `Values`, are unpacked to the args and kwargs of the next function in the chain. + - Any number of positional and keyword arguments are supported, with the same rules as in the pipe system. Multiple return values, or named return values, represented as a `Values`, are automatically unpacked to the args and kwargs of the next function in the chain. - `composelc`, `composerc`: curry each function before composing them. This comboes well with the passthrough of extra args/kwargs in `curry`. - An implicit top-level curry context is inserted around all the functions except the one that is applied last, to allow passthrough to the top level while applying the composed function. - `composel1`, `composer1`: 1-in-1-out chains (faster). @@ -1180,6 +1180,8 @@ The example we have here evaluates all items immediately, and specifically produ **Changed in v0.15.0.** *`curry` now supports kwargs, too, and binds parameters like Python itself does. Also, `@generic` and `@typed` functions are supported.* +*For advanced examples, see [the unit tests](../unpythonic/tests/test_fun.py).* + Our ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. Let's consider a simple example with positional arguments only. When we say: ```python @@ -1269,7 +1271,17 @@ curry(f, a, (g, x, y), b, c) because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by design; as with all things Python, *explicit is better than implicit*. -**Note**: to code in curried style, a [contract system](https://en.wikipedia.org/wiki/Design_by_contract) (such as [icontract](https://github.com/Parquery/icontract) or [PyContracts](https://github.com/AndreaCensi/contracts)) or the [mypy static type checker](http://mypy-lang.org/) can be useful; also, be careful with variadic functions. +**Note**: to code in curried style, a [contract system](https://en.wikipedia.org/wiki/Design_by_contract) or a type checker can be useful. Also, be careful with variadic functions, because any allowable arity will trigger the call. + +(The `map` function in the standard library is a particular offender here, since it requires at least one iterable to actually do anything but raise `TypeError`, but its call signature suggests it can be called without any iterables. Hence, for curry-friendliness we provide a wrapper `unpythonic.map` that *requires* at least one iterable.) + +- Contract systems for Python include [icontract](https://github.com/Parquery/icontract) and [PyContracts](https://github.com/AndreaCensi/contracts). + +- For static type checking, consider [mypy](http://mypy-lang.org/). + +- For run-time type checking, consider `@typed` or `@generic` right here in `unpythonic`. + +- You can also just use Python's type annotations; `unpythonic`'s `curry` type-checks the arguments before accepting the curried function. The annotations work if the stdlib function `typing.get_type_hints` can find them. #### ``fix``: break infinite recursion cycles diff --git a/doc/macros.md b/doc/macros.md index 61f7a347..a5149bb9 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1177,8 +1177,9 @@ Code within a ``with continuations`` block is treated specially. > - In a function definition inside the ``with continuations`` block: > - Most of the language works as usual; especially, any non-tail function calls can be made as usual. > - ``return value`` or ``return v0, ..., vn`` is actually a tail-call into ``cc``, passing the given value(s) as arguments. -> - As in other parts of ``unpythonic``, returning a tuple means returning multiple-values. +> - As in other parts of ``unpythonic``, returning a `Values` means returning multiple-return-values. > - This is important if the return value is received by the assignment targets of a ``call_cc[]``. If you get a ``TypeError`` concerning the arguments of a function with a name ending in ``_cont``, check your ``call_cc[]`` invocations and the ``return`` in the call_cc'd function. +> - **Changed in v0.15.0.** *Up to v0.14.3, multiple return values used to be represented as a `tuple`. Now returning a `tuple` means returning one value that is a tuple.* > - ``return func(...)`` is actually a tail-call into ``func``, passing along (by default) the current value of ``cc`` to become its ``cc``. > - Hence, the tail call is inserted between the end of the current function body and the start of the continuation ``cc``. > - To override which continuation to use, you can specify the ``cc=...`` kwarg, as in ``return func(..., cc=mycc)``. @@ -1260,9 +1261,9 @@ call_cc[f(...) if p else g(...)] **Assignment targets**: - - To destructure positional multiple-values (from a `Values` return value), use a tuple assignment target (comma-separated names, as usual). Destructuring *named* return values from a `call_cc` is currently not supported. + - To destructure positional multiple-values (from a `Values` return value of the function called by the `call_cc`), use a tuple assignment target (comma-separated names, as usual). Destructuring *named* return values from a `call_cc` is currently not supported due to syntactic limitations. - - The last assignment target may be starred. It is transformed into the vararg (a.k.a. ``*args``, star-args) of the continuation function. (It will capture a whole tuple, or any excess items, as usual.) + - The last assignment target may be starred. It is transformed into the vararg (a.k.a. ``*args``, star-args) of the continuation function created by the `call_cc`. (It will capture a whole tuple, or any excess items, as usual.) - To ignore the return value, just omit the assignment part. Useful if ``func`` was called only to perform its side-effects (the classic side effect is to stash ``cc`` somewhere for later use). @@ -1413,7 +1414,9 @@ In ``unpythonic`` specifically, a continuation is just a function. ([As John Shu The continuation function must be able to take as many positional arguments as the previous function in the TCO chain is trying to pass into it. Keep in mind that: - - In ``unpythonic``, a tuple represents multiple return values. So a ``return a, b``, which is being fed into the continuation, implies that the continuation must be able to take two positional arguments. + - In ``unpythonic``, multiple return values are represented as a `Values` object. So if your function does ``return Values(a, b)``, and that is being fed into the continuation, this implies that the continuation must be able to take two positional arguments. + + **Changed in v0.15.0.** *Up to v0.14.3, a `tuple` used to represent multiple-return-values; now it denotes a single return value that is a tuple. The `Values` type allows not only multiple return values, but also **named** return values. These are fed as kwargs.* - At the end of any function in Python, at least an implicit bare ``return`` always exists. It will try to pass in the value ``None`` to the continuation, so the continuation must be able to accept one positional argument. (This is handled automatically for continuations created by ``call_cc[]``. If no assignment targets are given, ``call_cc[]`` automatically creates one ignored positional argument that defaults to ``None``.) From 67ae4138c444b1b5a9d7f756d4f51d9efd632240 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 May 2021 00:28:01 +0300 Subject: [PATCH 401/832] mention that @generic has cross-cutting concerns --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 50257a42..872d4320 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ Since `unpythonic` is a relatively loose collection of language extensions and u To study a particular feature, just start from the entry point that piques your interest, and follow the definitions recursively. Use an IDE or Emacs's `anaconda-mode` ~for convenience~ to stay sane. Look at the automated tests; those double as usage examples, sometimes containing finer points that didn't make it to prose documentation. -`curry` has some [cross-cutting concerns](https://en.wikipedia.org/wiki/Cross-cutting_concern), but nothing that a grep wouldn't find. +`curry` has some [cross-cutting concerns](https://en.wikipedia.org/wiki/Cross-cutting_concern), but nothing that a grep wouldn't find. Same goes for the multiple-dispatch system (particularly `@generic`). The `lazify` and `continuations` macros are the most complex (and perhaps fearsome?) parts. As for the lazifier, grep also for `passthrough_lazy_args` and `maybe_force_args`. As for continuations, read the `tco` macro first, and keep in mind how that works when reading `continuations`. The `continuations` macro is essentially what [academics call](https://cs.brown.edu/~sk/Publications/Papers/Published/pmmwplck-python-full-monty/paper.pdf) *"a standard [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style) transformation"*, plus some technical details due to various bits of impedance mismatch. From ee46a8a4c80a0a678e34e74ace77612d177d4023 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 May 2021 00:33:27 +0300 Subject: [PATCH 402/832] add inspired-by note for `Values` --- doc/features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/features.md b/doc/features.md index 0e64f02b..b8e7fc28 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3691,6 +3691,8 @@ With `Values`, you can return multiple values positionally and by name. This com Having a `Values` type separate from `tuple` also helps with semantic accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value now means just that - one value that is a `tuple`. It is different from a `Values` that contains several positional return values (that are meant to be treated separately e.g. by a function composition utility). +Inspired by the [`values`](https://docs.racket-lang.org/reference/values.html) form of Racket. + #### When to use `Values` Most of the time, returning a tuple to denote multiple-return-values and unpacking it is just fine, and that is exactly what `unpythonic` does internally in many places. From e848f95759bc4a75a8369ac230d61a253e3aae82 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 May 2021 00:39:07 +0300 Subject: [PATCH 403/832] update comment --- unpythonic/syntax/autocurry.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 2dca5984..9f5ea550 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -88,6 +88,8 @@ def transform(self, tree): # Curry all calls; except as a small optimization, skip `Values(...)`, # which accepts any args and kwargs, so currying it does not make sense. # (It represents multiple-return-values in `unpythonic`.) + # This also allows other macros (that expand after `autocurry`) see + # the `Values(...)` call. Particularly, `lazify` is interested in it. if type(tree) is Call and not isx(tree.func, "Values"): if has_curry(tree): # detect decorated lambda with manual curry # the lambda inside the curry(...) is the next Lambda node we will descend into. From 19a09a6996f8d40afe39de0044fdd18398ec42f0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 May 2021 00:57:16 +0300 Subject: [PATCH 404/832] add TODO comment --- unpythonic/syntax/letsyntax.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index b0acc118..221032cd 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -4,6 +4,16 @@ # at macro expansion time. If you're looking for regular run-time let et al. macros, # see letdo.py. +# TODO: Coverage of code using `with block` and `with expr` is not reported correctly. +# +# TODO: As this is a toy macro system within the real macro system, that is to be expected; +# TODO: `mcpyrate` goes to some degree of trouble to produce correct coverage reporting for +# TODO: the real macro system, and we haven't duplicated that effort here. +# +# TODO: With `mcpyrate`, we don't really need `let_syntax` and `abbrev` anymore, so we could +# TODO: actually remove them; but their tests exercise some code paths that would otherwise +# TODO: remain untested. As of v0.15.0, we're keeping them for now. + __all__ = ["let_syntax", "abbrev", "expr", "block"] from mcpyrate.quotes import macros, q, a # noqa: F401 From a1d36a728c4fa9188711f4b18b1296552df92f9c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 31 May 2021 03:14:49 +0300 Subject: [PATCH 405/832] Fix wart in continuations subsystem It used to have to force the return value in `chain_conts`, although in `unpythonic` return values are never implicitly lazy. This came up in the Pytkell dialect tests; the dialect itself has `with lazify, autocurry`, and one test has a `with continuations` block. By arranging both `lazify` and `autocurry` to leave alone `Values(...)` as well as `(chain_conts(cc, pcc))(...)`, now we get clean expanded output also in the continuation-enabled case, and the continuations subsystem sees enough context to be able to skip lazifying the return value (which is, by that time, actually an argument to the function returned by `chain_conts`). Beside now actually honoring the idea that return values are never implicitly lazy, this also improves performance in code that uses this combination of features. Also, improve related comments while at it. --- unpythonic/syntax/autocurry.py | 38 +++++++++++++++++++++------------ unpythonic/syntax/lazify.py | 39 +++++++++++++++++++++++++--------- unpythonic/syntax/tailtools.py | 21 ------------------ 3 files changed, 53 insertions(+), 45 deletions(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 9f5ea550..40d29038 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -85,20 +85,30 @@ def transform(self, tree): return tree hascurry = self.state.hascurry - # Curry all calls; except as a small optimization, skip `Values(...)`, - # which accepts any args and kwargs, so currying it does not make sense. - # (It represents multiple-return-values in `unpythonic`.) - # This also allows other macros (that expand after `autocurry`) see - # the `Values(...)` call. Particularly, `lazify` is interested in it. - if type(tree) is Call and not isx(tree.func, "Values"): - if has_curry(tree): # detect decorated lambda with manual curry - # the lambda inside the curry(...) is the next Lambda node we will descend into. - hascurry = True - if not isx(tree.func, _iscurry): - tree.args = [tree.func] + tree.args - tree.func = q[h[currycall]] - if hascurry: # this must be done after the edit because the edit changes the children - self.generic_withstate(tree, hascurry=True) + if type(tree) is Call: + # Don't auto-curry some calls we know not to need it. This is both a performance optimization + # and allows other macros (particularly `lazify`) to be able to see the original calls. + # (It also generates cleaner expanded output.) + # - `Values(...)` accepts any args and kwargs, so currying it does not make sense. + # - `chain_conts(cc, pcc)(...)` handles a return value in `with continuations`. + # This has the effect that in `with continuations`, the tail-calls to continuation + # functions won't be curried, but perhaps that's ok. This allows the Pytkell dialect's + # `with lazify, autocurry` combo to work with an inner `with continuations`. + if (isx(tree.func, "Values") or + (type(tree.func) is Call and isx(tree.func.func, "chain_conts"))): + # However, *do* auto-curry in the positional and named args of the call. + tree.args = self.visit(tree.args) + tree.keywords = self.visit(tree.keywords) + return tree + else: # general case + if has_curry(tree): # detect decorated lambda with manual curry + # the lambda inside the curry(...) is the next Lambda node we will descend into. + hascurry = True + if not isx(tree.func, _iscurry): + tree.args = [tree.func] + tree.args + tree.func = q[h[currycall]] + if hascurry: # this must be done after the edit because the edit changes the children + self.generic_withstate(tree, hascurry=True) elif type(tree) in (FunctionDef, AsyncFunctionDef): if not any(isx(item, _iscurry) for item in tree.decorator_list): # no manual curry already diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 78c533c2..eae73d64 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -714,30 +714,49 @@ def transform_starred(tree, dstarred=False): thelambda.body = self.visit(thelambda.body) return tree - # namelambda() is used by let[] and do[] - # Lazy() is a strict function, takes a lambda, constructs a Lazy object - # _autoref_resolve doesn't need any special handling - # Values() doesn't need any special handling + # Don't lazify in calls to some specific functions we know to be strict. + # Some of these are performance optimizations; others must be left as-is + # for other macros to be able to see the original calls. (It also generates + # cleaner expanded output.) + # - `namelambda` (emitted by `let[]`, `do[]`, and `test[]`) + # - All known container constructor calls (listed in `_ctorcalls_all`). + # - `Lazy` takes a lambda, constructs a `Lazy` object; if we're calling `Lazy`, + # the expression is already lazy. + # - `_autoref_resolve` does the name lookup in `with autoref` blocks. + # + # Don't lazify in calls to return-value utilities, because return values + # are never implicitly lazy in `unpythonic`. + # - `Values` constructs a multiple-return-values and/or named return values. + # - `(chain_conts(cc1, cc2))(args)` handles a return value in `with continuations`. elif (isdo(tree) or is_decorator(tree.func, "namelambda") or any(isx(tree.func, s) for s in _ctorcalls_all) or isx(tree.func, _expanded_lazy_name) or isx(tree.func, "_autoref_resolve") or - isx(tree.func, "Values")): - # here we know the operator (.func) to be one of specific names; - # don't transform it to avoid confusing lazyrec[] (important if this - # is an inner call in the arglist of an outer, lazy call, since it - # must see any container constructor calls that appear in the args) + isx(tree.func, "Values") or + (type(tree.func) is Call and isx(tree.func.func, "chain_conts"))): + # Here we know the operator (.func) to be one of specific names; + # don't transform it to avoid confusing `lazyrec[]`. + # + # This is especially important, if this is an inner call in the + # arglist of an outer, lazy call, since it must see any container + # constructor calls that appear in the args. + # + # But *do* transform in the positional and named args of the call; + # doing so generates the code to force any promises that are passed + # to the function being called. # # TODO: correct forcing mode for recursion? We shouldn't need to forcibly use "full", # since maybe_force_args() already fully forces any remaining promises # in the args when calling a strict function. + # NOTE v0.15.0: In practice, using whatever is the currently active mode seems to be fine. tree.args = self.visit(tree.args) tree.keywords = self.visit(tree.keywords) return tree - else: + else: # general case thefunc = self.visit(tree.func) + # Lazify the arguments of the call. adata = [] for x in tree.args: if type(x) is Starred: # *args in Python 3.5+ diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 981903dd..dcca5a8f 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -38,7 +38,6 @@ from ..fun import identity from ..funutil import Values from ..it import uniqify -from ..lazyutil import force1, passthrough_lazy_args from ..tco import trampolined, jump # In `continuations`, we use `aif` and `it` as hygienically captured macros. @@ -738,7 +737,6 @@ def chain_conts(cc1, cc2, with_star=False): # cc1=_pcc, cc2=cc """Internal function, used in code generated by the continuations macro.""" if with_star: # to be chainable from a tail call, accept a multiple-values arglist if cc1 is not None: - @passthrough_lazy_args def cc(*rets, **kwrets): return jump(cc1, cc=cc2, *rets, **kwrets) else: @@ -749,32 +747,13 @@ def cc(*rets, **kwrets): cc = cc2 else: # for inert data value returns (this produces the multiple-values arglist) if cc1 is not None: - @passthrough_lazy_args def cc(return_value): - # Return values are never implicitly lazy in `unpythonic`, - # so why we need to `force1` here requires a comment. - # - # In general, we should treat these `cc` functions as lazy, - # so they won't force their args. Those args here are a return value, - # but due to `continuations`, it's not just a return, but a call - # into the `cc` function. - # - # Thus, returning a `Values` from a continuation-enabled function, - # that `Values` ends up here (or in the other branch, with no `cc1`). - # Because it's *technically* an argument for a lazy function, it gets - # a `lazy[]` wrapper added by `with lazify`. - # - # To determine whether we have one or multiple return values, we must - # force that wrapper promise, without touching anything inside. - return_value = force1(return_value) if isinstance(return_value, Values): return jump(cc1, cc=cc2, *return_value.rets, **return_value.kwrets) else: return jump(cc1, return_value, cc=cc2) else: - @passthrough_lazy_args def cc(return_value): - return_value = force1(return_value) if isinstance(return_value, Values): return jump(cc2, *return_value.rets, **return_value.kwrets) else: From 8b26635494221da7ce919664cdf5207d5564b6b0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 1 Jun 2021 00:09:24 +0300 Subject: [PATCH 406/832] consistent parenthesization --- unpythonic/syntax/autocurry.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 40d29038..0fea4484 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -90,7 +90,7 @@ def transform(self, tree): # and allows other macros (particularly `lazify`) to be able to see the original calls. # (It also generates cleaner expanded output.) # - `Values(...)` accepts any args and kwargs, so currying it does not make sense. - # - `chain_conts(cc, pcc)(...)` handles a return value in `with continuations`. + # - `(chain_conts(cc1, cc2))(...)` handles a return value in `with continuations`. # This has the effect that in `with continuations`, the tail-calls to continuation # functions won't be curried, but perhaps that's ok. This allows the Pytkell dialect's # `with lazify, autocurry` combo to work with an inner `with continuations`. From 1ac885a2a1fa330ae6e0defda18920fae23554fc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 1 Jun 2021 00:09:30 +0300 Subject: [PATCH 407/832] add note of advantages of compile-time vs. run-time --- CONTRIBUTING.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 872d4320..a45ba7b6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -179,6 +179,16 @@ As of the first half of 2021, the main target platforms are **CPython 3.8** and - When implementing something, if you run into an empty niche, add the missing utility, and implement your higher-level functionality in terms of it. - This keeps code at each level of abstraction short, and exposes parts that can later be combined in new ways. +- **Compile-time or run-time?** + - For anyone new to making programming languages: there's a reason the terms static/lexical/compile-time and dynamic/run-time are grouped together. + - At compile time (macros), you have access to the source code (or AST), including its lexical structure. (I.e. what is defined inside what, in the source code text.) + - You also have access to the macro bindings of the current expander, because [*for the macros, it's run time*](https://github.com/Technologicat/mcpyrate/blob/master/doc/troubleshooting.md#macro-expansion-time-where-exactly). + - A block macro (`with mac:`) takes effect **for the lexical content of that block**. + - At run time (regular code), you have access to run-time bindings of names (e.g. whether `curry` refers to `unpythonic.fun.curry` or something else), and the call stack. + - Keep in mind that in Python, knowing what a name at the top level of a module (i.e. a "global variable") points to *is only possible at run time*. Although it's uncommon, not to mention bad practice in most cases, *any code anywhere* may change the top-level bindings in *any* module (via `sys.modules`). + - A run-time context manager (`with mgr:`) takes effect **for the dynamic extent of that block**. + - Try to take advantage of whichever is the most appropriate for what you're doing. + - **Follow [PEP8](https://www.python.org/dev/peps/pep-0008/) style**, *including* the official recommendation to violate PEP8 when the guidelines do not apply. Specific to `unpythonic`: - Conserve vertical space when reasonable. Even on modern laptops, a display can only fit ~50 lines at a time. - `x = x or default` for initializing `x` inside the function body of `def f(x=None)` (when it makes no sense to publish the actual default value) is concise and very readable. From a92825fd0ded05afe15a7707b4cfb9dbaa86c808 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:23:51 +0300 Subject: [PATCH 408/832] update comment --- unpythonic/dialects/tests/test_lispython.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index 9e3cba42..a479a8ec 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -120,7 +120,7 @@ def f(k, acc): test[x == 3] with testset("integration with continuations"): - with continuations: # should be skipped by the implicit tco inserted by the dialect + with continuations: # has TCO; should be skipped by the implicit `with tco` inserted by the dialect k = None # kontinuation def setk(*args, cc): nonlocal k From 29634b8296ebf702b8e458b1187a86e29a291140 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:24:13 +0300 Subject: [PATCH 409/832] comments/docstrings/error message: MonadicList, not List --- unpythonic/amb.py | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/unpythonic/amb.py b/unpythonic/amb.py index 45e8cd3b..193af8db 100644 --- a/unpythonic/amb.py +++ b/unpythonic/amb.py @@ -14,7 +14,7 @@ - Presents the source code in the same order as it actually runs. -The implementation is based on the List monad. This is a hack with the bare +The implementation is based on the list monad. This is a hack with the bare minimum of components to make it work, complete with a semi-usable syntax. If you use `mcpyrate`: @@ -59,7 +59,7 @@ def forall(*lines): """Nondeterministically evaluate lines. This is essentially a bastardized variant of Haskell's do-notation, - specialized for the List monad. + specialized for the list monad. Examples:: @@ -83,8 +83,8 @@ def forall(*lines): - All choices are evaluated, depth first, and set of results is returned as a tuple. - - If a line returns an iterable, it is implicitly converted into a List - monad containing the same items. + - If a line returns an iterable, it is implicitly converted into a + list monad containing the same items. - This applies also to the RHS of a ``choice``. @@ -94,11 +94,11 @@ def forall(*lines): This allows easily returning a tuple (as one result item) from the computation, as in the above pythagorean triples example. - - If a line returns a single item, it is wrapped into a singleton List - (a List containing that one item). + - If a line returns a single item, it is wrapped into a singleton + list monad (a MonadicList containing that one item). - The final result (containing all the results) is converted from - List monad to tuple for output. + the list monad to tuple for output. - The values currently picked by the choices are bound to names in the environment. To access it, use a ``lambda e: ...`` like in @@ -212,7 +212,7 @@ def monadify(value, unpack=True): return MonadicList.from_iterable(value) except TypeError: pass # fall through - return MonadicList(value) # unit(List, value) + return MonadicList(value) # unit(MonadicList, value) class MonadicList: # TODO: This if anything is **the** place to use @typed. """A monadic list.""" @@ -223,7 +223,7 @@ def __init__(self, *elts): returns: M a """ # Accept the sentinel nil as a special **item** that, when passed to - # the List constructor, produces an empty list. + # the MonadicList constructor, produces an empty list. if len(elts) == 1 and elts[0] is nil: self.x = () else: @@ -243,8 +243,8 @@ def __rshift__(self, f): """ # bind ma f = join (fmap f ma) return self.fmap(f).join() - # done manually, essentially List.from_iterable(flatmap(lambda elt: f(elt), self.x)) - #return List.from_iterable(result for elt in self.x for result in f(elt)) + # done manually, essentially MonadicList.from_iterable(flatmap(lambda elt: f(elt), self.x)) + # return MonadicList.from_iterable(result for elt in self.x for result in f(elt)) def then(self, f): """Sequence, a.k.a. "then"; standard notation ">>" in Haskell. @@ -257,7 +257,7 @@ def then(self, f): """ cls = self.__class__ if not isinstance(f, cls): - raise TypeError(f"Expected a List monad, got {type(f)} with value {repr(f)}") + raise TypeError(f"Expected a MonadicList, got {type(f)} with value {repr(f)}") return self >> (lambda _: f) @classmethod @@ -282,10 +282,10 @@ def guard(cls, b): cancels the rest of that branch of the computation. """ if b: - return cls(True) # List with one element; value not intended to be actually used. - return cls() # 0-element List; short-circuit this branch of the computation. + return cls(True) # MonadicList with one element; value not intended to be actually used. + return cls() # 0-element MonadicList; short-circuit this branch of the computation. - # make List iterable so that "for result in f(elt)" works (when f outputs a List monad) + # make MonadicList iterable so that "for result in f(elt)" works (when f outputs a list monad) def __iter__(self): return iter(self.x) def __len__(self): @@ -330,7 +330,7 @@ def copy(self): @classmethod def lift(cls, f): - """Lift a regular function into a List-producing one. + """Lift a regular function into a MonadicList-producing one. f: a -> b returns: a -> M b @@ -355,7 +355,7 @@ def join(self): """ cls = self.__class__ if not all(isinstance(elt, cls) for elt in self.x): - raise TypeError(f"Expected a nested List monad, got {type(self.x)} with value {self.x}") + raise TypeError(f"Expected a nested MonadicList, got {type(self.x)} with value {self.x}") # list of lists - concat them return cls.from_iterable(elt for sublist in self.x for elt in sublist) From 7e30405022a7f49184dfe6088d8bde9c2e0e2264 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:24:44 +0300 Subject: [PATCH 410/832] update Pytkell tests now that curry handles kwargs --- unpythonic/dialects/tests/test_pytkell.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index 94c9a77a..46d86c62 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -74,8 +74,8 @@ def f(a, b): test[f(1, 2) == (1, 2)] test[(flip(f))(1, 2) == (2, 1)] # NOTE flip reverses all (doesn't just flip the first two) # noqa: F821 - # # TODO: this doesn't work, because curry sees f's arities as (2, 2) (kwarg handling!) - # test[(flip(f))(1, b=2) == (1, 2)] # b -> kwargs + # flip reverses only those arguments that are passed *positionally* + test[(flip(f))(1, b=2) == (1, 2)] # b -> kwargs # noqa: F821 # http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html with testset("iterables"): @@ -200,7 +200,7 @@ def f(k, acc): if k == 1: return acc return f(k - 1, k * acc) - return f(n, 1) # TODO: doesn't work as f(n, acc=1) due to curry's kwarg handling + return f(n, acc=1) test[fact(4) == 24] print("Performance...") From f3924bde694214ad7382de6445441c3121ba9fee Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:25:04 +0300 Subject: [PATCH 411/832] update comment --- unpythonic/dialects/tests/test_pytkell.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index 46d86c62..b21c18d4 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -145,7 +145,7 @@ def f(a, b): test[last(take(1001, s(0, 0.001, ...))) == 1] # noqa: F821 # iterables returned by s() support infix math - # (to add infix math support to some other iterable, m(iterable)) + # (to add infix math support to some other iterable, imathify(iterable)) c = s(1, 3, ...) + s(2, 4, ...) # noqa: F821 test[tuple(take(5, c)) == (3, 7, 11, 15, 19)] # noqa: F821 test[tuple(take(5, c)) == (23, 27, 31, 35, 39)] # consumed! # noqa: F821 From 0e9e3512e4e0f0ef5f7d87b0d619c029872eb243 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:42:36 +0300 Subject: [PATCH 412/832] add cautions about Pytkell being slow Doesn't matter though; this dialect is at its best in surfacing otherwise hard-to-find bugs or misdesigns in the feature integration of `unpythonic`, and in `mcpyrate`. I've already fixed a couple of things thanks to the existence of Pytkell. Also, it should be serviceable for teaching about lazy functions and autocurry in a Python-based setting. --- unpythonic/dialects/tests/test_pytkell.py | 25 ++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/unpythonic/dialects/tests/test_pytkell.py b/unpythonic/dialects/tests/test_pytkell.py index b21c18d4..3c78cd63 100644 --- a/unpythonic/dialects/tests/test_pytkell.py +++ b/unpythonic/dialects/tests/test_pytkell.py @@ -113,6 +113,15 @@ def f(a, b): # # pythagorean triples with testset("nondeterministic evaluation"): + # TODO: This is very slow in Pytkell; investigate whether the cause is `lazify`, `autocurry`, or both. + # + # Running the same code in a macro-enabled IPython (i.e. without Pytkell), there is no noticeable delay + # after you press enter, before it gives the result. If you want to try it, you'll need to: + # + # %load_ext mcpyrate.repl.iconsole + # from unpythonic.syntax import macros, forall, test + # from unpythonic import insist + # pt = forall[z << range(1, 21), # hypotenuse # noqa: F821 x << range(1, z + 1), # shorter leg # noqa: F821 y << range(x, z + 1), # longer leg # noqa: F821 @@ -203,9 +212,23 @@ def f(k, acc): return f(n, acc=1) test[fact(4) == 24] + # **CAUTION**: Pytkell is slow, because so much happens at run time. On an i7-4710MQ: + # + # - The performance test below, `fact(5000)`, completes in about 500ms. + # + # **Without** Pytkell, using a macro-enabled IPython session: + # + # - `fact(5000)` with the same definition (the `with tco` block above) completes in about 15ms. + # - `prod(range(1, 5001))` completes in about 7ms. (This is `unpythonic.prod`, which uses + # `unpythonic`'s custom fold implementation.) + # - The simplest thing that works: + # n = 1 + # for k in range(1, 5001): + # n *= k + # completes in about 5ms. print("Performance...") with timer() as tictoc: - fact(5000) # no crash, but Pytkell is a bit slow + fact(5000) # no crash print(" Time taken for factorial of 5000: {:g}s".format(tictoc.dt)) if __name__ == '__main__': From f44972d167c3d78e3a8cd1a54851d205c6cad763 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:47:47 +0300 Subject: [PATCH 413/832] add section breaks --- unpythonic/fun.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index e1c0c70b..8521a4ba 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -35,6 +35,8 @@ # we use @passthrough_lazy_args (and handle possible lazy args) to support unpythonic.syntax.lazify. from .lazyutil import passthrough_lazy_args, islazy, force, force1, maybe_force_args +# -------------------------------------------------------------------------------- + _success = sym("_success") _fail = sym("_fail") @register_decorator(priority=10) @@ -77,6 +79,7 @@ def memoized(*args, **kwargs): # memo[k] = f(*args, **kwargs) # return memo[k] # return memoized +# -------------------------------------------------------------------------------- # Parameter naming is consistent with `functools.partial`. # @@ -545,6 +548,8 @@ def iscurried(f): # return f(*args, **kwargs) # return curried +# -------------------------------------------------------------------------------- + def flip(f): """Decorator: flip (reverse) the positional arguments of f.""" @wraps(f) @@ -585,6 +590,8 @@ def rotated(*args, **kwargs): return rotated return rotate_k +# -------------------------------------------------------------------------------- + @passthrough_lazy_args def apply(f, arg0, *more, **kwargs): """Scheme/Racket-like apply. @@ -609,6 +616,8 @@ def apply(f, arg0, *more, **kwargs): lst = tuple(more[-1]) return maybe_force_args(f, *(args + lst), **kwargs) +# -------------------------------------------------------------------------------- + # Not marking this as lazy-aware works better with continuations (since this # is the default cont, and return values should be values, not lazy[]) def identity(*args, **kwargs): @@ -673,6 +682,8 @@ def constant(*a, **kw): return ret return constant +# -------------------------------------------------------------------------------- + def notf(f): # Racket: negate """Return a function that returns the logical not of the result of f. @@ -739,6 +750,8 @@ def disjoined(*args, **kwargs): return False return disjoined +# -------------------------------------------------------------------------------- + def _make_compose1(direction): """Make a function that composes functions from an iterable. @@ -930,6 +943,8 @@ def composelci(iterable): """Like composelc, but read the functions from an iterable.""" return composeli(map(curry, iterable)) +# -------------------------------------------------------------------------------- + # Helpers to insert one-in-one-out functions into multi-arg compose chains def tokth(k, f): """Return a function to apply f to args[k], pass the rest through. @@ -997,6 +1012,8 @@ def to(*specs): """ return composeli(tokth(k, f) for k, f in specs) +# -------------------------------------------------------------------------------- + @register_decorator(priority=80) def withself(f): """Decorator. Allow a lambda to refer to itself. From c9b7a56a265febc0f1ec679c2a5177d66a9fd615 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:48:05 +0300 Subject: [PATCH 414/832] presentation: move memoize_simple --- unpythonic/fun.py | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 8521a4ba..2198bb68 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -37,6 +37,16 @@ # -------------------------------------------------------------------------------- +#def memoize_simple(f): # essential idea, without exception handling +# memo = {} +# @wraps(f) +# def memoized(*args, **kwargs): +# k = tuplify_bindings(resolve_bindings(f, *args, **kwargs)) +# if k not in memo: +# memo[k] = f(*args, **kwargs) +# return memo[k] +# return memoized + _success = sym("_success") _fail = sym("_fail") @register_decorator(priority=10) @@ -70,15 +80,6 @@ def memoized(*args, **kwargs): memoized = passthrough_lazy_args(memoized) return memoized -#def memoize_simple(f): # essential idea, without exception handling -# memo = {} -# @wraps(f) -# def memoized(*args, **kwargs): -# k = tuplify_bindings(resolve_bindings(f, *args, **kwargs)) -# if k not in memo: -# memo[k] = f(*args, **kwargs) -# return memo[k] -# return memoized # -------------------------------------------------------------------------------- # Parameter naming is consistent with `functools.partial`. From 21b002690bb3de5592c562f0d487801ba11c1d19 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:48:14 +0300 Subject: [PATCH 415/832] update comment (wasn't strictly correct) --- unpythonic/fun.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 2198bb68..bc96549c 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -134,7 +134,7 @@ def partial(func, *args, **kwargs): _extract_self_or_cls(thecallable, args)), _partial=True) - else: # Not `@generic` or `@typed`; just a function that has type annotations. + else: # Not `@generic` or `@typed`; just a function that might have type annotations. # It's not very unpythonic-ic to provide this since we already have `@typed` for this use case, # but it's much more pythonic, if the type-checking `partial` works properly for code that does # not opt in to `unpythonic`'s multiple-dispatch subsystem. From 6745c6c668bd9394c01e153a8ec68149f9dda181 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:48:34 +0300 Subject: [PATCH 416/832] update docstring --- unpythonic/fun.py | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index bc96549c..7b8d878c 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -280,15 +280,8 @@ def f(x, y): assert f(y=2)(x=1) == (1, 2) - However, it is possible that the algorithm isn't perfect, so there may be small semantic - differences to regular one-step function calls. If you find any, please file an issue, - so these can at the very least be documented; and if doable with reasonable effort, - preferably fixed. - - It is still an error if **named** arguments are left over for an outer curry context. - Treating this case would require generalizing return values so that functions could - return named outputs. See: - https://github.com/Technologicat/unpythonic/issues/32 + If you notice any semantic differences in parameter binding when using `curry`, when compared + to regular one-step function calls, please file an issue. """ f = force(f) # lazify support: we need the value of f # trivial case first: interaction with call_ec and other replace-def-with-value decorators From 5ac2899cc0ce72e86b74fc4b31e22377ccd22945 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 00:49:04 +0300 Subject: [PATCH 417/832] Improve curry performance - Hoist `analyze_parameter_bindings` to the top level. This makes `curry` itself run faster, which is significant in code using `with autocurry`. See particularly the Pytkell test; this change makes `fact(5000)` run roughly two to three times faster. - Remove unnecessary `force1`, since in `unpythonic` return values are never implicitly lazy (in code using `with lazify`). (Negligible performance advantage, but cleaner.) - Presentation order in the source code: give the simple idea first. --- unpythonic/fun.py | 282 +++++++++++++++++++++++----------------------- 1 file changed, 144 insertions(+), 138 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 7b8d878c..42ac4000 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -32,8 +32,8 @@ from .regutil import register_decorator from .symbol import sym -# we use @passthrough_lazy_args (and handle possible lazy args) to support unpythonic.syntax.lazify. -from .lazyutil import passthrough_lazy_args, islazy, force, force1, maybe_force_args +# We use `@passthrough_lazy_args` and `maybe_force_args` to support unpythonic.syntax.lazify. +from .lazyutil import passthrough_lazy_args, islazy, force, maybe_force_args # -------------------------------------------------------------------------------- @@ -157,20 +157,22 @@ def partial(func, *args, **kwargs): # `functools.partial` already handles chaining partial applications, so send only the new args/kwargs to it. return functools_partial(func, *args, **kwargs) -make_dynvar(curry_context=[]) -@passthrough_lazy_args -def _currycall(f, *args, **kwargs): - """Co-operate with unpythonic.syntax.curry. +# -------------------------------------------------------------------------------- - In a ``with autocurry`` block, we need to call `f` also when ``f()`` has - transformed to ``curry(f)``, but definitions can be curried as usual. +#def curry_simple(f): # essential idea, without any extra features +# min_arity, _ = arities(f) +# @wraps(f) +# def curried(*args, **kwargs): +# if len(args) < min_arity: +# return curry(partial(f, *args, **kwargs)) +# return f(*args, **kwargs) +# return curried - Hence we provide this separate mode to curry-and-call even if no args. +make_dynvar(curry_context=[]) - This mode no-ops when ``f`` is not inspectable, instead of raising - an ``unpythonic.arity.UnknownArity`` exception. - """ - return curry(f, *args, _curry_force_call=True, _curry_allow_uninspectable=True, **kwargs) +def iscurried(f): + """Return whether f is a curried function.""" + return hasattr(f, "_is_curried_function") @register_decorator(priority=8) @passthrough_lazy_args @@ -301,117 +303,17 @@ def fallback(): # what to do when inspection fails return maybe_force_args(f, *args, **kwargs) return f - # Try to fail-fast with uninspectable builtins. - try: - signature(f) - except ValueError as err: # inspection failed in inspect.signature()? - msg = err.args[0] - if "no signature found" in msg: - return fallback() - raise - - # TODO: To make `curry` pay-as-you-go, look for opportunities to speed this up - # for non-`@generic` functions. Currently this more general `curry` for v0.15.0 - # (that handles kwargs correctly) can be even 50% slower than the more limited one - # (based on positional arity only) that was in v0.14.3. - - # actions - _call = sym("_call") - _call_with_passthrough = sym("_call_with_passthrough") - _keep_currying = sym("_keep_currying") - Analysis = namedtuple("Analysis", ["bound_arguments", "unbound_parameters", "extra_args", "extra_kwargs"]) - def analyze_parameter_bindings(f, args, kwargs): - # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by - # `inspect.signature`), but `functools.partial` objects have a `keywords` attribute, which - # contains what we want. - # - # To support kwargs properly, we must compute argument bindings anyway, so we also use the - # `func` and `args` attributes. This allows us to compute the bindings of all arguments - # against the original function. - if isinstance(f, functools_partial): - function = f.func - collected_args = f.args + args - collected_kwargs = {**f.keywords, **kwargs} - else: - function = f - collected_args = args - collected_kwargs = kwargs - - def _bind_arguments(thecallable): - # For this check we look for a complete match, hence `_partial=False`. - bound_arguments, unbound_parameters, (extra_args, extra_kwargs) = _bind(signature(thecallable), - collected_args, - collected_kwargs, - partial=False) - return Analysis(bound_arguments, unbound_parameters, extra_args, extra_kwargs) - - # `@generic` functions have several call signatures, so we must aggregate the results - # in a sensible way. For non-generics, there's just one call signature. - if not isgeneric(function): - # For non-generics, the curry-time type check occurs when we later call `partial`, - # so we don't need to do that here. We just compute the bindings of arguments to parameters. - analysis = _bind_arguments(function) - if not analysis.unbound_parameters and not analysis.extra_args and not analysis.extra_kwargs: - return _call, analysis - elif not analysis.unbound_parameters and (analysis.extra_args or analysis.extra_kwargs): - return _call_with_passthrough, analysis - assert analysis.unbound_parameters - return _keep_currying, analysis - - # Curry resolver for `@generic`/`@typed` (generic functions, multimethods, multiple dispatch). - # - # Iterate over multimethods, once per step: - # - # 1. If there is an exact match (all parameters bound, type check passes, no extra - # `args`/`kwargs`), call it. - # 2. If there is a complete match (all parameters bound, type check passes), but - # with extra `args`/`kwargs` (that cannot be accepted by the call signature), - # call it, arranging passthrough for the extra `args`/`kwargs`. - # 3. If there is at least one partial match (type check passes for bound arguments, - # unbound parameters remain), keep currying. In this case extra `args`/`kwargs`, - # if any, do not matter. This will fall into case 1 or 2 above after we get - # additional `args`/`kwargs` to complete a match. - # - # If none of the above match, we know at least one parameter got a binding - # that fails the type check. Raise `TypeError`. - # - # In steps 1 and 2, we use the same lookup order as the multiple dispatcher does; - # the first matching multimethod wins. Actual dispatch is still done by the dispatcher; - # we only compute the bindings to determine which case above the call falls into. - # - # `@typed` is a special case of `@generic` with just one multimethod registered. - # The resulting behavior is the same as for a non-generic function, because the - # above algorithm reduces to that. - - # We can't use the public `list_methods` here, because on OOP methods, - # decorators live on the unbound method (raw function). Thus we must - # extract `self`/`cls` from the arguments of the call (for linked - # dispatcher lookup in the MRO). - multimethods = _list_multimethods(function, - _extract_self_or_cls(function, - collected_args)) - # Step 1: exact match - for thecallable, type_signature in multimethods: - analysis = _bind_arguments(thecallable) - if not analysis.unbound_parameters and not analysis.extra_args and not analysis.extra_kwargs: - if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): - return _call, analysis - # Step 2: complete match, with extra args/kwargs - for thecallable, type_signature in multimethods: - analysis = _bind_arguments(thecallable) - if not analysis.unbound_parameters and (analysis.extra_args or analysis.extra_kwargs): - if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): - return _call_with_passthrough, analysis - # Step 3: partial match - for thecallable, type_signature in multimethods: - analysis = _bind_arguments(thecallable) - if analysis.unbound_parameters: - if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): - return _keep_currying, analysis - # No matter which multimethod we pick, at least one parameter gets a binding - # that fails the type check. - _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, - candidates=multimethods, _partial=True) + # Try to fail-fast with uninspectable builtins, even if no arguments were passed. + # (If we get arguments, there's no landmine, because calling the curried function + # will perform the signature analysis.) + if not (args or kwargs): + try: + signature(f) + except ValueError as err: # inspection failed in inspect.signature()? + msg = err.args[0] + if "no signature found" in msg: + return fallback() + raise @wraps(f) def curried(*args, **kwargs): @@ -421,7 +323,7 @@ def curried(*args, **kwargs): # the parameter bindings. All of `f`'s parameters must be bound (whether by position or by # name) before calling `f`. try: - action, analysis = analyze_parameter_bindings(f, args, kwargs) + action, analysis = _analyze_parameter_bindings(f, args, kwargs) except ValueError as err: # inspection failed in inspect.signature()? msg = err.args[0] if "no signature found" in msg: @@ -457,7 +359,6 @@ def curried(*args, **kwargs): if now_result.rets: # `leftmost`, not `first`, for unambiguous stack traces. leftmost, *others = now_result.rets - leftmost = force1(leftmost) # Extra positional arguments (`later_args`) are passed through *on the right*. # Hence any further positional return values are inserted before them. @@ -479,7 +380,7 @@ def curried(*args, **kwargs): later_kwargs = {**later_kwargs, **now_result.kwrets} else: # The only return value is also the leftmost one. - leftmost = force1(now_result) + leftmost = now_result if callable(leftmost): pass else: @@ -529,18 +430,123 @@ def curried(*args, **kwargs): return maybe_force_args(curried, *args, **kwargs) return curried -def iscurried(f): - """Return whether f is a curried function.""" - return hasattr(f, "_is_curried_function") +@passthrough_lazy_args +def _currycall(f, *args, **kwargs): + """Co-operate with unpythonic.syntax.curry. -#def curry_simple(f): # essential idea, without any extra features -# min_arity, _ = arities(f) -# @wraps(f) -# def curried(*args, **kwargs): -# if len(args) < min_arity: -# return curry(partial(f, *args, **kwargs)) -# return f(*args, **kwargs) -# return curried + In a ``with autocurry`` block, we need to call `f` also when ``f()`` has + transformed to ``curry(f)``, but definitions can be curried as usual. + + Hence we provide this separate mode to curry-and-call even if no args. + + This mode no-ops when ``f`` is not inspectable, instead of raising + an ``unpythonic.arity.UnknownArity`` exception. + """ + return curry(f, *args, _curry_force_call=True, _curry_allow_uninspectable=True, **kwargs) + +# actions during currying +_call = sym("_call") +_call_with_passthrough = sym("_call_with_passthrough") +_keep_currying = sym("_keep_currying") + +_Analysis = namedtuple("_Analysis", ["bound_arguments", "unbound_parameters", "extra_args", "extra_kwargs"]) + +# Internal helper for `curry`. +# +# For performance, it is important to have this function defined once at the top level +# of the module, instead of defining it as a closure each time `curry` is called. +def _analyze_parameter_bindings(f, args, kwargs): + # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by + # `inspect.signature`), but `functools.partial` objects have a `keywords` attribute, which + # contains what we want. + # + # To support kwargs properly, we must compute argument bindings anyway, so we also use the + # `func` and `args` attributes. This allows us to compute the bindings of all arguments + # against the original function. + if isinstance(f, functools_partial): + function = f.func + collected_args = f.args + args + collected_kwargs = {**f.keywords, **kwargs} + else: + function = f + collected_args = args + collected_kwargs = kwargs + + def _bind_arguments(thecallable): + # For this check we look for a complete match, hence `_partial=False`. + bound_arguments, unbound_parameters, (extra_args, extra_kwargs) = _bind(signature(thecallable), + collected_args, + collected_kwargs, + partial=False) + return _Analysis(bound_arguments, unbound_parameters, extra_args, extra_kwargs) + + # `@generic` functions have several call signatures, so we must aggregate the results + # in a sensible way. For non-generics, there's just one call signature. + if not isgeneric(function): + # For non-generics, the curry-time type check occurs when we later call `partial`, + # so we don't need to do that here. We just compute the bindings of arguments to parameters. + analysis = _bind_arguments(function) + if not analysis.unbound_parameters and not analysis.extra_args and not analysis.extra_kwargs: + return _call, analysis + elif not analysis.unbound_parameters and (analysis.extra_args or analysis.extra_kwargs): + return _call_with_passthrough, analysis + assert analysis.unbound_parameters + return _keep_currying, analysis + + # Curry resolver for `@generic`/`@typed` (generic functions, multimethods, multiple dispatch). + # + # Iterate over multimethods, once per step: + # + # 1. If there is an exact match (all parameters bound, type check passes, no extra + # `args`/`kwargs`), call it. + # 2. If there is a complete match (all parameters bound, type check passes), but + # with extra `args`/`kwargs` (that cannot be accepted by the call signature), + # call it, arranging passthrough for the extra `args`/`kwargs`. + # 3. If there is at least one partial match (type check passes for bound arguments, + # unbound parameters remain), keep currying. In this case extra `args`/`kwargs`, + # if any, do not matter. This will fall into case 1 or 2 above after we get + # additional `args`/`kwargs` to complete a match. + # + # If none of the above match, we know at least one parameter got a binding + # that fails the type check. Raise `TypeError`. + # + # In steps 1 and 2, we use the same lookup order as the multiple dispatcher does; + # the first matching multimethod wins. Actual dispatch is still done by the dispatcher; + # we only compute the bindings to determine which case above the call falls into. + # + # `@typed` is a special case of `@generic` with just one multimethod registered. + # The resulting behavior is the same as for a non-generic function, because the + # above algorithm reduces to that. + + # We can't use the public `list_methods` here, because on OOP methods, + # decorators live on the unbound method (raw function). Thus we must + # extract `self`/`cls` from the arguments of the call (for linked + # dispatcher lookup in the MRO). + multimethods = _list_multimethods(function, + _extract_self_or_cls(function, + collected_args)) + # Step 1: exact match + for thecallable, type_signature in multimethods: + analysis = _bind_arguments(thecallable) + if not analysis.unbound_parameters and not analysis.extra_args and not analysis.extra_kwargs: + if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): + return _call, analysis + # Step 2: complete match, with extra args/kwargs + for thecallable, type_signature in multimethods: + analysis = _bind_arguments(thecallable) + if not analysis.unbound_parameters and (analysis.extra_args or analysis.extra_kwargs): + if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): + return _call_with_passthrough, analysis + # Step 3: partial match + for thecallable, type_signature in multimethods: + analysis = _bind_arguments(thecallable) + if analysis.unbound_parameters: + if not _get_argument_type_mismatches(type_signature, analysis.bound_arguments): + return _keep_currying, analysis + # No matter which multimethod we pick, at least one parameter gets a binding + # that fails the type check. + _raise_multiple_dispatch_error(function, collected_args, collected_kwargs, + candidates=multimethods, _partial=True) # -------------------------------------------------------------------------------- From d8488702fbd1e7f56f24dd7f1f4030be20810a77 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 02:17:58 +0300 Subject: [PATCH 418/832] fix bug: `with test` sometimes injecting an unreachable `return`. Missing break in for-else, so whenever there was a `return` statement for the test already, it would inject a second, unreachable one, too. --- unpythonic/syntax/testingtools.py | 1 + 1 file changed, 1 insertion(+) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 4d7edb9b..116f4bc4 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -987,6 +987,7 @@ def _insert_funcname_here_(_insert_envname_here_): if not the_exprs and type(retval) is Compare: # inject the implicit the[] on the LHS retval.left = _inject_value_recorder(envname, retval.left) + break else: # When there is no return statement at the top level of the `with test` block, # we inject a `return True` to satisfy the test when the injected function From 1440d1712bd9d74698785a9febd41d8832c15c0d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 02:28:35 +0300 Subject: [PATCH 419/832] fix comments/docstrings: as of v0.15.0, the macro is `autocurry` --- unpythonic/fun.py | 2 +- unpythonic/syntax/lambdatools.py | 2 +- unpythonic/syntax/letdoutil.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 42ac4000..f1136272 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -432,7 +432,7 @@ def curried(*args, **kwargs): @passthrough_lazy_args def _currycall(f, *args, **kwargs): - """Co-operate with unpythonic.syntax.curry. + """Co-operate with unpythonic.syntax.autocurry. In a ``with autocurry`` block, we need to call `f` also when ``f()`` has transformed to ``curry(f)``, but definitions can be curried as usual. diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 4c6e23a9..abe4e50f 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -259,7 +259,7 @@ def iscurrywithfinallambda(tree): return type(tree.args[-1]) is Lambda # Detect an autocurry from an already expanded "with autocurry". - # CAUTION: These must match what unpythonic.syntax.curry.autocurry uses in its output. + # CAUTION: These must match what unpythonic.syntax.autocurry.autocurry uses in its output. currycall_name = "currycall" iscurryf = lambda name: name in ("curryf", "curry") # auto or manual curry in a "with autocurry" def isautocurrywithfinallambda(tree): diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 892d8ddc..8d630b30 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -18,7 +18,7 @@ letf_name = "letter" # must match what ``unpythonic.syntax.letdo._let_expr_impl`` uses in its output. dof_name = "dof" # name must match what ``unpythonic.syntax.letdo.do`` uses in its output. -currycall_name = "currycall" # output of ``unpythonic.syntax.curry`` +currycall_name = "currycall" # output of ``unpythonic.syntax.autocurry`` def _get_subscript_slice(tree): assert type(tree) is Subscript From 55581ddd120155d13c220c9dde10c1b880a25062 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 03:17:07 +0300 Subject: [PATCH 420/832] In `lazify`, always expand inner macros in recursive mode. This fixes a bug, for which the test case is: from mcpyrate.debug import macros, step_expansion from unpythonic.syntax import macros, test from unpythonic.syntax import macros, lazify, lazy with step_expansion: with lazify: with test: lazy[...] Without the recursive mode, the `lazy` won't get expanded before `lazify` performs its own AST edits (editing also `Subscript` nodes, thus breaking macro invocations), because it's nested inside the `with test` - and `test` expands outside-in, relying on the expander's recursive mode to expand any remaining inner macro invocations. (This makes debugging easier, because `step_expansion` will see it as a step.) A variant of this test case has been added to `test_lazify.py`. --- unpythonic/syntax/lazify.py | 5 +++- unpythonic/syntax/tests/test_lazify.py | 41 +++++++++++++++++++++++++- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index eae73d64..a6b7fa90 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -599,7 +599,10 @@ def _lazify(body): # Expand any inner macro invocations. Particularly, this expands away any `lazyrec[]` and `lazy[]` # so they become easier to work with. We also know that after this, any `Subscript` is really a # subscripting operation and not a macro invocation. - body = dyn._macro_expander.visit(body) + # + # We must explicitly use recursive mode to ensure we get rid of all macro invocations, because + # we may be running inside a `with step_expansion`, which uses the expand-once-only mode. + body = dyn._macro_expander.visit_recursively(body) # `lazify`'s analyzer needs the `ctx` attributes in `tree` to be filled in correctly. body = fix_ctx(body, copy_seen_nodes=False) # TODO: or maybe copy seen nodes? diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index d4f26c7f..bc6e1a3a 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -2,7 +2,10 @@ """Automatic lazy evaluation of function arguments.""" from ...syntax import macros, test, test_raises, error, the # noqa: F401 -from ...test.fixtures import session, testset +from ...test.fixtures import session, testset, returns_normally + +from mcpyrate.quotes import macros, q # noqa: F811 +from mcpyrate.compiler import run, temporary_module from ...syntax import (macros, lazify, lazy, lazyrec, # noqa: F811, F401 let, letseq, letrec, local, @@ -347,6 +350,42 @@ def f14(a, b): return f15(2 * a, 2 * b) test[f14(21, 1 / 0) == 42] + with testset("integration: expand nested inner macro invocations"): + # Here we need precise control over what the expander is doing, + # so we use `mcpyrate`'s run-time compiler access. + # + # Particularly, we need to enable expand-once mode to see whether the + # innermost macro expands correctly. This depends on `lazify` expanding + # inner macro invocations in recursive mode, regardless of the mode of + # the expander. + # + # If it doesn't, the innermost macro won't be expanded before `lazify` + # performs its own AST edits (editing also `Subscript` nodes), and in + # the result, it will no longer be a macro invocation, and will hence + # cause a `NameError` at run time. + # + # This block becomes a module. It's quoted, so macros won't expand + # when the parent module does. We're just constructing an AST here. + with q as quoted: + from mcpyrate.debug import macros, step_expansion # noqa: F811 + from unpythonic.syntax import macros, test # noqa: F811 + + from unpythonic.syntax import macros, lazify, lazy # noqa: F811, F401 + # TODO: This prints a lot of stuff, because that's its primary purpose. + # TODO: Here it would be nicer to use a macro that only enables expand-once mode. + with step_expansion: + with lazify: + # Here we need any macro that expands outside-in. The important thing is + # it doesn't recurse (`expander.visit`) on its own, instead relying on the + # expander's recursive mode to expand any remaining macro invocations inside + # the tree. Here we use `with test` in this dummy role. + with test: + lazy[...] # <-- this should get expanded, not raise NameError at run time + # And this is where we compile and run that AST, within a new temporary module. + # The filename should be descriptive, but not end in `.py`, since it's not an actual file. + with temporary_module(filename="tests in unpythonic.syntax.tests.test_lazify") as module: + test[returns_normally(run(quoted, module))] + # let bindings have a role similar to function arguments, so we auto-lazify there with testset("integration with let, letseq, letrec"): with lazify: From 001822da1ae8eaa306af65f5b5d432429591b736 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 03:35:26 +0300 Subject: [PATCH 421/832] meh, using run-time compiler access here breaks CI So let's do this the simple way. The drawback is the output is now printed when the whole module expands, not when this particular test runs. --- unpythonic/syntax/tests/test_lazify.py | 51 +++++++------------------- 1 file changed, 14 insertions(+), 37 deletions(-) diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index bc6e1a3a..2b48c3c5 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -2,10 +2,9 @@ """Automatic lazy evaluation of function arguments.""" from ...syntax import macros, test, test_raises, error, the # noqa: F401 -from ...test.fixtures import session, testset, returns_normally +from ...test.fixtures import session, testset -from mcpyrate.quotes import macros, q # noqa: F811 -from mcpyrate.compiler import run, temporary_module +from mcpyrate.debug import macros, step_expansion # noqa: F811 from ...syntax import (macros, lazify, lazy, lazyrec, # noqa: F811, F401 let, letseq, letrec, local, @@ -351,40 +350,18 @@ def f14(a, b): test[f14(21, 1 / 0) == 42] with testset("integration: expand nested inner macro invocations"): - # Here we need precise control over what the expander is doing, - # so we use `mcpyrate`'s run-time compiler access. - # - # Particularly, we need to enable expand-once mode to see whether the - # innermost macro expands correctly. This depends on `lazify` expanding - # inner macro invocations in recursive mode, regardless of the mode of - # the expander. - # - # If it doesn't, the innermost macro won't be expanded before `lazify` - # performs its own AST edits (editing also `Subscript` nodes), and in - # the result, it will no longer be a macro invocation, and will hence - # cause a `NameError` at run time. - # - # This block becomes a module. It's quoted, so macros won't expand - # when the parent module does. We're just constructing an AST here. - with q as quoted: - from mcpyrate.debug import macros, step_expansion # noqa: F811 - from unpythonic.syntax import macros, test # noqa: F811 - - from unpythonic.syntax import macros, lazify, lazy # noqa: F811, F401 - # TODO: This prints a lot of stuff, because that's its primary purpose. - # TODO: Here it would be nicer to use a macro that only enables expand-once mode. - with step_expansion: - with lazify: - # Here we need any macro that expands outside-in. The important thing is - # it doesn't recurse (`expander.visit`) on its own, instead relying on the - # expander's recursive mode to expand any remaining macro invocations inside - # the tree. Here we use `with test` in this dummy role. - with test: - lazy[...] # <-- this should get expanded, not raise NameError at run time - # And this is where we compile and run that AST, within a new temporary module. - # The filename should be descriptive, but not end in `.py`, since it's not an actual file. - with temporary_module(filename="tests in unpythonic.syntax.tests.test_lazify") as module: - test[returns_normally(run(quoted, module))] + # TODO: This prints a lot of stuff, because that's its primary purpose. + # TODO: Here it would be nicer to use a macro that only enables expand-once mode. + with step_expansion: + with lazify: + # Here we need any macro that expands outside-in. The important thing is + # it doesn't recurse (`expander.visit`) on its own, instead relying on the + # expander's recursive mode to expand any remaining macro invocations inside + # the tree. + # + # Here `with test` is nice, because it asserts the block returns normally at run time. + with test: + lazy[...] # <-- this should get expanded, not raise NameError at run time # let bindings have a role similar to function arguments, so we auto-lazify there with testset("integration with let, letseq, letrec"): From 9f410bbabcff12ec84e2b1a5d7bc8a63ac7dc71c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 2 Jun 2021 03:37:25 +0300 Subject: [PATCH 422/832] meh, fix comment --- unpythonic/syntax/tests/test_lazify.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 2b48c3c5..f24bc3d9 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -350,6 +350,16 @@ def f14(a, b): test[f14(21, 1 / 0) == 42] with testset("integration: expand nested inner macro invocations"): + # Here we need to enable expand-once mode to see whether the innermost + # macro expands correctly. This depends on `lazify` expanding inner + # macro invocations in recursive mode, regardless of the mode of the + # expander. + # + # If it doesn't, the innermost macro won't be expanded before `lazify` + # performs its own AST edits (editing also `Subscript` nodes), and in + # the result, it will no longer be a macro invocation, and will hence + # cause a `NameError` at run time. + # # TODO: This prints a lot of stuff, because that's its primary purpose. # TODO: Here it would be nicer to use a macro that only enables expand-once mode. with step_expansion: From 53614d26d6ff285f6b2dc290f9c1c8463283b08b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:11:51 +0300 Subject: [PATCH 423/832] consistent presentation order --- doc/dialects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects.md b/doc/dialects.md index 90349ec2..ab01d2ee 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -32,8 +32,8 @@ Hence *dialects*. As examples of what can be done with a dialects system together with a kitchen-sink language extension macro package such as `unpythonic`, we currently provide the following dialects: - [**Lispython**: The love child of Python and Scheme](dialects/lispython.md) - - [**Pytkell**: Because it's good to have a kell](dialects/pytkell.md) - [**Listhell**: It's not Lisp, it's not Python, it's not Haskell](dialects/listhell.md) + - [**Pytkell**: Because it's good to have a kell](dialects/pytkell.md) All three dialects support `unpythonic`'s ``continuations`` block macro, to add ``call/cc`` to the language; but it is not enabled automatically. From cde66637d2483a485e528365dbec2afd3155e2e8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:12:03 +0300 Subject: [PATCH 424/832] fix search'n'misplace --- doc/dialects/lispython.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 3874914a..cc039313 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -142,7 +142,7 @@ Lispython works with ``with continuations``, because: - The same applies to the outside-in pass of ``namedlambda``. Its inside-out pass, on the other hand, must come after ``continuations``, which it does, since the dialect's implicit ``with namedlambda`` is in a lexically outer position with respect to the ``with continuations``. -Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in factorial tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython happily auto-injects a ``return`` to whatever is the last statement in any particular function. +Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in fact tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython happily auto-injects a ``return`` to whatever is the last statement in any particular function. ## Why extend Python? From 5378033eef50eb6aa5d3b17426a72dfc4324434f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:12:10 +0300 Subject: [PATCH 425/832] fix wrong link --- doc/dialects/pytkell.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index b91ad174..4df72cc6 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -98,7 +98,7 @@ If you need more stuff, `unpythonic` is effectively the standard library of Pytk ## What Pytkell is -Pytkell is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.pytkell`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_pytkell.py). +Pytkell is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.pytkell`](../../unpythonic/dialects/pytkell.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_pytkell.py). Pytkell essentially makes Python feel slightly more haskelly. From 8ebe5757b65677e5e0ab75674c184e4ae5e84317 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:13:20 +0300 Subject: [PATCH 426/832] use visit_recursively to switch to inside-out processing Then the macros behave as expected also when the macro invocation using inside-out processing is inside a `with step_expansion` (which uses the expander's expand-once mode). --- doc/macros.md | 2 +- unpythonic/syntax/__init__.py | 4 ++-- unpythonic/syntax/autocurry.py | 2 +- unpythonic/syntax/autoref.py | 2 +- unpythonic/syntax/dbg.py | 2 +- unpythonic/syntax/forall.py | 2 +- unpythonic/syntax/lambdatools.py | 4 ++-- unpythonic/syntax/letdo.py | 8 ++++---- unpythonic/syntax/letsyntax.py | 10 +++++----- unpythonic/syntax/tailtools.py | 4 ++-- unpythonic/syntax/util.py | 8 ++++---- 11 files changed, 24 insertions(+), 24 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index a5149bb9..5e2d4cec 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2193,7 +2193,7 @@ with autoreturn: Of these, `autoreturn` expands outside-in, while `lazify` and `tco` are both two-pass macros. -We aim to improve the macro docs in the future. For now, to see if something is a two-pass macro, grep the codebase for `expander.visit`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). +We aim to improve the macro docs in the future. For now, to see if something is a two-pass macro, grep the codebase for `expander.visit_recursively`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). See our [notes on macros](../doc/design-notes.md#detailed-notes-on-macros) for more information. diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 8a78ddda..4d3f494b 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -58,13 +58,13 @@ # def mymacrointerface(tree, *, expander, *kw): # # perform your outside-in processing here # -# tree = expander.visit(tree) # recurse explicitly +# tree = expander.visit_recursively(tree) # recurse explicitly # # # perform your inside-out processing here # # return tree # -# If the line `tree = expander.visit(tree)` is omitted, the macro expands outside-in. +# If the line `tree = expander.visit_recursively(tree)` is omitted, the macro expands outside-in. # Note this default is different from MacroPy's! # TODO: 0.16: With `mcpyrate` we could start looking at values, not names, when the aim is to detect hygienically captured `unpythonic` constructs. See use sites of `isx`; refer to `mcpyrate.quotes.is_captured_value` and `mcpyrate.quotes.lookup_value`. diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 0fea4484..c9126742 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -68,7 +68,7 @@ def add3(a, b, c): if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("autocurry does not take an as-part") # pragma: no cover - tree = expander.visit(tree) + tree = expander.visit_recursively(tree) return _autocurry(block_body=tree) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 8fdbc407..53d923a7 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -164,7 +164,7 @@ def _autoref(block_body, args, asname): if not block_body: raise SyntaxError("expected at least one statement inside the 'with autoref' block") # pragma: no cover - block_body = dyn._macro_expander.visit(block_body) + block_body = dyn._macro_expander.visit_recursively(block_body) # second pass, inside-out diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 13f0891f..98cae8fd 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -103,7 +103,7 @@ def dbg(tree, *, args, syntax, expander, **kw): if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("dbg (block mode) does not take an as-part") # pragma: no cover - tree = expander.visit(tree) + tree = expander.visit_recursively(tree) if syntax == "expr": return _dbg_expr(tree) diff --git a/unpythonic/syntax/forall.py b/unpythonic/syntax/forall.py index 95e4337e..e956d037 100644 --- a/unpythonic/syntax/forall.py +++ b/unpythonic/syntax/forall.py @@ -35,7 +35,7 @@ def forall(tree, *, syntax, expander, **kw): if syntax != "expr": raise SyntaxError("forall is an expr macro only") # pragma: no cover - tree = expander.visit(tree) + tree = expander.visit_recursively(tree) return _forall(exprs=tree) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index abe4e50f..d30b54e2 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -376,7 +376,7 @@ def transform(self, tree): # outside in: transform in unexpanded let[] forms newbody = NamedLambdaTransformer().visit(block_body) - newbody = dyn._macro_expander.visit(newbody) + newbody = dyn._macro_expander.visit_recursively(newbody) # inside out: transform in expanded autocurry newbody = NamedLambdaTransformer().visit(newbody) @@ -437,7 +437,7 @@ def _envify(block_body): # first pass, outside-in userlambdas = detect_lambda(block_body) - block_body = dyn._macro_expander.visit(block_body) + block_body = dyn._macro_expander.visit_recursively(block_body) # second pass, inside-out def getargs(tree): # tree: FunctionDef, AsyncFunctionDef, Lambda diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index f435cb41..ee007f1e 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -376,14 +376,14 @@ def _let_expr_impl(bindings, body, mode): # (It is important we expand at least that immediately after, to resolve its local variables, # because those may have the same lexical names as some of the let-bindings.) body = _implicit_do(body) - body = dyn._macro_expander.visit(body) + body = dyn._macro_expander.visit_recursively(body) if not bindings: # Optimize out a `let` with no bindings. The macro layer cannot trigger # this case, because our syntaxes always require at least one binding. # So this check is here just to protect against use with no bindings directly # from other syntax transformers, which in theory could attempt anything. return body # pragma: no cover - bindings = dyn._macro_expander.visit(bindings) + bindings = dyn._macro_expander.visit_recursively(bindings) names, values = zip(*[b.elts for b in bindings]) # --> (k1, ..., kn), (v1, ..., vn) names = [getname(k, accept_attr=False) for k in names] # any duplicates will be caught by env at run-time @@ -510,13 +510,13 @@ def _let_decorator_impl(bindings, body, mode, kind): assert kind in ("decorate", "call") if type(body) not in (FunctionDef, AsyncFunctionDef): raise SyntaxError("Expected a function definition to decorate") # pragma: no cover - body = dyn._macro_expander.visit(body) + body = dyn._macro_expander.visit_recursively(body) if not bindings: # Similarly as above, this cannot trigger from the macro layer no # matter what that layer does. This is here to optimize away a `dlet` # with no bindings, when used directly from other syntax transformers. return body # pragma: no cover - bindings = dyn._macro_expander.visit(bindings) + bindings = dyn._macro_expander.visit_recursively(bindings) names, values = zip(*[b.elts for b in bindings]) # --> (k1, ..., kn), (v1, ..., vn) names = [getname(k, accept_attr=False) for k in names] # any duplicates will be caught by env at run-time diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 221032cd..1a52807d 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -235,8 +235,8 @@ def register_bindings(): target.append((name, args, value, "expr")) if expand_inside: - bindings = dyn._macro_expander.visit(bindings) - body = dyn._macro_expander.visit(body) + bindings = dyn._macro_expander.visit_recursively(bindings) + body = dyn._macro_expander.visit_recursively(body) register_bindings() body = _substitute_templates(templates, body) body = _substitute_barenames(barenames, body) @@ -340,7 +340,7 @@ def isbinding(tree): # `let_syntax` mode (expand_inside): respect lexical scoping of nested `let_syntax`/`abbrev` expanded = False if expand_inside and (is_let_syntax(stmt) or is_abbrev(stmt)): - stmt = dyn._macro_expander.visit(stmt) + stmt = dyn._macro_expander.visit_recursively(stmt) expanded = True stmt = _substitute_templates(templates, stmt) @@ -351,14 +351,14 @@ def isbinding(tree): check_stray_blocks_and_exprs(value) # before expanding it! if expand_inside and not expanded: - value = dyn._macro_expander.visit(value) + value = dyn._macro_expander.visit_recursively(value) target = templates if args else barenames target.append((name, args, value, mode)) else: check_stray_blocks_and_exprs(stmt) # before expanding it! if expand_inside and not expanded: - stmt = dyn._macro_expander.visit(stmt) + stmt = dyn._macro_expander.visit_recursively(stmt) new_block_body.append(stmt) new_block_body = eliminate_ifones(new_block_body) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index dcca5a8f..5ee2760b 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -700,7 +700,7 @@ def _tco(block_body): userlambdas = detect_lambda(block_body) known_ecs = list(uniqify(detect_callec(block_body))) - block_body = dyn._macro_expander.visit(block_body) + block_body = dyn._macro_expander.visit_recursively(block_body) # second pass, inside-out transform_retexpr = partial(_transform_retexpr) @@ -781,7 +781,7 @@ def _continuations(block_body): known_ecs = list(uniqify(detect_callec(block_body))) with _continuations_level.changed_by(+1): - block_body = dyn._macro_expander.visit(block_body) + block_body = dyn._macro_expander.visit_recursively(block_body) # second pass, inside-out diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 78c697d7..f1afbf14 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -90,12 +90,12 @@ def detect_lambda(tree): """Find lambdas in tree. Helper for two-pass block macros. A two-pass block macro first performs some processing outside-in, then calls - `expander.visit(tree)` to make any nested macro invocations expand, and then - performs some processing inside-out. + `expander.visit_recursively(tree)` to make any nested macro invocations expand, + and then performs some processing inside-out. Run ``detect_lambda(tree)`` in the outside-in pass, before calling - `expander.visit(tree)`, because nested macro invocations may generate - more lambdas that your block macro is not interested in. + `expander.visit_recursively(tree)`, because nested macro invocations + may generate more lambdas that your block macro is not interested in. The return value is a ``list``of ``id(lam)``, where ``lam`` is a Lambda node that appears in ``tree``. This list is suitable as ``userlambdas`` for the From d4e2a4f65acb58674be2412ac93cf3abb8be28bf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:14:48 +0300 Subject: [PATCH 427/832] update comment --- unpythonic/syntax/lazify.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index a6b7fa90..0115805c 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -576,10 +576,10 @@ def _is_literal_container(tree, maps_only=False): # it is too easy to accidentally set up an infinite recursion. # # This is ok: -# force1(lst)[0] = (10 * (force1(lst()[0]) if isinstance(lst, Lazy1) else force1(lst[0]))) +# force1(lst)[0] = (10 * (force1(lst()[0]) if isinstance(lst, Lazy) else force1(lst[0]))) # # but this blows up (by infinite recursion) later when we eventually force lst[0]: -# force1(lst)[0] = Lazy1(lambda: (10 * (force1(lst()[0]) if isinstance(lst, Lazy1) else force1(lst[0])))) +# force1(lst)[0] = Lazy(lambda: (10 * (force1(lst()[0]) if isinstance(lst, Lazy) else force1(lst[0])))) # # We **could** solve this by forcing and capturing the current value before assigning, # instead of allowing the RHS to refer to a lazy list element. But on the other hand, From ee0ed12b57f9c3aea47f4d95d1dbe13fa6b02312 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:19:03 +0300 Subject: [PATCH 428/832] document caveat: @generic/@typed are not compatible with lazify --- doc/features.md | 3 +++ unpythonic/dispatch.py | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/doc/features.md b/doc/features.md index b8e7fc28..e032dfbf 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3098,6 +3098,9 @@ In `unpythonic`, the terminology is as follows: The term *multimethod* distinguishes them from the OOP sense of *method*, already established in Python, as well as reminds that multiple arguments participate in dispatching. +**CAUTION**: Code using the `with lazify` macro cannot usefully use `@generic` or `@typed`, because all arguments of each function call will be wrapped in a promise (`unpythonic.lazyutil.Lazy`) that carries no type information on its contents. + + #### ``generic``: multiple dispatch with type annotation syntax The ``generic`` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. This not only kills boilerplate, but makes the dispatch extensible, since the dispatcher lives outside the original function definition. There is no need to monkey-patch the original to add a new case. diff --git a/unpythonic/dispatch.py b/unpythonic/dispatch.py index a0268ecc..5f06e414 100644 --- a/unpythonic/dispatch.py +++ b/unpythonic/dispatch.py @@ -234,6 +234,10 @@ def example(): See the limitations in `unpythonic.typecheck` for which features of the `typing` module are supported and which are not. + + Code using the `with lazify` macro cannot usefully use `@generic` or `@typed`, + because all arguments of each function call will be wrapped in a promise + (`unpythonic.lazyutil.Lazy`) that carries no type information on its contents. """ return _setup(_function_fullname(f), f) @@ -299,6 +303,12 @@ def typed(f): Once a `@typed` function has been created, no more multimethods can be attached to it. + + **CAUTION**: + + Code using the `with lazify` macro cannot usefully use `@generic` or `@typed`, + because all arguments of each function call will be wrapped in a promise + (`unpythonic.lazyutil.Lazy`) that carries no type information on its contents. """ s = generic(f) del s._register # remove the ability to register more methods From 7d76f3cd4b177424676ff1c9fda8bf1fba490f3e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:31:22 +0300 Subject: [PATCH 429/832] continuations doesn't support async, so raise an error It never has supported async; this is a robustness improvement. --- CHANGELOG.md | 1 + unpythonic/syntax/tailtools.py | 17 ++++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d541cdf2..be326178 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -177,6 +177,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - This change fixes a `flake8` [E741](https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes) warning, and the new name for the parameter is more descriptive. - **Miscellaneous.** + - Robustness: the `with continuations` macro now raises `SyntaxError` if async constructs (`async def` or `await`) appear lexically inside the block, because interaction of `with continuations` with Python's async subsystem has never been implemented. See [issue #4](https://github.com/Technologicat/unpythonic/issues/4). - The functions `raisef`, `tryf`, `equip_with_traceback`, and `async_raise` now live in `unpythonic.excutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - The functions `call` and `callwith` now live in `unpythonic.funutil`. They are still available in the top-level namespace of `unpythonic`, as usual. - The functions `almosteq`, `fixpoint`, `partition_int`, and `ulp` now live in `unpythonic.numutil`. They are still available in the top-level namespace of `unpythonic`, as usual. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 5ee2760b..54f9eab7 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -15,6 +15,7 @@ Call, Name, Starred, Constant, BoolOp, And, Or, With, AsyncWith, If, IfExp, Try, Assign, Return, Expr, + Await, copy_location) import sys @@ -1059,17 +1060,31 @@ def transform_callcc(owner, body): class StrayCallccChecker(ASTVisitor): def examine(self, tree): if iscallcc(tree): - raise SyntaxError("call_cc[...] only allowed at the top level of a def or async def, or at the top level of the block; must appear as an expr or an assignment RHS") # pragma: no cover + raise SyntaxError("call_cc[...] only allowed at the top level of a def, or at the top level of the block; must appear as an expr or an assignment RHS") # pragma: no cover if type(tree) in (Assign, Expr): v = tree.value if type(v) is Call and type(v.func) is Name and v.func.id == "call_cc": raise SyntaxError("call_cc(...) should be call_cc[...] (note brackets; it's a macro)") # pragma: no cover self.generic_visit(tree) + # TODO: Interaction of `continuations` with async functions is not implemented. + # So for robustness, we raise a syntax error for now. + class AsyncDefChecker(ASTVisitor): + def examine(self, tree): + if type(tree) is AsyncFunctionDef: + raise SyntaxError("`with continuations` does not currently support `async` functions") + elif type(tree) is AsyncWith: + raise SyntaxError("`with continuations` does not currently support `async` context managers") + elif type(tree) is Await: + raise SyntaxError("`with continuations` does not currently support `await`") + self.generic_visit(tree) + # ------------------------------------------------------------------------- # Main processing logic begins here # ------------------------------------------------------------------------- + AsyncDefChecker().visit(block_body) + # Disallow return at the top level of the block, because it would behave # differently depending on whether placed before or after the first call_cc[] # invocation. (Because call_cc[] internally creates a function and calls it.) From 492fa1df6cc245b309896ff6a5ea9de91273b08b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:54:05 +0300 Subject: [PATCH 430/832] Move the explicit recursion into the syntax transformer functions That's where it should be (not in the macro interface!), in case other macros call those syntax transformers. --- unpythonic/syntax/autocurry.py | 8 +++++--- unpythonic/syntax/dbg.py | 18 ++++++++++++------ unpythonic/syntax/forall.py | 11 ++++++++--- 3 files changed, 25 insertions(+), 12 deletions(-) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index c9126742..ea8c398c 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -12,6 +12,8 @@ from .util import (suggest_decorator_index, isx, has_curry, sort_lambda_decorators) +from ..dynassign import dyn + # CAUTION: unpythonic.syntax.lambdatools.namedlambda depends on the exact names # "curryf" and "currycall" to detect an auto-curried expression with a final lambda. from ..fun import curry as curryf, _currycall as currycall @@ -68,9 +70,8 @@ def add3(a, b, c): if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("autocurry does not take an as-part") # pragma: no cover - tree = expander.visit_recursively(tree) - - return _autocurry(block_body=tree) + with dyn.let(_macro_expander=expander): + return _autocurry(block_body=tree) _iscurry = lambda name: name in ("curry", "currycall") @@ -129,5 +130,6 @@ def transform(self, tree): return self.generic_visit(tree) + block_body = dyn._macro_expander.visit_recursively(block_body) newbody = AutoCurryTransformer(hascurry=False).visit(block_body) return sort_lambda_decorators(newbody) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 98cae8fd..0eb10d72 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -103,12 +103,12 @@ def dbg(tree, *, args, syntax, expander, **kw): if syntax == "block" and kw['optional_vars'] is not None: raise SyntaxError("dbg (block mode) does not take an as-part") # pragma: no cover - tree = expander.visit_recursively(tree) - - if syntax == "expr": - return _dbg_expr(tree) - else: # syntax == "block": - return _dbg_block(body=tree, args=args) + # Expand inside-out. + with dyn.let(_macro_expander=expander): + if syntax == "expr": + return _dbg_expr(tree) + else: # syntax == "block": + return _dbg_block(body=tree, args=args) def dbgprint_block(ks, vs, *, filename=None, lineno=None, sep=", ", **kwargs): """Default debug printer for the ``dbg`` macro, block variant. @@ -213,6 +213,9 @@ def _dbg_block(body, args): pfunc = q[h[dbgprint_block]] pname = "print" # override standard print function within this block + # TODO: Do we really need to expand inside-out here? + body = dyn._macro_expander.visit_recursively(body) + class DbgBlockTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): @@ -231,6 +234,9 @@ def transform(self, tree): return DbgBlockTransformer().visit(body) def _dbg_expr(tree): + # TODO: Do we really need to expand inside-out here? + tree = dyn._macro_expander.visit_recursively(tree) + ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] filename = q[h[callsite_filename]()] # Careful here! We must `h[]` the `dyn`, but not `dbgprint_expr` itself, diff --git a/unpythonic/syntax/forall.py b/unpythonic/syntax/forall.py index e956d037..f85ed0b1 100644 --- a/unpythonic/syntax/forall.py +++ b/unpythonic/syntax/forall.py @@ -11,6 +11,7 @@ from .letdoutil import isenvassign, UnexpandedEnvAssignView from ..amb import monadify +from ..dynassign import dyn from ..misc import namelambda from ..amb import insist, deny # for re-export only # noqa: F401 @@ -35,13 +36,17 @@ def forall(tree, *, syntax, expander, **kw): if syntax != "expr": raise SyntaxError("forall is an expr macro only") # pragma: no cover - tree = expander.visit_recursively(tree) - - return _forall(exprs=tree) + # Inside-out macro. + with dyn.let(_macro_expander=expander): + return _forall(exprs=tree) def _forall(exprs): if type(exprs) is not Tuple: # pragma: no cover, let's not test macro expansion errors. raise SyntaxError("forall body: expected a sequence of comma-separated expressions") # pragma: no cover + + # Expand inside-out to easily support lexical scoping. + exprs = dyn._macro_expander.visit_recursively(exprs) + itemno = 0 def build(lines, tree): if not lines: From 29d819ec4ea30a2a23e0ea249b7be77c791a6de9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 01:59:24 +0300 Subject: [PATCH 431/832] add docstring; improve variable naming --- unpythonic/syntax/autoref.py | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 53d923a7..e1119730 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -151,10 +151,20 @@ def autoref(tree, *, args, syntax, expander, **kw): @passthrough_lazy_args def _autoref_resolve(args): - *objs, s = [force1(x) for x in args] + """Perform an autoref lookup in a `with autoref` block. + + `args`: list [obj0, ..., objN, attrname] + + Each `obj` is tried, left to right, and the first one that + `hasattr(obj, attrname)` wins. The return value is the tuple + `(True, getattr(obj, attrname))`. + + If no obj matches, the return value is `(False, None)`. + """ + *objs, attrname = [force1(x) for x in args] for o in objs: - if hasattr(o, s): - return True, force1(getattr(o, s)) + if hasattr(o, attrname): + return True, force1(getattr(o, attrname)) return False, None def _autoref(block_body, args, asname): From b880a1cd7ba3c0627e17d6ff289415f06b2895aa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 02:09:25 +0300 Subject: [PATCH 432/832] improve comments --- unpythonic/syntax/lambdatools.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index d30b54e2..f21a9cdc 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -437,6 +437,7 @@ def _envify(block_body): # first pass, outside-in userlambdas = detect_lambda(block_body) + # Expand inside-out to easily support lexical scoping. block_body = dyn._macro_expander.visit_recursively(block_body) # second pass, inside-out @@ -520,6 +521,10 @@ def isourupdate(thecall): newvalue = self.visit(view.value) return q[a[envset](u[view.name], a[newvalue])] # transform references to currently active bindings + # x --> e14.x + # It doesn't matter if this hits an already expanded inner `with envify`, + # because the gensymmed environment name won't be in our bindings, and the "x" + # has become the `attr` in an `Attribute` node. elif type(tree) is Name and tree.id in bindings.keys(): # We must be careful to preserve the Load/Store/Del context of the name. # The default lets `mcpyrate` fix it later. From 5d6a29fdd300928832b0f59b9095f538ae244b90 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 02:22:34 +0300 Subject: [PATCH 433/832] Complete the xmas tree combo doc for macros --- doc/macros.md | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 5e2d4cec..47f1490f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2148,7 +2148,7 @@ The macros in ``unpythonic.syntax`` are designed to work together, but some care For simplicity, **the block macros make no attempt to prevent invalid combos** (unless there is a specific technical reason to do that for some particular combination). Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. -The **AST edits** performed by the block macros are designed to run **in the following order (leftmost first)**: +The **AST edits** performed by the block macros are designed to run in the following order (leftmost first): ``` prefix > autoreturn, quicklambda > multilambda > continuations or tco > ... @@ -2167,13 +2167,27 @@ with mac: ... ``` -The invocation `with mac` is *lexically on the outside*, thus the macro expander sees it first. The expansion order is then: +The invocation `with mac` is *lexically on the outside*, thus the macro expander sees it first. The expansion order then becomes: 1. First pass (outside in) of `with mac`. 2. Explicit recursion by `with mac`. This expands the `with cheese`. 3. Second pass (inside out) of `with mac`. -So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it is actually a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, even though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. See [the dialect examples](../unpythonic/dialects/) for combo invocations that are known to work. +So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it happens to be a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, even though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. + +Considering that: + + - Outside-in: `prefix`, `autoreturn`, `quicklambda`, `multilambda` + - Two-pass: `envify`, `lazify`, `namedlambda`, `autoref`, `autocurry`, `tco`/`continuations` + +the correct **xmas tree invocation** is: + +```python +with prefix, autoreturn, quicklambda, multilambda, envify, lazify, namedlambda, autoref, autocurry, tco: + ... +``` + +[The dialect examples](dialects.md) use this ordering. See our [notes on macros](design-notes.md#detailed-notes-on-macros) for some more details. Example combo in the single-line format: @@ -2191,16 +2205,13 @@ with autoreturn: ... ``` -Of these, `autoreturn` expands outside-in, while `lazify` and `tco` are both two-pass macros. - -We aim to improve the macro docs in the future. For now, to see if something is a two-pass macro, grep the codebase for `expander.visit_recursively`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). - -See our [notes on macros](../doc/design-notes.md#detailed-notes-on-macros) for more information. - **NOTE**: In MacroPy, there sometimes were [differences](https://github.com/azazel75/macropy/issues/21) between the behavior of the single-line and multi-line invocation format, but in `mcpyrate`, they should behave the same. With `mcpyrate`, there is still [a minor difference](https://github.com/Technologicat/mcpyrate/issues/3) if there are at least three nested macro invocations, and a macro is scanning the tree for another macro invocation; then the tree looks different depending on whether the single-line or the multi-line format was used. The differences in that are as one would expect knowing [how `with` statements look like](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#With) in the Python AST. The reason the difference manifests only for three or more macro invocations is that `mcpyrate` pops the macro that is being expanded before it hands over the tree to the macro code; hence if there are only two, the inner tree will have only one "context manager" in its `with`. +**NOTE** to the curious, and to future documentation maintainers: To see if something is a two-pass macro, grep the codebase for `expander.visit_recursively`; that is the *explicit recursion* mentioned above, and means that within that function, anything below that line will run in the inside-out pass. See [the `mcpyrate` manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md#expand-macros-inside-out). + + ### Emacs syntax highlighting This Elisp snippet can be used to add syntax highlighting for keywords specific to `mcpyrate` and `unpythonic.syntax` to your Emacs setup: From e9f0243341420ab0bdc69058a207e601086cfce5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 02:22:38 +0300 Subject: [PATCH 434/832] use the advertised xmas combo order for macro invocations --- unpythonic/dialects/lispython.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index 32c50cf4..63d94e17 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -37,7 +37,7 @@ def transform_ast(self, tree): # tree is an ast.Module let_syntax, abbrev, block, expr, cond) from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn, Values # noqa: F401, F811 - with autoreturn, quicklambda, multilambda, tco, namedlambda: + with autoreturn, quicklambda, multilambda, namedlambda, tco: __paste_here__ # noqa: F821, just a splicing marker. tree.body = splice_dialect(tree.body, template, "__paste_here__") return tree From 3de737079e7f8e71268dd46071ad393ffdf2285d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 7 Jun 2021 02:59:11 +0300 Subject: [PATCH 435/832] update hovercraft full of eels essay --- doc/design-notes.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 77272724..de5b5359 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -82,25 +82,31 @@ If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.co Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). -In general, I like Python, and my hat's off to the devs. It's no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the particular points above, if I agreed, I wouldn't be doing this, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. +In general, I like Python. Also, my hat is off to the devs. It is no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the particular points above, if I agreed, I would not have built `unpythonic`, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. -I think that with macros, Python can be so much more than just a beginner's language, and that language-level extensibility is the logical endpoint of that. I don't get the sentiment against metaprogramming, or toward some language-level features. For me, macros (and full-module transforms a.k.a. dialects) are just another tool for creating abstractions, at yet another level. We can already extract procedures, methods, and classes. Why limit that ability - namely, the ability to create abstractions - to what an [eager](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) language can express at run time? If the point is to keep code understandable, then it's a matter of education. It's perfectly possible to write unreadable code without macros, and in Python, no less. And it's perfectly possible to write readable code with macros. I'm willing to admit the technical objection that *macros don't compose*; but that doesn't make them useless. +I think that with macros, Python can be so much more than just a beginner's language. Language-level extensibility is just the logical endpoint of that. I do not share the sentiment of the Python community against metaprogramming, or toward some language-level features. For me, macros (and full-module transforms a.k.a. dialects) are just another tool for creating abstractions, at yet another level. We can already extract procedures, methods, and classes. Why limit that ability - namely, the ability to create abstractions - to what an [eager](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) language can express at run time? -Of the particular points above, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hang, when entering an accidentally infinite mutual recursion. I do occasionally make such mistakes when developing complex algorithms. But sometimes, I'd like to enable TCO selectively. If you ask for it, you know what to expect. This is precisely why `unpythonic.syntax` has `with tco`. I'm not very happy with having a custom TCO layer on top of a language core that doesn't like the idea, because TCO support in the core (like Scheme and Racket have) would simplify the implementation of certain other language extensions; but then again, [this is exactly what Clojure did](https://clojuredocs.org/clojure.core/trampoline), too. +If the point is to keep code understandable, I respect the goal; but that is a matter of education. It is perfectly possible to write unreadable code without macros, and in Python, no less. And it is perfectly possible to write readable code with macros. I am willing to admit the technical objection that *macros do not compose*; but that does not make them useless. -I think a multi-expression `lambda` is, on the surface, a good idea, but really the issue is that Python's `lambda` construct itself is broken. It's essentially a duplicate of `def`, but lacking some features. We would be much better off if `def` was an expression. Much of the time, anonymous functions aren't such a great idea, but defining closures inline is - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. (Also, why are lambdas strictly anonymous? In cases where it is useful to be able to omit a name (because sometimes there are many small helpers and [naming is hard](https://martinfowler.com/bliki/TwoHardThings.html)), why not include the source location information in the auto-generated name, instead of just `""`?) +Of the particular points above, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hang, when entering an accidentally infinite mutual recursion. I do occasionally make such mistakes when developing complex algorithms - especially when quickly sketching out new ideas. But sometimes, it would be nice to enable TCO selectively. If you ask for it, you know what to expect. This is precisely why `unpythonic.syntax` has `with tco`. I am not very happy with a custom TCO layer on top of a language core that eschews the whole idea, because TCO support in the core (like Scheme and Racket have) would simplify the implementation of certain other language extensions; but then again, [this is exactly what Clojure did](https://clojuredocs.org/clojure.core/trampoline), in similar technical circumstances. -The macros in `unpythonic.syntax` inject lots of lambdas, because that makes them much simpler to implement than if we had to always lift a `def` statement into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if you could define also slightly more complex actions inline (see [Racket's pattern matcher](https://docs.racket-lang.org/reference/match.html) for a comparison). It's unlikely you'll need the action functions elsewhere, and it's just silly to define a bunch of functions *before* the call to `match`. If this isn't a job for either something like `let-where` (to invert the presentation order locally) or multi-expression lambdas (to define the actions inline), I don't know what is. +As for a multi-expression `lambda`, on the surface it sounds like a good idea. But really the issue is that in Python, the `lambda` construct itself is broken. It is essentially a duplicate of `def`, but lacking some features. As of Python 3.8, the latest insult to injury is the lack of support for type annotations. A more uniform solution would be to make `def` into an expression. Much of the time, anonymous functions are not a good idea, because names help understanding and debugging - especially when all you have is a traceback. But defining closures inline **is** a great idea - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. + +The macros in `unpythonic.syntax` inject many lambdas, because that makes them much simpler to implement than if we had to always lift a `def` statement into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if one could define also slightly more complex actions inline (see [Racket's pattern matcher](https://docs.racket-lang.org/reference/match.html) for a comparison). It is unlikely that the action functions will be needed elsewhere, and it is just silly to define a bunch of functions *before* the call to `match`. If this is not a job for either something like `let-where` (to invert the presentation order locally) or a multi-expression lambda (to define the actions inline), I do not know what is. + +While on the topic of usability, why are lambdas strictly anonymous? In cases where it is useful to be able to omit a name, because sometimes many small helper functions may be needed and [naming is hard](https://martinfowler.com/bliki/TwoHardThings.html), why not include the source location information in the auto-generated name, instead of just `""`? (As of v0.15.0, the `with namedlambda` macro does this.) On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the parser mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I don't want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) -As for true multi-shot continuations... `unpythonic.syntax` has `with continuations` for that, but I'm not sure if I'll ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. However, the feature is great to have for teaching the concept of continuations in a programming course, when teaching in Python. For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. Python's generators) are often all that's needed to simplify certain patterns, especially those involving backtracking. I'm a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! +As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant one, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. + +Finally, what to think of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/))? It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its users, and it is not very popular in the grand *scheme* of things (pun not intended). -Finally, how about subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/))? It is pretty much the point of language-level extensibility, to allow users to do that if they want. I wouldn't worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its users, and it is not very popular; it's hard to say what the programming community at large would do with an extensible language. +What I can say is, `unpythonic` is not meant for the average Python project, either. If used intelligently, it can make code shorter, yet readable. For a lone developer who needs to achieve as much as possible in the fewest lines reasonably possible, it seems to me that language extension - and in general, as Alexis King put it, [climbing the infinite ladder of abstraction](https://lexi-lambda.github.io/blog/2016/08/11/climbing-the-infinite-ladder-of-abstraction/) - is the way to go. In a large project with a high developer turnover, the optimal solution is different. -What I can say is, `unpythonic` is not meant for the average Python project, either. But if used intelligently, it can make your code shorter, yet readable. Obviously, in a large project with a high developer turnover, the optimal solution looks different. +For general programming in the early 2020s, Python has the ecosystem advantage, so it does not make sense to move to anything else, at least yet. So, let us empower what we have. Even if, in order to achieve that, we have build something that could be considered *unpythonic*. ## Killer features of Common Lisp From fefd9f0be4dec122cf0ebff048388c106159e067 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 8 Jun 2021 23:25:39 +0300 Subject: [PATCH 436/832] macro name: autocurry --- doc/dialects/lispython.md | 2 +- doc/dialects/listhell.md | 2 +- doc/dialects/pytkell.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index cc039313..a4ccf3ae 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -117,7 +117,7 @@ The aforementioned block macros are enabled implicitly for the whole module; thi Of the other block macros in ``unpythonic.syntax``, code written in Lispython supports only ``continuations``. ``autoref`` should also be harmless enough (will expand too early, but shouldn't matter). -``prefix``, ``curry``, ``lazify`` and ``envify`` are **not compatible** with the ordering of block macros implicit in the Lispython dialect. +``prefix``, ``autocurry``, ``lazify`` and ``envify`` are **not compatible** with the ordering of block macros implicit in the Lispython dialect. ``prefix`` is an outside-in macro that should expand first, so it should be placed in a lexically outer position with respect to the ones Lispython invokes implicitly; but nothing can be more outer than the dialect template. diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index f171320b..2e956da5 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -71,7 +71,7 @@ It's also a minimal example of how to make an AST-transforming dialect. ## Comboability -Only outside-in macros that should expand after ``curry`` (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``curry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Listhell dialect. +Only outside-in macros that should expand after ``autocurry`` (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``autocurry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Listhell dialect. ## Notes diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index 4df72cc6..cd3dccdd 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -69,7 +69,7 @@ assert x == 42 ## Features -In terms of ``unpythonic.syntax``, we implicitly enable ``curry`` and ``lazify`` for the whole module. +In terms of ``unpythonic.syntax``, we implicitly enable ``autocurry`` and ``lazify`` for the whole module. We also import some macros and functions to serve as dialect builtins: @@ -107,9 +107,9 @@ It's also a minimal example of how to make an AST-transforming dialect. ## Comboability -**Not** comboable with most of the block macros in ``unpythonic.syntax``, because ``curry`` and ``lazify`` appear in the dialect template, hence at the lexically outermost position. +**Not** comboable with most of the block macros in ``unpythonic.syntax``, because ``autocurry`` and ``lazify`` appear in the dialect template, hence at the lexically outermost position. -Only outside-in macros that should expand after ``lazify`` has recorded its userlambdas (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``curry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Pytkell dialect. +Only outside-in macros that should expand after ``lazify`` has recorded its userlambdas (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``autocurry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Pytkell dialect. ## CAUTION From 655086c8ca8f5ffa2feb64ebf5414e7ff25f42eb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 8 Jun 2021 23:26:08 +0300 Subject: [PATCH 437/832] wording, formatting --- doc/design-notes.md | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index de5b5359..3dba46a0 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -54,6 +54,7 @@ Finally, when the whole purpose of the feature is to automatically transform a p When to implement your own feature as a syntactic macro, see the discussion in Chapter 8 of [Paul Graham: On Lisp](http://paulgraham.com/onlisp.html). MacroPy's documentation also provides [some advice on the topic](https://macropy3.readthedocs.io/en/latest/discussion.html). + ## Macros do not Compose Making macros work together is nontrivial, essentially because *macros don't compose*. [As pointed out by John Shutt](https://fexpr.blogspot.com/2013/12/abstractive-power.html), in a multilayered language extension implemented with macros, the second layer of macros needs to understand all of the first layer. The issue is that the macro abstraction leaks the details of its expansion. Contrast with functions, which operate on values: the process that was used to arrive at a value doesn't matter. It's always possible for a function to take this value and transform it into another value, which can then be used as input for the next layer of functions. That's composability at its finest. @@ -66,16 +67,18 @@ Some aspects in the design of `unpythonic` could be simplified by expanding macr The lack of composability is a problem mainly when using macros to create a language extension, because the features of the extended language often interact. Macros can also be used in a much more everyday way, where composability is mostly a non-issue - to abstract and name common patterns that just happen to be of a nature that cannot be extracted as a regular function. See [Peter Seibel: Practical Common Lisp, chapter 3](http://www.gigamonkeys.com/book/practical-a-simple-database.html) for an example. + ## Language Discontinuities The very act of extending a language creates points of discontinuity between the extended language and the original. This can become a particularly bad source of extra complexity, if the extension can be enabled locally for a piece of code - as is the case with block macros. Then the design of the extended language must consider how to treat interactions between pieces of code that use the extension and those that don't. Then exponentiate those design considerations by the number of extensions that can be enabled independently. This issue is simply absent when designing a new language from scratch. For an example, look at what the rest of `unpythonic` has to do to make `lazify` behave as the user expects! Grep the codebase for `lazyutil`; especially the `passthrough_lazy_args` decorator, and its sister, the utility `maybe_force_args`. The decorator is essentially just an annotation for the `lazify` transformer, that marks a function as *not necessarily needing* evaluation of its arguments. Such functions often represent language-level constructs, such as `let` or `curry`, that essentially just *pass through* user data to other user-provided code, without *accessing* that data. The annotation is honored by the compiler when programming in the lazy (call-by-need) extended language, and otherwise it does nothing. Another pain point is the need of a second trampoline implementation (that only differs in one minor detail) just to make `lazify` interact correctly with TCO (while not losing an order of magnitude of performance in the trampoline used with standard Python). -For another example, it's likely that e.g. `continuations` still doesn't integrate completely seamlessly - and I'm not sure if that is possible even in principle. Calling a traditional function from a [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style) function is no problem; the traditional function uses no continuations, and (barring exceptions) will always return normally. The other way around can be a problem. Also, having TCO implemented as a trampoline system on top of the base language (instead of being already provided under the hood, like in Scheme) makes the `continuations` transformer more complex than absolutely necessary. +For another example, it is likely that e.g. `continuations` still does not integrate completely seamlessly - and I am not sure if that is possible even in principle. Calling a traditional function from a [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style) function is no problem; the traditional function uses no continuations, and (barring exceptions) will always return normally. The other way around can be a problem. Also, having TCO implemented as a trampoline system on top of the base language (instead of being already provided under the hood, like in Scheme) makes the `continuations` transformer more complex than absolutely necessary. For a third example, consider *decorated lambdas*. This is an `unpythonic` extension - essentially, a compiler feature implemented (by calling some common utility code) by each of the transformers of the pure-macro features - that understands a lambda enclosed in a nested sequence of single-argument function calls *as a decorated function definition*. This is painful, because the Python AST has no place to store the decorator list for a lambda; Python sees it just as a nested sequence of function calls, terminating in a lambda. This has to be papered over by the transformers. We also introduce a related complication, the decorator registry (see `regutil`), so that we can automatically sort decorator invocations - so that pure-macro features know at which index to inject a particular decorator (so it works properly) when they need to do that. Needing such a registry is already a complication, but the *decorated lambda* machinery feels the pain more acutely. + ## What Belongs in Python? If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), it is because they come with the territory. @@ -86,11 +89,13 @@ In general, I like Python. Also, my hat is off to the devs. It is no mean feat t I think that with macros, Python can be so much more than just a beginner's language. Language-level extensibility is just the logical endpoint of that. I do not share the sentiment of the Python community against metaprogramming, or toward some language-level features. For me, macros (and full-module transforms a.k.a. dialects) are just another tool for creating abstractions, at yet another level. We can already extract procedures, methods, and classes. Why limit that ability - namely, the ability to create abstractions - to what an [eager](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) language can express at run time? -If the point is to keep code understandable, I respect the goal; but that is a matter of education. It is perfectly possible to write unreadable code without macros, and in Python, no less. And it is perfectly possible to write readable code with macros. I am willing to admit the technical objection that *macros do not compose*; but that does not make them useless. +If the point is to keep code understandable, I respect the goal; but that is a matter of education. It is perfectly possible to write unreadable code without macros, and in Python, no less. Just use a complex class hierarchy so that the programmer reading the code must hunt through everything to find each method definition; write big functions without abstracting the steps of the overall algorithm; keep lots of mutable state, and store it in top-level variables; and maybe top that off with an overuse of dependency injection. No one will be able to figure out how the program works, at least not in any reasonable amount of time. + +It is also perfectly possible to write readable code with macros. Just keep in mind that macros are a different kind of abstraction, and use them where that kind of abstraction lends itself to building a clean solution. I am willing to admit the technical objection that *macros do not compose*; but that does not make them useless. Of the particular points above, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hang, when entering an accidentally infinite mutual recursion. I do occasionally make such mistakes when developing complex algorithms - especially when quickly sketching out new ideas. But sometimes, it would be nice to enable TCO selectively. If you ask for it, you know what to expect. This is precisely why `unpythonic.syntax` has `with tco`. I am not very happy with a custom TCO layer on top of a language core that eschews the whole idea, because TCO support in the core (like Scheme and Racket have) would simplify the implementation of certain other language extensions; but then again, [this is exactly what Clojure did](https://clojuredocs.org/clojure.core/trampoline), in similar technical circumstances. -As for a multi-expression `lambda`, on the surface it sounds like a good idea. But really the issue is that in Python, the `lambda` construct itself is broken. It is essentially a duplicate of `def`, but lacking some features. As of Python 3.8, the latest insult to injury is the lack of support for type annotations. A more uniform solution would be to make `def` into an expression. Much of the time, anonymous functions are not a good idea, because names help understanding and debugging - especially when all you have is a traceback. But defining closures inline **is** a great idea - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. +As for a multi-expression `lambda`, on the surface it sounds like a good idea. But really the issue is that in Python, the `lambda` construct itself is broken. It is essentially a duplicate of `def`, but lacking some features. As of Python 3.8, the latest addition of insult to injury is the lack of support for type annotations. A more uniform solution would be to make `def` into an expression. Much of the time, anonymous functions are not a good idea, because names help understanding and debugging - especially when all you have is a traceback. But defining closures inline **is** a great idea - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. The macros in `unpythonic.syntax` inject many lambdas, because that makes them much simpler to implement than if we had to always lift a `def` statement into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if one could define also slightly more complex actions inline (see [Racket's pattern matcher](https://docs.racket-lang.org/reference/match.html) for a comparison). It is unlikely that the action functions will be needed elsewhere, and it is just silly to define a bunch of functions *before* the call to `match`. If this is not a job for either something like `let-where` (to invert the presentation order locally) or a multi-expression lambda (to define the actions inline), I do not know what is. @@ -98,15 +103,15 @@ While on the topic of usability, why are lambdas strictly anonymous? In cases wh On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the parser mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. -It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I don't want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) +It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) -As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant one, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. +As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. -Finally, what to think of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/))? It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its users, and it is not very popular in the grand *scheme* of things (pun not intended). +Finally, there is the issue of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/)). It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its user, and it is not very popular in the programming community at large. -What I can say is, `unpythonic` is not meant for the average Python project, either. If used intelligently, it can make code shorter, yet readable. For a lone developer who needs to achieve as much as possible in the fewest lines reasonably possible, it seems to me that language extension - and in general, as Alexis King put it, [climbing the infinite ladder of abstraction](https://lexi-lambda.github.io/blog/2016/08/11/climbing-the-infinite-ladder-of-abstraction/) - is the way to go. In a large project with a high developer turnover, the optimal solution is different. +What I can say is, `unpythonic` is not meant for the average Python project, either. If used intelligently, it can make code shorter, yet readable. For a lone developer who needs to achieve as much as possible in the fewest lines reasonably possible, it seems to me that language extension - and in general, as Alexis King put it, [climbing the infinite ladder of abstraction](https://lexi-lambda.github.io/blog/2016/08/11/climbing-the-infinite-ladder-of-abstraction/) - is the way to go. In a large project with a high developer turnover, the situation is different. -For general programming in the early 2020s, Python has the ecosystem advantage, so it does not make sense to move to anything else, at least yet. So, let us empower what we have. Even if, in order to achieve that, we have build something that could be considered *unpythonic*. +For general programming in the early 2020s, Python still has the ecosystem advantage, so it does not make sense to move to anything else, at least yet. So, let us empower what we have. Even if we have to build something that could be considered *unpythonic*. ## Killer features of Common Lisp @@ -139,6 +144,7 @@ But for those of us that [don't like parentheses](https://srfi.schemers.org/srfi - PyPy (the JIT-enabled Python interpreter) itself is not the full story; the [RPython](https://rpython.readthedocs.io/en/latest/) toolchain from the PyPy project can *automatically produce a JIT for an interpreter for any new dynamic language implemented in the RPython language* (which is essentially a restricted dialect of Python 2.7). Now **that's** higher-order magic if anything is. - For the use case of numerics specifically, instead of Python, [Julia](https://docs.julialang.org/en/v1/manual/methods/) may be a better fit for writing high-level, yet performant code. It's a spiritual heir of Common Lisp, Fortran, *and Python*. Compilation to efficient machine code, with the help of gradual typing and automatic type inference, is a design goal. + ## Common Lisp, Python, and productivity The various essays by Paul Graham, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for Lisp. So how does the programming world look in that light now, 20 years later? @@ -159,6 +165,7 @@ Haskell aims at code-data equivalence from a third angle (memoized pure function Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world isn't that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead (without restarting the whole app at each change). Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. + ## Python is not a Lisp The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macros.md)) is to make Python lambdas slightly more useful - which was really the starting point for the whole `unpythonic` experiment. @@ -177,6 +184,7 @@ The oft-quoted single-expression limitation of the Python ``lambda`` is ultimate Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. + ## On ``let`` and Python Why no `let*`, as a function? In Python, name lookup always occurs at runtime. Python gives us no compile-time guarantees that no binding refers to a later one - in [Racket](http://racket-lang.org/), this guarantee is the main difference between `let*` and `letrec`. @@ -193,6 +201,7 @@ The [macro versions](macros.md) of the `let` constructs **are** lexically scoped Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). + ## Assignment syntax Why the clunky `e.set("foo", newval)` or `e << ("foo", newval)`, which do not directly mention `e.foo`? This is mainly because in Python, the language itself is not customizable. If we could define a new operator `e.foo newval` to transform to `e.set("foo", newval)`, this would be easily solved. @@ -212,6 +221,7 @@ If we later choose go this route nevertheless, `<<` is a better choice for the s The current solution for the assignment syntax issue is to use macros, to have both clean syntax at the use site and a relatively hackfree implementation. + ## TCO syntax and speed Benefits and costs of ``return jump(...)``: @@ -236,6 +246,7 @@ For other libraries bringing TCO to Python, see: - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. + ## No Monads? (Beside List inside ``forall``.) @@ -244,6 +255,7 @@ Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done else If you want to roll your own monads for whatever reason, there's [this silly hack](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/monads.py) that wasn't packaged into this; or just read Stephan Boyer's quick introduction [[part 1]](https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern) [[part 2]](https://www.stephanboyer.com/post/10/monads-part-2-impure-computations) [[super quick intro]](https://www.stephanboyer.com/post/83/super-quick-intro-to-monads) and figure it out, it's easy. (Until you get to `State` and `Reader`, where [this](http://brandon.si/code/the-state-monad-a-tutorial-for-the-confused/) and maybe [this](https://gaiustech.wordpress.com/2010/09/06/on-monads/) can be helpful.) + ## No Types? The `unpythonic` project will likely remain untyped indefinitely, since I don't want to enter that particular marshland with things like `curry` and `with continuations`. It may be possible to gradually type some carefully selected parts - but that's currently not on [the roadmap](https://github.com/Technologicat/unpythonic/milestones). I'm not against it, if someone wants to contribute. @@ -273,6 +285,7 @@ More on type systems: - In physics, units as used for dimension analysis are essentially a form of static typing. - This has been discussed on LtU, see e.g. [[1]](http://lambda-the-ultimate.org/node/33) [[2]](http://lambda-the-ultimate.org/classic/message11877.html). + ## Detailed Notes on Macros - ``continuations`` and ``tco`` are mutually exclusive, since ``continuations`` already implies TCO. @@ -322,6 +335,7 @@ More on type systems: - When in doubt, you can use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. In ``mcpyrate``, this is almost equivalent to having the macros invoked in a single ``with`` statement, in the same order. - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. (And of course, while testing, try to keep the input as simple as possible.) + ## Miscellaneous notes - [Nick Coghlan (2011): Traps for the unwary in Python's import system](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html). From 793748ca3edb1e0dc042e81ae1683d1ad1bdf02a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 8 Jun 2021 23:26:22 +0300 Subject: [PATCH 438/832] John Shutt (Kernel Lisp author) died in 2021; link the news on LtU --- doc/readings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/readings.md b/doc/readings.md index 80200ac0..e4acee43 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -149,7 +149,7 @@ The common denominator is programming. Some relate to language design, some to c - A special `uninitialized` value (which the paper calls ☠) is needed, because Scope - in the sense of controlling lexical name resolution - is a static (purely lexical) concept, but whether a particular name (once lexically resolved) has been initialized (or, say, whether it has been deleted) is a dynamic (run-time) feature. (I would say "property", if that word didn't have an entirely different technical meaning in Python.) - Our `continuations` macro essentially does what the authors call *a standard [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style) transformation*, plus some technical details due to various bits of impedance mismatch. -- [John Shutt's blog](https://fexpr.blogspot.com/) contains many interesting posts on programming language design. He's the author of the [Kernel](https://web.cs.wpi.edu/~jshutt/kernel.html) Lisp dialect. Some pickings from the blog: +- [John Shutt's blog](https://fexpr.blogspot.com/) contains many interesting posts on programming language design. He [was](http://lambda-the-ultimate.org/node/5623) the author of the [Kernel](https://web.cs.wpi.edu/~jshutt/kernel.html) Lisp dialect. Some pickings from his blog: - [Fexpr (2011)](https://fexpr.blogspot.com/2011/04/fexpr.html). - The common wisdom that macros were a better choice is misleading. - [Bypassing no-go theorems (2013)](https://fexpr.blogspot.com/2013/07/bypassing-no-go-theorems.html). From a60a3863847859c41d4fd13bfc676dea582751a7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 8 Jun 2021 23:26:55 +0300 Subject: [PATCH 439/832] add more readings --- doc/readings.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/doc/readings.md b/doc/readings.md index e4acee43..eb838c22 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -168,6 +168,64 @@ The common denominator is programming. Some relate to language design, some to c - [Types vs. traits for dispatch](https://discourse.julialang.org/t/types-vs-traits-for-dispatch/46296) (discussion) - We have a demonstration in [unpythonic.tests.test_dispatch](../unpythonic/tests/test_dispatch.py). +- [Pascal Costanza's Highly Opinionated Guide to Lisp (2013)](http://www.p-cos.net/lisp/guide.html) + +- R. Kent Dybvig, Simon Peyton Jones, Amr Sabry (2007). A Monadic Framework for Delimited Continuations. Journal of functional programming, 17(6), 687-730. Preprint [here](https://legacy.cs.indiana.edu/~dyb/pubs/monadicDC.pdf). + - Particularly approachable explanation of delimited continuations. + - Could try building that for `unpythonic` in a future version. While our outermost `call_cc` already somewhat acts like a prompt, we're currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and terminate the capture there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. + +- [Wat: Concurrency and Metaprogramming for JS](https://github.com/manuel/wat-js) + - [pywat: Interpreter of the Wat language written in Python](https://github.com/piokuc/pywat) + - [Example of Wat in Manuel Simoni's blog (2013)](http://axisofeval.blogspot.com/2013/05/green-threads-in-browser-in-20-lines-of.html) + - This suggests building proper delimited continuations shouldn't be that hard in Python. + +- [Richard P. Gabriel, Kent M. Pitman (2001): Technical Issues of Separation in Function Cells and Value Cells](https://dreamsongs.com/Separation.html) + - A discussion of [Lisp-1 vs. Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2). + +- [`hoon`: The C of Functional Programming](https://github.com/cgyarvin/urbit/blob/master/doc/book/0-intro.markdown#hoon) + - *The above link points to an old version from 2013; see below for a link to the latest version. I have given the old link here first, because it explains the philosophy differently from the latest documentation.* + - Some days I wonder if this `unpythonic` endeavor even makes any sense, and then I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. From the doc linked above: + + *So we could describe Hoon as a pure, strict, higher-order typed functional language. But don't do this in front of a Haskell purist, unless you put quotes around "typed," "functional," and possibly even "language." We could also say "object-oriented," with the same scare quotes for the cult of Eiffel.* + + While I am not sure if I will ever *use* `hoon`, it is hard not to like a language that puts quotes around "language". Few languages go that far in shaking up preconceptions. Critically examining what we believe, and why, often leads to useful insights. + + The claim that `hoon` is not a language, but a "language", fully makes sense after reading some of the documentation. `hoon` is essentially an *ab initio* language with an axiomatic approach to defining its operational semantics, similarly to how *Arc* approaches defining Lisp. Furthermore, `hoon` is the *functional equivalent of C* to the underlying virtual assembly language, `nock`. From a certain viewpoint, the "language" essentially consists of *glorified Nock macros*. Glorified assembly macros are pretty much all a *low-level* [HLL](https://en.wikipedia.org/wiki/High-level_programming_language) essentially is, so the claim seems about right. + + Nock is a peculiar assembly language. According to the comments in [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), it is a *Turing-complete non-lambda automaton*. The instruction set is permanently frozen, as if it was a physical CPU chip. Opcodes are just natural numbers, 0 through 11, and it is very minimalistic. For example, there is not even a decrement opcode. This is because from an axiomatic viewpoint, decrement can be defined recursively via increment. At which point, every systems programmer objects, rightfully, that no one sane actually does so. Indeed, the `hoon` standard library uses C FFI to take advantage of the physical processor's instruction set to perform arithmetic operations. Each piece of C code used for such acceleration purposes is termed a *jet*. + + Since - by the fact that the programmer called a particular standard library function - the system knows we want to compute a decrement (or a multiplication, a power, maybe some floating point operation, etc.), it can *accelerate* that particular operation by using the available hardware. + + The important point is, you *could* write out a `nock` macro that does the same thing, only it would be unbearably slow. In the axiomatic perspective - which is about proving programs correct - speed does not matter. At the same time, FFI gives speed for the real world. + + To summarize; as someone already put it, `hoon` offers a glimpse into an alternate universe of systems programming, where the functional camp won. It may also be a useful tool, or a source for further unconventional ideas - but to know for sure, I will have to read more about it. + + *NOTE: Using natural numbers for the opcodes at first glance sounds like a [Gödel numbering](https://en.wikipedia.org/wiki/G%C3%B6del_numbering) for the program space; but actually, the input to the VM contains some linked-list structure, which is not represented that way. Also, **any** programming language imposes its own Gödel numbering on the program space. Just take, for example, the UTF-8 representation of the source code text (which, in Python terms, is a `bytes` object), and interpret those bytes as one single bignum.* + + *Obviously, any interesting programs correspond to very large numbers, and are few and far between, so decoding random numbers via a Gödel numbering is not a practical way to generate interesting programs. [Genetic programming](https://en.wikipedia.org/wiki/Genetic_programming) works much better, because unlike Gödel numbering, it was actually designed specifically to do that. GP takes advantage of the semantic structure present in the source code (or AST) representation.* + + *The purpose of the original Gödel numbering was to prove Gödel's incompleteness theorem. In the case of `nock`, my impression is that the opcodes are natural numbers just for flavoring purposes. If you are building an ab initio software stack, what better way to announce that than to use natural numbers as your virtual machine's opcodes?* + + - From the language definition, [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon): + ``` + ++ doos :: sleep until + |= hap=path ^- (unit ,@da) + (doze:(wink:(vent bud (dink (dint hap))) now 0 (beck ~)) now [hap ~]) + :: + ``` + The Lisp family (particularly the Common Lisp branch) has a reputation for silly terminology, but `hoon` takes that a step further. + + However, I think I will adopt the verb *bunt*, meaning *to take the default value of*. That is such a common operation in programming that I find it hard to believe there is no standard abbreviation. + + Judging by the docs, `hoon` is definitely ha-ha-only-serious, but I am not sure of whether it is serious-serious. It does advertise itself as the functional-programming equivalent of C. See the comments to the entry on Manuel Simoni's blog - some people do think `hoon` is actually useful. + + So maybe there is a place for `unpythonic`, too. + + - The development of `urbit` has [moved to a new repository](https://github.com/urbit/urbit). + - The [latest `hoon` docs](https://urbit.org/docs/hoon/). + - Interestingly, `hoon` has uniform support for *wide* and *tall* modes; it does not use parentheses, but uses a single space (in characteristic `hoon` fashion, termed an *ace*) versus multiple spaces (respectively, a *gap*). "Multiple spaces" allows also newlines, like in LaTeX. + - `hoon` does not have syntactic macros. The reason given in the docs is the same as sometimes heard in the Python community - having a limited number of standard control structures, you always know what you're looking at. + # Python-related FP resources From 041ef3a6527e4e052fe41f49e58fe196c2a4d8c1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 00:16:52 +0300 Subject: [PATCH 440/832] improve lispython doc --- doc/dialects/lispython.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index a4ccf3ae..153cefd5 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -77,8 +77,8 @@ In terms of ``unpythonic.syntax``, we implicitly enable ``tco``, ``autoreturn``, - TCO in both ``def`` and ``lambda``, fully automatic - Omit ``return`` in any tail position, like in Lisps - Multiple-expression lambdas, ``lambda x: [expr0, ...]`` - - Named lambdas (whenever the machinery can figure out a name) - The underscore: ``f[_*3] --> lambda x: x*3`` (name ``f`` is **reserved**) + - Automatically named lambdas whenever the machinery can figure out a name; when not, source location is auto-injected into the name. We also import some macros and functions to serve as dialect builtins: @@ -187,7 +187,7 @@ def foo(n): return accumulate ``` -The problem is that assignment to a lexical variable (including formal parameters) is a statement in Python. Python 3.8's walrus operator does not solve this, because `n := n + i` by itself is a syntax error. +The problem is that assignment to a lexical variable (including formal parameters) is a statement in Python. Python 3.8's walrus operator does not solve this, because `n := n + i` by itself is a syntax error, and even if parenthesized, `(n := n + i)` insists on creating a new local variable `n`. If we abbreviate ``accumulate`` as a lambda, it needs a ``let`` environment to write in, to use `unpythonic`'s expression-assignment (`name << value`). @@ -210,6 +210,8 @@ with envify: ``envify`` is not part of the Lispython dialect definition, because this particular, perhaps rarely used, feature is not really worth a global performance hit whenever a function is entered. +Note that ``envify`` is **not** compatible with Lispython, because it would need to appear in a lexically outer position compared to macros already invoked by the dialect template. If you need an envified Lispython, copy `unpythonic/dialects/lispython.py` and modify the template therein. [The xmas tree combo](../macros.md#the-xmas-tree-combo) says `envify` should come lexically after `multilambda`, but before `namedlambda`. + ## CAUTION From 38e43f6f86823c36ee179953960885bfce82268f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 00:18:19 +0300 Subject: [PATCH 441/832] rename f[] macro to fn[] Less often used as a function name (code examples, local temporaries, etc.), and less ambiguous that this is a syntactic construct that means "function". --- CHANGELOG.md | 18 ++++---- doc/dialects/lispython.md | 2 +- doc/macros.md | 32 +++++++------- unpythonic/dialects/lispython.py | 2 +- unpythonic/dialects/tests/test_lispython.py | 8 +++- unpythonic/syntax/lambdatools.py | 46 ++++++++++----------- unpythonic/syntax/tests/test_lambdatools.py | 8 ++-- unpythonic/syntax/tests/test_tco.py | 8 ++-- 8 files changed, 67 insertions(+), 57 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index be326178..3b2d2f32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -156,16 +156,18 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `pipe` family - `compose` family - All multiple-return-values in code using the `with continuations` macro. (The continuations system essentially composes continuation functions.) - - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.f`, because they used to be provided by `macropy`, and `mcpyrate` does not provide them. + - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.fn` (note name change!), because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **API differences.** - - The macros `lazy` and `f` can be imported from the syntax interface module, `unpythonic.syntax`, and the class `Lazy` is available at the top level of `unpythonic`. - - Unlike `macropy`'s `Lazy`, our `Lazy` does not define `__call__`; instead, it defines the method `force`, which has the same effect (it computes if necessary, and then returns the value of the promise). - - When you import the macro `quicklambda`, you **must** import also the macro `f`. - - The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else it is available to be used as a regular variable. + - The quick lambda is now named `fn[]` instead of `f[]` (as in MacroPy). This was changed because `f` is often used as a function name in code examples, local temporaries, and similar. Also, `fn[]` is a less ambiguous abbreviation for a syntactic construct that means *function*, while remaining shorter than the equivalent `lambda`. Compare `fn[_ * 2]` and `lambda x: x * 2`, or `fn[_ * _]` and `lambda x, y: x * y`. + - Note that in `mcpyrate`, macros can be as-imported, so this change affects just the *default* name of `fn[]`. But that is exactly what is important: have a sensible default name, to remove the need to as-import so often. + - The macros `lazy` and `fn` can be imported from the syntax interface module, `unpythonic.syntax`, and the class `Lazy` is available at the top level of `unpythonic`. + - Unlike `macropy`'s `Lazy`, our `Lazy` does not define `__call__`; instead, it defines the method `force`, which has the same effect (it computes if necessary, and then returns the value of the promise). You can also use the function `unpythonic.force`, which has the extra advantage that it passes through a non-promise input unchanged (so you don't need to care whether `x` is a promise before calling `force(x)`; this is sometimes useful). + - When you import the macro `quicklambda`, you **must** import also the macro `fn`. + - The underscore `_` is no longer a macro on its own. The `fn` macro treats the underscore magically, as before, but anywhere else it is available to be used as a regular variable. - **Behavior differences.** - - `f[]` now respects nesting: an invocation of `f[]` will not descend into another nested `f[]`. - - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `f[]` invocations lexically inside the block to expand before any other macros in that block do. - - Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro respects the as-import, by internally querying the expander to determine the name(s) the macro `f` is currently bound to. + - `fn[]` now respects nesting: an invocation of `fn[]` will not descend into another nested `fn[]`. + - The `with quicklambda` macro is still provided, and used just as before. Now it causes any `fn[]` invocations lexically inside the block to expand before any other macros in that block do. + - Since in `mcpyrate`, macros can be as-imported, you can rename `fn` at import time to have any name you want. The `quicklambda` block macro respects the as-import, by internally querying the expander to determine the name(s) the macro `fn` is currently bound to. - For the benefit of code using the `with lazify` macro, laziness is now better respected by the `compose` family, `andf` and `orf`. The utilities themselves are marked lazy, and arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain. - Rename the `curry` macro to `autocurry`, to prevent name shadowing of the `curry` function. The new name is also more descriptive. - Move the functions `force1` and `force` from `unpythonic.syntax` to `unpythonic`. Make the `Lazy` class (promise implementation) public. (They actually come from `unpythonic.lazyutil`.) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 153cefd5..279ac549 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -77,8 +77,8 @@ In terms of ``unpythonic.syntax``, we implicitly enable ``tco``, ``autoreturn``, - TCO in both ``def`` and ``lambda``, fully automatic - Omit ``return`` in any tail position, like in Lisps - Multiple-expression lambdas, ``lambda x: [expr0, ...]`` - - The underscore: ``f[_*3] --> lambda x: x*3`` (name ``f`` is **reserved**) - Automatically named lambdas whenever the machinery can figure out a name; when not, source location is auto-injected into the name. + - The underscore: e.g. `fn[_ * 3]` becomes `lambda x: x * 3`, and `fn[_ * _]` becomes `lambda x, y: x * y`. We also import some macros and functions to serve as dialect builtins: diff --git a/doc/macros.md b/doc/macros.md index 47f1490f..9efa4d56 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -39,7 +39,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose [**Tools for lambdas**](#tools-for-lambdas) - [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. - [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. -- [``f``: underscore notation (quick lambdas) for Python](#f-underscore-notation-quick-lambdas-for-python) +- [``fn``: underscore notation (quick lambdas) for Python](#f-underscore-notation-quick-lambdas-for-python) - [``quicklambda``: expand quick lambdas first](#quicklambda-expand-quick-lambdas-first) - [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) @@ -704,37 +704,41 @@ The naming is performed using the function ``unpythonic.misc.namelambda``, which Support for other forms of assignment may or may not be added in a future version. -### ``f``: underscore notation (quick lambdas) for Python. +### ``fn``: underscore notation (quick lambdas) for Python. -**Changed in v0.15.0.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves. The underscore `_` is no longer a macro on its own. The `f` macro treats the underscore magically, as before, but anywhere else the underscore is available to be used as a regular variable. If you use `f[]`, change your import of this macro to `from unpythonic.syntax import macros, f`.* +**Changed in v0.15.0.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves.* -The syntax ``f[...]`` creates a lambda, where each underscore in the ``...`` part introduces a new parameter. The macro does not descend into any nested ``f[]``. +*The name is now `fn[]`. This was changed because `f` is often used as a function name in code examples, local temporaries, and similar. Also, `fn[]` is a less ambiguous abbreviation for a syntactic construct that means *function*, while remaining shorter than the equivalent `lambda`.* + +*The underscore `_` is no longer a macro on its own. The `fn` macro treats the underscore magically, as `f` did before, but anywhere else the underscore is available to be used as a regular variable. If you use `fn[]`, change your import of this macro to `from unpythonic.syntax import macros, fn`.* + +The syntax ``fn[...]`` creates a lambda, where each underscore in the ``...`` part introduces a new parameter. The macro does not descend into any nested ``fn[]``. Example: ```python -func = f[_ * _] # --> func = lambda x, y: x * y +func = fn[_ * _] # --> func = lambda x, y: x * y ``` -Since in `mcpyrate`, macros can be as-imported, you can rename `f` at import time to have any name you want. The `quicklambda` block macro (see below) respects the as-import. Now you **must** import also the macro `f` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `f` is currently bound to. +Since in `mcpyrate`, macros can be as-imported, you can rename `fn` at import time to have any name you want. The `quicklambda` block macro (see below) respects the as-import. Now you **must** import also the macro `fn` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `fn` is currently bound to. ### ``quicklambda``: expand quick lambdas first To be able to transform correctly, the block macros in ``unpythonic.syntax`` that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all ``lambda`` definitions written with Python's standard ``lambda``. -However, the ``f`` macro uses the syntax ``f[...]``, which (to the analyzer) does not look like a lambda definition. This macro changes the expansion order, forcing any ``f[...]`` lexically inside the block to expand before any other macros do. +However, the ``fn`` macro uses the syntax ``fn[...]``, which (to the analyzer) does not look like a lambda definition. This macro changes the expansion order, forcing any ``fn[...]`` lexically inside the block to expand before any other macros do. -Any expression of the form ``f[...]``, where ``f`` is any name bound in the current macro expander to the macro `unpythonic.syntax.f`, is understood as a quick lambda. (In plain English, this respects as-imports of the macro ``f``.) +Any expression of the form ``fn[...]``, where ``fn`` is any name bound in the current macro expander to the macro `unpythonic.syntax.fn`, is understood as a quick lambda. (In plain English, this respects as-imports of the macro ``fn``.) Example - a quick multilambda: ```python -from unpythonic.syntax import macros, multilambda, quicklambda, f, local +from unpythonic.syntax import macros, multilambda, quicklambda, fn, local with quicklambda, multilambda: - func = f[[local[x << _], - local[y << _], - x + y]] + func = fn[[local[x << _], + local[y << _], + x + y]] assert func(1, 2) == 3 ``` @@ -744,10 +748,10 @@ This is of course rather silly, as an unnamed formal parameter can only be menti with quicklambda, tco: def g(x): return 2*x - func1 = f[g(3*_)] # tail call + func1 = fn[g(3*_)] # tail call assert func1(10) == 60 - func2 = f[3*g(_)] # no tail call + func2 = fn[3*g(_)] # no tail call assert func2(10) == 60 ``` diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index 63d94e17..73bdedd0 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -28,7 +28,7 @@ def transform_ast(self, tree): # tree is an ast.Module with q as template: __lang__ = "Lispython" # noqa: F841, just provide it to user code. from unpythonic.syntax import (macros, tco, autoreturn, # noqa: F401, F811 - multilambda, quicklambda, namedlambda, f, + multilambda, quicklambda, namedlambda, fn, where, let, letseq, letrec, dlet, dletseq, dletrec, diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index a479a8ec..9c081714 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -73,11 +73,15 @@ def f(k, acc): test[square(3) == 9] test[square.__name__ == "square"] - # the underscore (NOTE: due to this, "f" is a reserved name in lispython) - cube = f[_**3] # noqa: F821 + # the underscore (NOTE: due to this, "fn" is a reserved name in Lispython) + cube = fn[_**3] # noqa: F821 test[cube(3) == 27] test[cube.__name__ == "cube"] + my_mul = fn[_ * _] # noqa: F821 + test[my_mul(2, 3) == 6] + test[my_mul.__name__ == "my_mul"] + # lambdas can have multiple expressions and local variables # # If you need to return a literal list from a lambda, use an extra set of diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index f21a9cdc..764367d4 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -3,7 +3,7 @@ __all__ = ["multilambda", "namedlambda", - "f", + "fn", "quicklambda", "envify"] @@ -124,64 +124,64 @@ def namedlambda(tree, *, syntax, expander, **kw): with dyn.let(_macro_expander=expander): return _namedlambda(block_body=tree) -def f(tree, *, syntax, expander, **kw): +def fn(tree, *, syntax, expander, **kw): """[syntax, expr] Underscore notation (quick lambdas) for Python. Usage:: - f[body] + fn[body] - The ``f[]`` macro creates a lambda. Each underscore in ``body`` + The ``fn[]`` macro creates a lambda. Each underscore in ``body`` introduces a new parameter. Example:: - func = f[_ * _] + func = fn[_ * _] expands to:: func = lambda a0, a1: a0 * a1 - The underscore is interpreted magically by ``f[]``; but ``_`` itself - is not a macro, and has no special meaning outside ``f[]``. The underscore - does **not** need to be imported for ``f[]`` to recognize it. + The underscore is interpreted magically by ``fn[]``; but ``_`` itself + is not a macro, and has no special meaning outside ``fn[]``. The underscore + does **not** need to be imported for ``fn[]`` to recognize it. - The macro does not descend into any nested ``f[]``. + The macro does not descend into any nested ``fn[]``. """ if syntax != "expr": raise SyntaxError("f is an expr macro only") # pragma: no cover # What's my name in the current expander? (There may be several names.) # https://github.com/Technologicat/mcpyrate/blob/master/doc/quasiquotes.md#hygienic-macro-recursion - bindings = extract_bindings(expander.bindings, f) + bindings = extract_bindings(expander.bindings, fn) mynames = list(bindings.keys()) - return _f(tree, mynames) + return _fn(tree, mynames) def quicklambda(tree, *, syntax, expander, **kw): - """[syntax, block] Make ``f`` quick lambdas expand first. + """[syntax, block] Make ``fn`` quick lambdas expand first. To be able to transform correctly, the block macros in ``unpythonic.syntax`` that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all ``lambda`` definitions written with Python's standard ``lambda``. - However, the ``f`` macro uses the syntax ``f[...]``, which (to the analyzer) + However, the ``fn`` macro uses the syntax ``f[...]``, which (to the analyzer) does not look like a lambda definition. This macro changes the expansion - order, forcing any ``f[...]`` lexically inside the block to expand before + order, forcing any ``fn[...]`` lexically inside the block to expand before any other macros do. - Any expression of the form ``f[...]``, where ``f`` is any name bound in the - current macro expander to the macro `unpythonic.syntax.f`, is understood as - a quick lambda. (In plain English, this respects as-imports of the macro ``f``.) + Any expression of the form ``fn[...]``, where ``fn`` is any name bound in the + current macro expander to the macro `unpythonic.syntax.fn`, is understood as + a quick lambda. (In plain English, this respects as-imports of the macro ``fn``.) Example - a quick multilambda:: - from unpythonic.syntax import macros, multilambda, quicklambda, f, local + from unpythonic.syntax import macros, multilambda, quicklambda, fn, local with quicklambda, multilambda: - func = f[[local[x << _], - local[y << _], - x + y]] + func = fn[[local[x << _], + local[y << _], + x + y]] assert func(1, 2) == 3 (This is of course rather silly, as an unnamed argument can only be mentioned @@ -200,7 +200,7 @@ def quicklambda(tree, *, syntax, expander, **kw): # the original expander. Thus it leaves all other macros alone. This is the # official `mcpyrate` way to immediately expand only some particular macros # inside the current macro invocation. - bindings = extract_bindings(expander.bindings, f) + bindings = extract_bindings(expander.bindings, fn) return MacroExpander(bindings, expander.filename).visit(tree) def envify(tree, *, syntax, expander, **kw): @@ -411,7 +411,7 @@ def transform(self, tree): # # Used under the MIT license. # Copyright (c) 2013-2018, Li Haoyi, Justin Holmgren, Alberto Berti and all the other contributors. -def _f(tree, mynames=()): +def _fn(tree, mynames=()): class UnderscoreTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index 7349fd36..8a4ad43b 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -4,7 +4,7 @@ from ...syntax import macros, test, test_raises, warn # noqa: F401 from ...test.fixtures import session, testset -from ...syntax import (macros, multilambda, namedlambda, quicklambda, f, # noqa: F401, F811 +from ...syntax import (macros, multilambda, namedlambda, quicklambda, fn, # noqa: F401, F811 envify, local, let, autocurry, autoreturn) from functools import wraps @@ -173,9 +173,9 @@ def decorated(*args, **kwargs): # Outside-in macros. with quicklambda: with multilambda: - func = f[[local[x << _], # noqa: F821, F823, `quicklambda` implicitly defines `f[]` to mean `lambda`. - local[y << _], # noqa: F821 - x + y]] # noqa: F821 + func = fn[[local[x << _], # noqa: F821, F823, `quicklambda` implicitly defines `fn[]` to mean `lambda`. + local[y << _], # noqa: F821 + x + y]] # noqa: F821 test[func(1, 2) == 3] with testset("envify (formal parameters as an unpythonic env)"): diff --git a/unpythonic/syntax/tests/test_tco.py b/unpythonic/syntax/tests/test_tco.py index 49f95957..87691a5e 100644 --- a/unpythonic/syntax/tests/test_tco.py +++ b/unpythonic/syntax/tests/test_tco.py @@ -5,7 +5,7 @@ from ...test.fixtures import session, testset, returns_normally from ...syntax import (macros, tco, autoreturn, autocurry, do, let, letseq, dletrec, # noqa: F401, F811 - quicklambda, f, continuations, call_cc) + quicklambda, fn, continuations, call_cc) from ...ec import call_ec from ...fploop import looped_over @@ -143,7 +143,7 @@ def result(loop, x, acc): test[looped_over(range(10), acc=0)(lambda loop, x, acc: loop(acc + x)) == 45] with testset("integration with quicklambda"): - # f[] must expand first so that tco sees it as a lambda. + # Use `quicklambda` to force `fn[]` to expand first, so that tco sees it as a lambda. # `quicklambda` is an outside-in macro, so placed on the outside, it expands first. with quicklambda: with tco: @@ -152,10 +152,10 @@ def g(x): # TODO: Improve test to actually detect the tail call. # TODO: Now we just test this runs without errors. - func1 = f[g(3 * _)] # tail call # noqa: F821, _ is magic. + func1 = fn[g(3 * _)] # tail call # noqa: F821, _ is magic. test[func1(10) == 60] - func2 = f[3 * g(_)] # no tail call # noqa: F821, _ is magic. + func2 = fn[3 * g(_)] # no tail call # noqa: F821, _ is magic. test[func2(10) == 60] with testset("integration with continuations"): From cb6d4dee7890cc463a416c6f04b05676cdb2be35 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 00:19:23 +0300 Subject: [PATCH 442/832] improve xmas tree combo doc Particularly, now we give the important information first. Furthermore, subsections have been added to group the finer points. --- doc/macros.md | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 9efa4d56..28ff9bd6 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2150,7 +2150,28 @@ Is this just a set of macros, a language extension, or a compiler for a new lang The macros in ``unpythonic.syntax`` are designed to work together, but some care needs to be taken regarding the order in which they expand. This complexity unfortunately comes with any pick-and-mix-your-own-language kit, because some features inevitably interact. For example, it is possible to lazify [continuation-enabled](https://en.wikipedia.org/wiki/Continuation-passing_style) code, but running the transformations the other way around produces nonsense. -For simplicity, **the block macros make no attempt to prevent invalid combos** (unless there is a specific technical reason to do that for some particular combination). Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. +The correct **xmas tree invocation** is: + +```python +with prefix, autoreturn, quicklambda, multilambda, envify, lazify, namedlambda, autoref, autocurry, tco: + ... +``` + +Here `tco` can be replaced with `continuations`, if needed. + +We have taken into account that: + + - Outside-in: `prefix`, `autoreturn`, `quicklambda`, `multilambda` + - Two-pass: `envify`, `lazify`, `namedlambda`, `autoref`, `autocurry`, `tco`/`continuations` + +[The dialect examples](dialects.md) use this ordering. + +For simplicity, **the block macros make no attempt to prevent invalid combos**, unless there is a specific technical reason to do that for some particular combination. Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. + +As an example of a specific technical reason, the `tco` macro skips already expanded `with continuations` blocks lexically contained within the `with tco`. This allows the [Lispython dialect](dialects/lispython.md) to support `continuations`. + + +#### AST edit order vs. macro invocation order The **AST edits** performed by the block macros are designed to run in the following order (leftmost first): @@ -2161,7 +2182,7 @@ prefix > autoreturn, quicklambda > multilambda > continuations or tco > ... The ``let_syntax`` (and ``abbrev``) block may be placed anywhere in the chain; just keep in mind what it does. -The ``dbg`` block can be run at any position after ``prefix`` and before ``tco`` (or ``continuations``). (It must be able to see function calls in Python's standard format, for detecting calls to the print function.) +The ``dbg`` block can be run at any position after ``prefix`` and before ``tco`` (or ``continuations``). It must be able to see function calls in Python's standard format, for detecting calls to the print function. The correct ordering for **block macro invocations** - which is the actual user-facing part - is somewhat complicated by the fact that some of the above are two-pass macros. Consider this artificial example, where `mac` is a two-pass macro: @@ -2177,21 +2198,12 @@ The invocation `with mac` is *lexically on the outside*, thus the macro expander 2. Explicit recursion by `with mac`. This expands the `with cheese`. 3. Second pass (inside out) of `with mac`. -So, for example, even though `lazify` must *perform its AST editing* after `autocurry`, it happens to be a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, even though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. +So, for example, even though `lazify` must *perform its AST edits* after `autocurry`, it happens to be a two-pass macro. The first pass (outside in) only performs some preliminary analysis; the actual lazification happens in the second pass (inside out). So the correct invocation comboing these two is `with lazify, autocurry`. Similarly, `with lazify, continuations` is correct, even though the CPS transformation must occur first; these are both two-pass macros that perform their edits in the inside-out pass. -Considering that: - - - Outside-in: `prefix`, `autoreturn`, `quicklambda`, `multilambda` - - Two-pass: `envify`, `lazify`, `namedlambda`, `autoref`, `autocurry`, `tco`/`continuations` +Further details on individual block macros can be found in our [notes on macros](design-notes.md#detailed-notes-on-macros). -the correct **xmas tree invocation** is: - -```python -with prefix, autoreturn, quicklambda, multilambda, envify, lazify, namedlambda, autoref, autocurry, tco: - ... -``` -[The dialect examples](dialects.md) use this ordering. See our [notes on macros](design-notes.md#detailed-notes-on-macros) for some more details. +#### Single-line vs. multiline invocation format Example combo in the single-line format: @@ -2200,7 +2212,7 @@ with autoreturn, lazify, tco: ... ``` -In the multiline format: +The same combo in the multiline format: ```python with autoreturn: @@ -2209,7 +2221,7 @@ with autoreturn: ... ``` -**NOTE**: In MacroPy, there sometimes were [differences](https://github.com/azazel75/macropy/issues/21) between the behavior of the single-line and multi-line invocation format, but in `mcpyrate`, they should behave the same. +In MacroPy (which was used up to v0.14.3), there sometimes were [differences](https://github.com/azazel75/macropy/issues/21) between the behavior of the single-line and multi-line invocation format, but in `mcpyrate` (which is used by v0.15.0 and later), they should behave the same. With `mcpyrate`, there is still [a minor difference](https://github.com/Technologicat/mcpyrate/issues/3) if there are at least three nested macro invocations, and a macro is scanning the tree for another macro invocation; then the tree looks different depending on whether the single-line or the multi-line format was used. The differences in that are as one would expect knowing [how `with` statements look like](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#With) in the Python AST. The reason the difference manifests only for three or more macro invocations is that `mcpyrate` pops the macro that is being expanded before it hands over the tree to the macro code; hence if there are only two, the inner tree will have only one "context manager" in its `with`. From ef8cd79c3e0934cfbe733c573236d4a7b797c9d1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 00:20:06 +0300 Subject: [PATCH 443/832] spelling: Lispython --- unpythonic/dialects/tests/test_lispython.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index 9c081714..e41913ec 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -8,7 +8,7 @@ from ...syntax import macros, continuations, call_cc # noqa: F401, F811 -# `unpythonic` is effectively `lispython`'s stdlib; not everything gets imported by default. +# `unpythonic` is effectively Lispython's stdlib; not everything gets imported by default. from ...fold import foldl # Of course, all of Python's stdlib is available too. From b245d9bf8687071ea7f0c809a43c3cabdbb5c5df Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 01:57:17 +0300 Subject: [PATCH 444/832] wording --- unpythonic/dialects/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/__init__.py b/unpythonic/dialects/__init__.py index 67d6d7df..9de09993 100644 --- a/unpythonic/dialects/__init__.py +++ b/unpythonic/dialects/__init__.py @@ -8,7 +8,7 @@ We provide these dialects mainly to demonstrate how to use that subsystem to customize Python beyond what a local macro expander can do. -For examples of how to use the dialects, see the unit tests. +For examples of how to use these particular dialects, see the unit tests. """ # re-exports From e41de428956fae074cedebbec17042d66b1b324e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 01:57:43 +0300 Subject: [PATCH 445/832] add Lispy, a Lispython-lite that only changes the semantics. Lispy is a more pythonic variant of Lispython. It does not introduce any implicit imports, beyond what the dialect template injects to make the semantic changes happen at macro expansion time. This makes IDEs happy, because any name that appears in user code must be explicitly defined there, as usual in Python. --- doc/dialects/lispython.md | 74 +++++++++++++++++++++++--------- unpythonic/dialects/__init__.py | 2 +- unpythonic/dialects/lispython.py | 29 +++++++++++++ 3 files changed, 83 insertions(+), 22 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 279ac549..f93bd43c 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -18,6 +18,8 @@ - [Lispython: The love child of Python and Scheme](#lispython-the-love-child-of-python-and-scheme) - [Features](#features) + - [The `Lispy` variant](#the-lispy-variant) + - [The `Lispython` variant](#the-lispython-variant) - [What Lispython is](#what-lispython-is) - [Comboability](#comboability) - [Lispython and continuations (call/cc)](#lispython-and-continuations-callcc) @@ -55,9 +57,6 @@ square = lambda x: x**2 assert square(3) == 9 assert square.__name__ == "square" -# - brackets denote a multiple-expression lambda body -# (if you want to have one expression that is a literal list, -# double the brackets: `lambda x: [[5 * x]]`) # - local[name << value] makes an expression-local variable g = lambda x: [local[y << 2 * x], y + 1] @@ -72,30 +71,63 @@ assert ll(1, 2, 3) == llist((1, 2, 3)) ## Features -In terms of ``unpythonic.syntax``, we implicitly enable ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, and ``quicklambda`` for the whole module: +In terms of ``unpythonic.syntax``, we implicitly enable ``autoreturn``, ``tco``, ``multilambda``, ``namedlambda``, and ``quicklambda`` for the whole module: + + - In tail position, the ``return`` keyword can be omitted, like in Lisps. + - In a `def`, the last statement at the top level of the `def` is in tail position. + - If the tail position contains an expression, a ``return`` will be automatically injected, with that expression as the return value. + - It is still legal to use `return` whenever you would in Python; this just makes the `return` keyword non-mandatory in places where a Lisp would not require it. + - To be technically correct, Schemers and Racketeers should read this as, *"in places where a Lisp would not require explicitly invoking an escape continuation"*. + - Automatic tail-call optimization (TCO) for both ``def`` and ``lambda``. + - In a `def`, the last statement at the top level of the `def` is in tail position. + - Tail positions *inside an expression* that itself appears in tail position are: + - Both the `body` and `orelse` branches of an if-expression. (Exactly one of them runs, hence both are in tail position.) + - The lexically last item of an `and`/`or` chain. + - Note the analysis is performed at compile time, whence it does **not** care about the short-circuit behavior that occurs at run time. + - The last item of a `do[]`. + - The last item of an implicit `do[]` in a `let[]` where the body uses the extra bracket syntax. (All `let` constructs provided by `unpythonic.syntax` are supported.) + - For the gritty details, see the source code of `unpythonic.syntax.tailtools._transform_retexpr`. + - Multiple-expression lambdas, using bracket syntax, for example ``lambda x: [expr0, ...]``. + - Brackets denote a multiple-expression lambda body. Technically, the brackets create a `do[]` environment. + - If you want your lambda to have one expression that is a literal list, double the brackets: `lambda x: [[5 * x]]`. + - Lambdas are automatically named whenever the machinery can figure out a name from the surrounding context. + - When not, source location is auto-injected into the name. - - TCO in both ``def`` and ``lambda``, fully automatic - - Omit ``return`` in any tail position, like in Lisps - - Multiple-expression lambdas, ``lambda x: [expr0, ...]`` - - Automatically named lambdas whenever the machinery can figure out a name; when not, source location is auto-injected into the name. - - The underscore: e.g. `fn[_ * 3]` becomes `lambda x: x * 3`, and `fn[_ * _]` becomes `lambda x, y: x * y`. +The multi-expression lambda syntax uses ``do[]``, so it also allows lambdas to manage local variables using ``local[name << value]`` and ``delete[name]``. See the documentation of ``do[]`` for details. -We also import some macros and functions to serve as dialect builtins: +If you need more stuff, `unpythonic` is effectively the standard library of Lispython, on top of what Python itself already provides. - - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax`` - - ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil``, ``prod`` - - ``dyn``, for dynamic assignment - - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, the `pipe` family, the `compose` family, and the `with continuations` macro.) +There are **two variants** of the dialect, `Lispython` and `Lispy`. -For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. -The multi-expression lambda syntax uses ``do[]``, so it also allows lambdas to manage local variables using ``local[name << value]`` and ``delete[name]``. See the documentation of ``do[]`` for details. +### The `Lispy` variant -The builtin ``let[]`` constructs are ``let``, ``letseq``, ``letrec``, the decorator versions ``dlet``, ``dletseq``, ``dletrec``, the block versions (decorator, call immediately, replace def'd name with result) ``blet``, ``bletseq``, ``bletrec``, and the code-splicing variants ``let_syntax`` and ``abbrev``. Bindings may be made using any syntax variant supported by ``unpythonic.syntax``. +In the `Lispy` variant, that's it - the dialect changes the semantics only. Nothing is imported implicitly, except the macros injected by the dialect template (to perform the whole-module semantic changes at macro expansion time). -The builtin ``do[]`` constructs are ``do`` and ``do0``. +This is the pythonic variant of Lispython, keeping in line with *explicit is better than implicit*. The rule is: *if a name appears in user code, it must be defined explicitly*, as is usual in Python. -If you need more stuff, `unpythonic` is effectively the standard library of Lispython, on top of what Python itself already provides. +Note this implies that you must **explicitly import** the `local[]` macro if you want to declare local variables in a multiple-expression lambda, and the `fn[]` macro if you want to take advantage of the implicit `quicklambda`. Both are available in `unpythonic.syntax`, as usual. + +The point of the implicit `quicklambda` is that all invocations of `fn[]`, if there are any, will expand early, so that other macros that expect lambdas to be in standard Python notation will get exactly that. This includes other macros invoked by the dialect definition, namely `multilambda` and `namedlambda`. + +The main point of `Lispy`, compared to plain Python, is automatic TCO. The ability to omit `return` is a minor convenience, and the other three features only improve the usability of lambdas. + + +### The `Lispython` variant + +In the `Lispython` variant, we implicitly import some macros and functions to serve as dialect builtins, keeping in line with expectations for a ~language in the~ somewhat distant relative of the Lisp family: + + - ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil``, ``prod``. + - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax``. + - The underscore: e.g. `fn[_ * 3]` becomes `lambda x: x * 3`, and `fn[_ * _]` becomes `lambda x, y: x * y`. + - ``dyn``, for dynamic assignment. + - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, the `pipe` family, the `compose` family, and the `with continuations` macro.) + +For detailed documentation of the language features, see [``unpythonic.syntax``](../macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. + +The dialect builtin ``let[]`` constructs are ``let``, ``letseq``, ``letrec``, the decorator versions ``dlet``, ``dletseq``, ``dletrec``, the block versions (decorator, call immediately, replace def'd name with result) ``blet``, ``bletseq``, ``bletrec``, and the code-splicing variants ``let_syntax`` and ``abbrev``. Bindings may be made using any syntax variant supported by ``unpythonic.syntax``. + +The dialect builtin ``do[]`` constructs are ``do`` and ``do0``. ## What Lispython is @@ -142,7 +174,7 @@ Lispython works with ``with continuations``, because: - The same applies to the outside-in pass of ``namedlambda``. Its inside-out pass, on the other hand, must come after ``continuations``, which it does, since the dialect's implicit ``with namedlambda`` is in a lexically outer position with respect to the ``with continuations``. -Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in fact tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython happily auto-injects a ``return`` to whatever is the last statement in any particular function. +Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in fact tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython's implicit ``autoreturn`` happily auto-injects a ``return`` to whatever is the last statement in any particular function. ## Why extend Python? @@ -151,7 +183,7 @@ Be aware, though, that the combination of the ``autoreturn`` implicit in the dia Python, on the other hand, has a slight edge in usability to the end-user programmer, and importantly, a huge ecosystem of libraries, second to ``None``. Python is where science happens (unless you're in CS). Python is an almost-Lisp that has delivered on [the productivity promise](http://paulgraham.com/icad.html) of Lisp. Python also gets many things right, such as well developed support for lazy sequences, and decorators. -In certain other respects, Python the base language leaves something to be desired, if you have been exposed to Racket (or Haskell, but that's a different story). Writing macros is harder due to the irregular syntax, but thankfully MacroPy already exists, and any set of macros only needs to be created once. +In certain other respects, Python the base language leaves something to be desired, if you have been exposed to Racket (or Haskell, but that's a different story). Writing macros is harder due to the irregular syntax, but thankfully macro expanders already exist, and any set of macros only needs to be created once. Practicality beats purity ([ZoP §9](https://www.python.org/dev/peps/pep-0020/)): hence, fix the minor annoyances that would otherwise quickly add up, and reap the benefits of both worlds. If Python is software glue, Lispython is an additive that makes it flow better. diff --git a/unpythonic/dialects/__init__.py b/unpythonic/dialects/__init__.py index 9de09993..1326d62d 100644 --- a/unpythonic/dialects/__init__.py +++ b/unpythonic/dialects/__init__.py @@ -12,6 +12,6 @@ """ # re-exports -from .lispython import Lispython # noqa: F401 +from .lispython import Lispython, Lispy # noqa: F401 from .listhell import Listhell # noqa: F401 from .pytkell import Pytkell # noqa: F401 diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index 73bdedd0..b6a893b6 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -41,3 +41,32 @@ def transform_ast(self, tree): # tree is an ast.Module __paste_here__ # noqa: F821, just a splicing marker. tree.body = splice_dialect(tree.body, template, "__paste_here__") return tree + + +class Lispy(Dialect): + """**Pythonistas rejoice!** + + O language like Lisp, like Python! + Semantic changes sensibly carry, + Python's primary virtue vindicate. + Ire me not with implicit imports, + Let my IDE label mistakes. + """ + + def transform_ast(self, tree): # tree is an ast.Module + with q as template: + __lang__ = "Lispy" # noqa: F841, just provide it to user code. + from unpythonic.syntax import (macros, tco, autoreturn, # noqa: F401, F811 + multilambda, quicklambda, namedlambda) + # The important point is none of these expect the user code to look like + # anything but regular Python, so IDEs won't yell about undefined names; + # just the semantics are slightly different. + # + # Even if the user code uses `fn[]` (to make `quicklambda` actually do anything), + # that macro must be explicitly imported. It works, because `splice_dialect` + # hoists macro-imports from the top level of the user code into the top level + # of the template. + with autoreturn, quicklambda, multilambda, namedlambda, tco: + __paste_here__ # noqa: F821, just a splicing marker. + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree From bb08bef530e24d6d0b65e815531b1226c0678ae0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 02:16:34 +0300 Subject: [PATCH 446/832] also tco expects standard lambdas --- doc/dialects/lispython.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index f93bd43c..42b8a1fc 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -108,7 +108,7 @@ This is the pythonic variant of Lispython, keeping in line with *explicit is bet Note this implies that you must **explicitly import** the `local[]` macro if you want to declare local variables in a multiple-expression lambda, and the `fn[]` macro if you want to take advantage of the implicit `quicklambda`. Both are available in `unpythonic.syntax`, as usual. -The point of the implicit `quicklambda` is that all invocations of `fn[]`, if there are any, will expand early, so that other macros that expect lambdas to be in standard Python notation will get exactly that. This includes other macros invoked by the dialect definition, namely `multilambda` and `namedlambda`. +The point of the implicit `quicklambda` is that all invocations of `fn[]`, if there are any, will expand early, so that other macros that expect lambdas to be in standard Python notation will get exactly that. This includes other macros invoked by the dialect definition, namely `multilambda`, `namedlambda`, and `tco`. The main point of `Lispy`, compared to plain Python, is automatic TCO. The ability to omit `return` is a minor convenience, and the other three features only improve the usability of lambdas. From 8db9f144eeb031926b3350cc58803a40b3879dd3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 02:17:01 +0300 Subject: [PATCH 447/832] alphabetize imports --- unpythonic/syntax/lambdatools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 764367d4..39cb0711 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -21,8 +21,8 @@ from mcpyrate.walkers import ASTTransformer from ..dynassign import dyn -from ..misc import namelambda from ..env import env +from ..misc import namelambda from .astcompat import getconstant, Str, NamedExpr from .letdo import _implicit_do, _do From a266056f1ca76c915236b36ba1506c466575eb86 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 02:18:24 +0300 Subject: [PATCH 448/832] add unpythonic.syntax._ to make IDEs happy You can `from unpythonic.syntax import _` to silence any "undefined name" errors regarding the use of `_`, e.g. in `fn[_ * 3]`. It's a regular run-time object that does nothing; its only purpose is to give the name an explicit definition. (Technically, it's an `unpythonic.symbol.sym`, which seemed appropriate for this.) --- doc/macros.md | 13 ++++++++----- unpythonic/syntax/lambdatools.py | 10 +++++++++- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 28ff9bd6..1117b574 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -710,7 +710,9 @@ Support for other forms of assignment may or may not be added in a future versio *The name is now `fn[]`. This was changed because `f` is often used as a function name in code examples, local temporaries, and similar. Also, `fn[]` is a less ambiguous abbreviation for a syntactic construct that means *function*, while remaining shorter than the equivalent `lambda`.* -*The underscore `_` is no longer a macro on its own. The `fn` macro treats the underscore magically, as `f` did before, but anywhere else the underscore is available to be used as a regular variable. If you use `fn[]`, change your import of this macro to `from unpythonic.syntax import macros, fn`.* +*The underscore `_` is no longer a macro on its own. The `fn` macro treats the underscore magically, as `f` did before, but anywhere else the underscore is available to be used as a regular variable. If you use `fn[]`, change your import of this macro to `from unpythonic.syntax import macros, fn`.** + +*The underscore does **not** need to be imported for `fn[]` to recognize it. But if you want to make your IDE happy, there is a symbol named `_` in `unpythonic.syntax` you can import to silence any "undefined name" errors regarding the use of `_`. It is a regular run-time object, not a macro.* The syntax ``fn[...]`` creates a lambda, where each underscore in the ``...`` part introduces a new parameter. The macro does not descend into any nested ``fn[]``. @@ -734,6 +736,7 @@ Example - a quick multilambda: ```python from unpythonic.syntax import macros, multilambda, quicklambda, fn, local +from unpythonic.syntax import _ # optional, makes IDEs happy with quicklambda, multilambda: func = fn[[local[x << _], @@ -742,16 +745,16 @@ with quicklambda, multilambda: assert func(1, 2) == 3 ``` -This is of course rather silly, as an unnamed formal parameter can only be mentioned once. If we're giving names to them, a regular ``lambda`` is shorter to write. A more realistic combo is: +This is of course rather silly, as an unnamed formal parameter can only be mentioned once. If we are giving names to them, a regular ``lambda`` is shorter to write. A more realistic combo is: ```python with quicklambda, tco: def g(x): - return 2*x - func1 = fn[g(3*_)] # tail call + return 2 * x + func1 = fn[g(3 * _)] # tail call assert func1(10) == 60 - func2 = fn[3*g(_)] # no tail call + func2 = fn[3 * g(_)] # no tail call assert func2(10) == 60 ``` diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 39cb0711..5127852e 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -3,7 +3,7 @@ __all__ = ["multilambda", "namedlambda", - "fn", + "fn", "_", "quicklambda", "envify"] @@ -23,6 +23,7 @@ from ..dynassign import dyn from ..env import env from ..misc import namelambda +from ..symbol import sym from .astcompat import getconstant, Str, NamedExpr from .letdo import _implicit_do, _do @@ -146,6 +147,10 @@ def fn(tree, *, syntax, expander, **kw): is not a macro, and has no special meaning outside ``fn[]``. The underscore does **not** need to be imported for ``fn[]`` to recognize it. + But if you want to make your IDE happy, there is a symbol named ``_`` in + `unpythonic.syntax` you can import to silence any "undefined name" errors + regarding the use of ``_``. It is a regular run-time object, not a macro. + The macro does not descend into any nested ``fn[]``. """ if syntax != "expr": @@ -158,6 +163,8 @@ def fn(tree, *, syntax, expander, **kw): return _fn(tree, mynames) +_ = sym("_") # for those who want to make their IDEs happy + def quicklambda(tree, *, syntax, expander, **kw): """[syntax, block] Make ``fn`` quick lambdas expand first. @@ -177,6 +184,7 @@ def quicklambda(tree, *, syntax, expander, **kw): Example - a quick multilambda:: from unpythonic.syntax import macros, multilambda, quicklambda, fn, local + from unpythonic.syntax import _ # optional, makes IDEs happy with quicklambda, multilambda: func = fn[[local[x << _], From 0a785a3be5a905e4af771c068953dddbf354e637 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 02:20:51 +0300 Subject: [PATCH 449/832] oops, add missing unit test module --- unpythonic/dialects/tests/test_lispy.py | 107 ++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 unpythonic/dialects/tests/test_lispy.py diff --git a/unpythonic/dialects/tests/test_lispy.py b/unpythonic/dialects/tests/test_lispy.py new file mode 100644 index 00000000..eb58da92 --- /dev/null +++ b/unpythonic/dialects/tests/test_lispy.py @@ -0,0 +1,107 @@ +# -*- coding: utf-8 -*- +"""Test the Lispy dialect. + +Like Lispython, but more pythonic: nothing is imported implicitly, +except the macros injected by the dialect template (to perform the +whole-module semantic changes at macro expansion time). +""" + +from ...dialects import dialects, Lispy # noqa: F401 + +from ...syntax import macros, test, the # noqa: F401 +from ...test.fixtures import session, testset + +from ...syntax import macros, continuations, call_cc, letrec, fn, local, cond # noqa: F401, F811 +from ...syntax import _ # optional, makes IDEs happy +from ...funutil import Values + +def runtests(): + print(f"Hello from {__lang__}!") # noqa: F821, the dialect template defines it. + + # auto-TCO (both in defs and lambdas), implicit return in tail position + with testset("implicit tco, implicit autoreturn"): + def fact(n): + def f(k, acc): + if k == 1: + return acc # "return" still available for early return + f(k - 1, k * acc) + f(n, acc=1) + test[fact(4) == 24] + fact(5000) # no crash (and correct result, since Python uses bignums transparently) + + t = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp << (lambda x:(x != 0) and evenp(x - 1))] in # noqa: F821 + evenp(10000)] # no crash # noqa: F821 + test[t is True] + + # lambdas are named automatically + with testset("implicit namedlambda"): + square = lambda x: x**2 + test[square(3) == 9] + test[square.__name__ == "square"] + + # the underscore (in Lispy, the `fn` macro must be imported explicitly) + cube = fn[_**3] + test[cube(3) == 27] + test[cube.__name__ == "cube"] + + my_mul = fn[_ * _] + test[my_mul(2, 3) == 6] + test[my_mul.__name__ == "my_mul"] + + # lambdas can have multiple expressions and local variables + # + # If you need to return a literal list from a lambda, use an extra set of + # brackets; the outermost brackets always enable multiple-expression mode. + # + with testset("implicit multilambda"): + # In Lispy, the `local` macro must be imported explicitly. + # `local[name << value]` makes a local variable in a multilambda (or in any `do[]` environment). + mylam = lambda x: [local[y << 2 * x], # noqa: F821 + y + 1] # noqa: F821 + test[mylam(10) == 21] + + a = lambda x: [local[t << x % 2], # noqa: F821 + cond[t == 0, "even", # noqa: F821 + t == 1, "odd", + None]] # cond[] requires an else branch + test[a(2) == "even"] + test[a(3) == "odd"] + + # MacroPy #21; namedlambda must be in its own with block in the + # dialect implementation or this particular combination will fail + # (uncaught jump, __name__ not set). + # + # With `mcpyrate` this shouldn't matter, but we're keeping the example. + with testset("autonamed letrec lambdas, multiple-expression let body"): + t = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp << (lambda x:(x != 0) and evenp(x - 1))] in # noqa: F821 + [local[x << evenp(100)], # noqa: F821, multi-expression let body is a do[] environment + (x, evenp.__name__, oddp.__name__)]] # noqa: F821 + test[t == (True, "evenp", "oddp")] + + with testset("integration with continuations"): + with continuations: # has TCO; should be skipped by the implicit `with tco` inserted by the dialect + k = None # kontinuation + def setk(*args, cc): + nonlocal k + k = cc # current continuation, i.e. where to go after setk() finishes + Values(*args) # multiple-return-values + def doit(): + lst = ['the call returned'] + *more, = call_cc[setk('A')] + lst + list(more) + test[doit() == ['the call returned', 'A']] + # We can now send stuff into k, as long as it conforms to the + # signature of the assignment targets of the "call_cc". + test[k('again') == ['the call returned', 'again']] + test[k('thrice', '!') == ['the call returned', 'thrice', '!']] + + # We must have some statement here to make the implicit autoreturn happy, + # because the continuations testset is the last one, and the top level of + # a `with continuations` block is not allowed to have a `return`. + pass + +if __name__ == '__main__': + with session(__file__): + runtests() From 6a291b357e998d12a696916bf407746f020e6a67 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 02:28:45 +0300 Subject: [PATCH 450/832] improve doc --- doc/dialects/lispython.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 42b8a1fc..f5858a4d 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -115,7 +115,7 @@ The main point of `Lispy`, compared to plain Python, is automatic TCO. The abili ### The `Lispython` variant -In the `Lispython` variant, we implicitly import some macros and functions to serve as dialect builtins, keeping in line with expectations for a ~language in the~ somewhat distant relative of the Lisp family: +In the `Lispython` variant, we implicitly import some macros and functions to serve as dialect builtins, keeping in line with expectations for a ~language in the~ *somewhat distant relative of the* Lisp family: - ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil``, ``prod``. - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax``. @@ -132,7 +132,7 @@ The dialect builtin ``do[]`` constructs are ``do`` and ``do0``. ## What Lispython is -Lispython is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.lispython`](../../unpythonic/dialects/lispython.py). Usage examples can be found in [the unit tests](../../unpythonic/dialects/tests/test_lispython.py). +Lispython is a dialect of Python implemented via macros and a thin whole-module AST transformation. The dialect definition lives in [`unpythonic.dialects.lispython`](../../unpythonic/dialects/lispython.py). Usage examples can be found in the unit tests, [for `Lispy`](../../unpythonic/dialects/tests/test_lispy.py) and [for `Lispython`](../../unpythonic/dialects/tests/test_lispython.py). Lispython essentially makes Python feel slightly more lispy, in parts where that makes sense. From 37636b06ae6dca256e49a3daf760f1907ee27e6f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 02:37:30 +0300 Subject: [PATCH 451/832] add Python 3.8+ solution to PG's accumulator-generator puzzle --- doc/dialects/lispython.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index f5858a4d..038265c4 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -208,7 +208,7 @@ foo = lambda n0: let[[n << n0] in (lambda i: n << n + i)] ``` -This still sets up a separate place for the accumulator (that is, separate from the argument of the outer function). The modern pure Python solution avoids that, but needs many lines: +This still sets up a separate place for the accumulator (that is, separate from the argument of the outer function). The pure Python 3 solution avoids that, but needs many lines: ```python def foo(n): @@ -219,7 +219,17 @@ def foo(n): return accumulate ``` -The problem is that assignment to a lexical variable (including formal parameters) is a statement in Python. Python 3.8's walrus operator does not solve this, because `n := n + i` by itself is a syntax error, and even if parenthesized, `(n := n + i)` insists on creating a new local variable `n`. +The Python 3.8+ solution, using the new walrus operator, is one line shorter: + +```python +def foo(n): + def accumulate(i): + nonlocal n + return (n := n + i) + return accumulate +``` + +This is rather clean, but still needs the `nonlocal` declaration, which is a statement. If we abbreviate ``accumulate`` as a lambda, it needs a ``let`` environment to write in, to use `unpythonic`'s expression-assignment (`name << value`). From 529357dcd5553bc57e4206b393ff72b8e3e8e4be Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 04:01:14 +0300 Subject: [PATCH 452/832] refactor essays into their own doc --- CONTRIBUTING.md | 1 + README.md | 1 + doc/design-notes.md | 63 +------------- doc/dialects.md | 1 + doc/dialects/lispython.md | 1 + doc/dialects/listhell.md | 1 + doc/dialects/pytkell.md | 1 + doc/essays.md | 167 ++++++++++++++++++++++++++++++++++++++ doc/features.md | 1 + doc/macros.md | 1 + doc/readings.md | 51 ++---------- doc/repl.md | 1 + doc/troubleshooting.md | 1 + 13 files changed, 188 insertions(+), 103 deletions(-) create mode 100644 doc/essays.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a45ba7b6..e23e569e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,6 +7,7 @@ - [REPL server](doc/repl.md) - [Troubleshooting](doc/troubleshooting.md) - [Design notes](doc/design-notes.md) +- [Essays](doc/essays.md) - [Additional reading](doc/readings.md) - **Contribution guidelines** diff --git a/README.md b/README.md index 57a6cb99..2d645c8d 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (langua - [REPL server](doc/repl.md): interactively hot-patch your running Python program. - [Troubleshooting](doc/troubleshooting.md): possible solutions to possibly common issues. - [Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic``. +- [Essays](doc/essays.md): for writings on the philosophy of ``unpythonic``, things that inspired it, and related discoveries. - [Additional reading](doc/readings.md): links to material relevant in the context of ``unpythonic``. - [Contribution guidelines](CONTRIBUTING.md): for understanding the codebase, or if you're interested in making a code or documentation PR. diff --git a/doc/design-notes.md b/doc/design-notes.md index 3dba46a0..b7e1fbd0 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -7,6 +7,7 @@ - [REPL server](repl.md) - [Troubleshooting](troubleshooting.md) - **Design notes** +- [Essays](essays.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) @@ -16,9 +17,7 @@ - [Design Philosophy](#design-philosophy) - [Macros do not Compose](#macros-do-not-compose) - [Language Discontinuities](#language-discontinuities) - - [What Belongs in Python?](#what-belongs-in-python) - - [Killer features of Common Lisp](#killer-features-of-common-lisp) - - [Common Lisp, Python, and productivity](#common-lisp-python-and-productivity) + - [`unpythonic` and the Killer Features of Common Lisp](#unpythonic-and-the-killer-features-of-common-lisp) - [Python is not a Lisp](#python-is-not-a-lisp) - [On ``let`` and Python](#on-let-and-python) - [Assignment syntax](#assignment-syntax) @@ -79,42 +78,7 @@ For another example, it is likely that e.g. `continuations` still does not integ For a third example, consider *decorated lambdas*. This is an `unpythonic` extension - essentially, a compiler feature implemented (by calling some common utility code) by each of the transformers of the pure-macro features - that understands a lambda enclosed in a nested sequence of single-argument function calls *as a decorated function definition*. This is painful, because the Python AST has no place to store the decorator list for a lambda; Python sees it just as a nested sequence of function calls, terminating in a lambda. This has to be papered over by the transformers. We also introduce a related complication, the decorator registry (see `regutil`), so that we can automatically sort decorator invocations - so that pure-macro features know at which index to inject a particular decorator (so it works properly) when they need to do that. Needing such a registry is already a complication, but the *decorated lambda* machinery feels the pain more acutely. -## What Belongs in Python? - -If you feel [my hovercraft is full of eels](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), it is because they come with the territory. - -Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). - -In general, I like Python. Also, my hat is off to the devs. It is no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the particular points above, if I agreed, I would not have built `unpythonic`, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. - -I think that with macros, Python can be so much more than just a beginner's language. Language-level extensibility is just the logical endpoint of that. I do not share the sentiment of the Python community against metaprogramming, or toward some language-level features. For me, macros (and full-module transforms a.k.a. dialects) are just another tool for creating abstractions, at yet another level. We can already extract procedures, methods, and classes. Why limit that ability - namely, the ability to create abstractions - to what an [eager](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) language can express at run time? - -If the point is to keep code understandable, I respect the goal; but that is a matter of education. It is perfectly possible to write unreadable code without macros, and in Python, no less. Just use a complex class hierarchy so that the programmer reading the code must hunt through everything to find each method definition; write big functions without abstracting the steps of the overall algorithm; keep lots of mutable state, and store it in top-level variables; and maybe top that off with an overuse of dependency injection. No one will be able to figure out how the program works, at least not in any reasonable amount of time. - -It is also perfectly possible to write readable code with macros. Just keep in mind that macros are a different kind of abstraction, and use them where that kind of abstraction lends itself to building a clean solution. I am willing to admit the technical objection that *macros do not compose*; but that does not make them useless. - -Of the particular points above, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hang, when entering an accidentally infinite mutual recursion. I do occasionally make such mistakes when developing complex algorithms - especially when quickly sketching out new ideas. But sometimes, it would be nice to enable TCO selectively. If you ask for it, you know what to expect. This is precisely why `unpythonic.syntax` has `with tco`. I am not very happy with a custom TCO layer on top of a language core that eschews the whole idea, because TCO support in the core (like Scheme and Racket have) would simplify the implementation of certain other language extensions; but then again, [this is exactly what Clojure did](https://clojuredocs.org/clojure.core/trampoline), in similar technical circumstances. - -As for a multi-expression `lambda`, on the surface it sounds like a good idea. But really the issue is that in Python, the `lambda` construct itself is broken. It is essentially a duplicate of `def`, but lacking some features. As of Python 3.8, the latest addition of insult to injury is the lack of support for type annotations. A more uniform solution would be to make `def` into an expression. Much of the time, anonymous functions are not a good idea, because names help understanding and debugging - especially when all you have is a traceback. But defining closures inline **is** a great idea - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. - -The macros in `unpythonic.syntax` inject many lambdas, because that makes them much simpler to implement than if we had to always lift a `def` statement into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if one could define also slightly more complex actions inline (see [Racket's pattern matcher](https://docs.racket-lang.org/reference/match.html) for a comparison). It is unlikely that the action functions will be needed elsewhere, and it is just silly to define a bunch of functions *before* the call to `match`. If this is not a job for either something like `let-where` (to invert the presentation order locally) or a multi-expression lambda (to define the actions inline), I do not know what is. - -While on the topic of usability, why are lambdas strictly anonymous? In cases where it is useful to be able to omit a name, because sometimes many small helper functions may be needed and [naming is hard](https://martinfowler.com/bliki/TwoHardThings.html), why not include the source location information in the auto-generated name, instead of just `""`? (As of v0.15.0, the `with namedlambda` macro does this.) - -On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the parser mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. - -It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) - -As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. - -Finally, there is the issue of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/)). It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its user, and it is not very popular in the programming community at large. - -What I can say is, `unpythonic` is not meant for the average Python project, either. If used intelligently, it can make code shorter, yet readable. For a lone developer who needs to achieve as much as possible in the fewest lines reasonably possible, it seems to me that language extension - and in general, as Alexis King put it, [climbing the infinite ladder of abstraction](https://lexi-lambda.github.io/blog/2016/08/11/climbing-the-infinite-ladder-of-abstraction/) - is the way to go. In a large project with a high developer turnover, the situation is different. - -For general programming in the early 2020s, Python still has the ecosystem advantage, so it does not make sense to move to anything else, at least yet. So, let us empower what we have. Even if we have to build something that could be considered *unpythonic*. - - -## Killer features of Common Lisp +## `unpythonic` and the Killer Features of Common Lisp In my opinion, Common Lisp has three legendary killer features: @@ -145,27 +109,6 @@ But for those of us that [don't like parentheses](https://srfi.schemers.org/srfi - For the use case of numerics specifically, instead of Python, [Julia](https://docs.julialang.org/en/v1/manual/methods/) may be a better fit for writing high-level, yet performant code. It's a spiritual heir of Common Lisp, Fortran, *and Python*. Compilation to efficient machine code, with the help of gradual typing and automatic type inference, is a design goal. -## Common Lisp, Python, and productivity - -The various essays by Paul Graham, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for Lisp. So how does the programming world look in that light now, 20 years later? - -The base abstraction level of programming languages, even those in popular use, has increased. The trend was visible already then, and was indeed noted in the essays. The focus on low-level languages such as C++ has decreased. Java is still popular, but high-level FP languages that compile to JVM bytecode (Kotlin, Scala, Clojure) are rising. - -Python has become highly popular, and is now also closer to Lisp than it was 20 years ago, especially after `MacroPy` introduced syntactic macros to Python (in 2013, [according to the git log](https://github.com/lihaoyi/macropy/commits/python2/macropy/__init__.py)). Python wasn't bad as a Lisp replacement even back in 2000 - see Peter Norvig's essay [Python for Lisp Programmers](https://norvig.com/python-lisp.html). Some more historical background, specifically on lexically scoped closures (and the initial lack thereof), can be found in [PEP 3104](https://www.python.org/dev/peps/pep-3104/), [PEP 227](https://www.python.org/dev/peps/pep-0227/), and [Historical problems with closures in JavaScript and Python](http://giocc.com/problems-with-closures-in-javascript-and-python.html). - -In 2020, does it still make sense to learn [the legendary](https://xkcd.com/297/) Common Lisp? - -To know exactly what it has to offer, yes. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. Having more perspectives at one's disposal makes one a better programmer. - -But as a practical tool? Is CL hands-down better than Python? Maybe no. Python has already delivered on 90% of the productivity promise of Lisp. Both languages cut down significantly on [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet). Python has a huge library ecosystem. [`mcpyrate`](https://github.com/Technologicat/mcpyrate) and `unpythonic` are trying to push the language-level features a further 5%. (A full 100% is likely impossible when extending an existing language; if nothing else, there will be seams.) - -As for productivity, [it may be](https://medium.com/smalltalk-talk/lisp-smalltalk-and-the-power-of-symmetry-8bd96aaa0c0c) that a form of code-data equivalence (symmetry!), not macros specifically, is what makes Lisp powerful. If so, there may be other ways to reach that equivalence. For example Smalltalk, like Lisp, *runs in the same context it's written in*. All Smalltalk data are programs. Smalltalk [may be making a comeback](https://hackernoon.com/how-to-evangelize-a-programming-language-0p7p3y02), in the form of [Pharo](https://pharo.org/). - -Haskell aims at code-data equivalence from a third angle (memoized pure functions are in essence infinite lookup tables), but I haven't used it in practice, so I don't have the experience to say whether this is enough to make it feel powerful in the same way. - -Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world isn't that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead (without restarting the whole app at each change). Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. - - ## Python is not a Lisp The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macros.md)) is to make Python lambdas slightly more useful - which was really the starting point for the whole `unpythonic` experiment. diff --git a/doc/dialects.md b/doc/dialects.md index ab01d2ee..3443416c 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -10,6 +10,7 @@ - [REPL server](repl.md) - [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) +- [Essays](essays.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 038265c4..161ef5f3 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -10,6 +10,7 @@ - [REPL server](../repl.md) - [Troubleshooting](../troubleshooting.md) - [Design notes](../design-notes.md) +- [Essays](../essays.md) - [Additional reading](../readings.md) - [Contribution guidelines](../../CONTRIBUTING.md) diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index 2e956da5..b5e8f441 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -10,6 +10,7 @@ - [REPL server](../repl.md) - [Troubleshooting](../troubleshooting.md) - [Design notes](../design-notes.md) +- [Essays](../essays.md) - [Additional reading](../readings.md) - [Contribution guidelines](../../CONTRIBUTING.md) diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index cd3dccdd..f1325d9f 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -10,6 +10,7 @@ - [REPL server](../repl.md) - [Troubleshooting](../troubleshooting.md) - [Design notes](../design-notes.md) +- [Essays](../essays.md) - [Additional reading](../readings.md) - [Contribution guidelines](../../CONTRIBUTING.md) diff --git a/doc/essays.md b/doc/essays.md new file mode 100644 index 00000000..e2f78af0 --- /dev/null +++ b/doc/essays.md @@ -0,0 +1,167 @@ +**Navigation** + +- [README](../README.md) +- [Pure-Python feature set](features.md) +- [Syntactic macro feature set](macros.md) +- [Examples of creating dialects using `mcpyrate`](dialects.md) +- [REPL server](repl.md) +- [Troubleshooting](troubleshooting.md) +- [Design notes](design-notes.md) +- **Essays** +- [Additional reading](readings.md) +- [Contribution guidelines](../CONTRIBUTING.md) + + +**Table of Contents** + +- [What Belongs in Python?](#what-belongs-in-python) +- [`hoon`: The C of Functional Programming](#hoon-the-c-of-functional-programming) +- [Common Lisp, Python, and productivity](#common-lisp-python-and-productivity) + + + + +# What Belongs in Python? + +You may feel that [my hovercraft is full of eels](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html). It is because they come with the territory. + +Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). + +In general, I like Python. My hat is off to the devs. It is no mean feat to create a high-level language that focuses on readability and approachability, keep it alive for 30 years and counting, and have a large part of the programming community adopt it. But regarding the particular points above, if I agreed, I would not have built `unpythonic`, or [`mcpyrate`](https://github.com/Technologicat/mcpyrate) either. + +I think that with macros, Python can be so much more than just a beginner's language. Language-level extensibility is just the logical endpoint of that. I do not share the sentiment of the Python community against metaprogramming, or toward some language-level features. For me, macros (and full-module transforms a.k.a. dialects) are just another tool for creating abstractions, at yet another level. We can already extract procedures, methods, and classes. Why limit that ability - namely, the ability to create abstractions - to what an [eager](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) language can express at run time? + +If the point is to keep code understandable, I respect the goal; but that is a matter of education. It is perfectly possible to write unreadable code without macros, and in Python, no less. Just use a complex class hierarchy so that the programmer reading the code must hunt through everything to find each method definition; write big functions without abstracting the steps of the overall algorithm; keep lots of mutable state, and store it in top-level variables; and maybe top that off with an overuse of dependency injection. No one will be able to figure out how the program works, at least not in any reasonable amount of time. + +It is also perfectly possible to write readable code with macros. Just keep in mind that macros are a different kind of abstraction, and use them where that kind of abstraction lends itself to building a clean solution. I am willing to admit the technical objection that *macros do not compose*; but that does not make them useless. + +Of the particular points above, in my opinion TCO should at least be an option. I like that *by default*, Python will complain about a call stack overflow rather than hang, when entering an accidentally infinite mutual recursion. I do occasionally make such mistakes when developing complex algorithms - especially when quickly sketching out new ideas. But sometimes, it would be nice to enable TCO selectively. If you ask for it, you know what to expect. This is precisely why `unpythonic.syntax` has `with tco`. I am not very happy with a custom TCO layer on top of a language core that eschews the whole idea, because TCO support in the core (like Scheme and Racket have) would simplify the implementation of certain other language extensions; but then again, [this is exactly what Clojure did](https://clojuredocs.org/clojure.core/trampoline), in similar technical circumstances. + +As for a multi-expression `lambda`, on the surface it sounds like a good idea. But really the issue is that in Python, the `lambda` construct itself is broken. It is essentially a duplicate of `def`, but lacking some features. As of Python 3.8, the latest addition of insult to injury is the lack of support for type annotations. A more uniform solution would be to make `def` into an expression. Much of the time, anonymous functions are not a good idea, because names help understanding and debugging - especially when all you have is a traceback. But defining closures inline **is** a great idea - and sometimes, the most readily understandable presentation order for an algorithm requires to do that in an expression position. The convenience is similar to being able to nest `def` statements, an ability Python already has. + +The macros in `unpythonic.syntax` inject many lambdas, because that makes them much simpler to implement than if we had to always lift a `def` statement into the nearest enclosing statement context. Another case in point is [`pampy`](https://github.com/santinic/pampy). The code to perform a pattern match would read a lot nicer if one could define also slightly more complex actions inline (see [Racket's pattern matcher](https://docs.racket-lang.org/reference/match.html) for a comparison). It is unlikely that the action functions will be needed elsewhere, and it is just silly to define a bunch of functions *before* the call to `match`. If this is not a job for either something like `let-where` (to invert the presentation order locally) or a multi-expression lambda (to define the actions inline), I do not know what is. + +While on the topic of usability, why are lambdas strictly anonymous? In cases where it is useful to be able to omit a name, because sometimes many small helper functions may be needed and [naming is hard](https://martinfowler.com/bliki/TwoHardThings.html), why not include the source location information in the auto-generated name, instead of just `""`? (As of v0.15.0, the `with namedlambda` macro does this.) + +On a point raised [here by the BDFL](https://www.artima.com/weblogs/viewpost.jsp?thread=147358), with respect to indentation-sensitive vs. indentation-insensitive parser modes; having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the parser mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. + +It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) + +As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. + +Finally, there is the issue of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/)). It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its user, and it is not very popular in the programming community at large. + +What I can say is, `unpythonic` is not meant for the average Python project, either. If used intelligently, it can make code shorter, yet readable. For a lone developer who needs to achieve as much as possible in the fewest lines reasonably possible, it seems to me that language extension - and in general, as Alexis King put it, [climbing the infinite ladder of abstraction](https://lexi-lambda.github.io/blog/2016/08/11/climbing-the-infinite-ladder-of-abstraction/) - is the way to go. In a large project with a high developer turnover, the situation is different. + +For general programming in the early 2020s, Python still has the ecosystem advantage, so it does not make sense to move to anything else, at least yet. So, let us empower what we have. Even if we have to build something that could be considered *unpythonic*. + + +# `hoon`: The C of Functional Programming + +Some days I wonder if this `unpythonic` endeavor even makes any sense. Then, turning the pages of the [book of sand](https://en.wikipedia.org/wiki/The_Book_of_Sand) that is the web, I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. + +Its philosophy is best described by this gem from an [early version of its documentation](https://github.com/cgyarvin/urbit/blob/master/doc/book/0-intro.markdown#hoon): + +*So we could describe Hoon as a pure, strict, higher-order typed functional language. But don't do this in front of a Haskell purist, unless you put quotes around "typed," "functional," and possibly even "language." We could also say "object-oriented," with the same scare quotes for the cult of Eiffel.* + +While I am not sure if I will ever *use* `hoon`, it is hard not to like a language that puts quotes around "language". Few languages go that far in shaking up preconceptions. Critically examining what we believe, and why, often leads to useful insights. + +The claim that `hoon` is not a language, but a "language", fully makes sense after reading some of the documentation. `hoon` is essentially an *ab initio* language with an axiomatic approach to defining its operational semantics, similarly to how *Arc* approaches defining Lisp. Furthermore, `hoon` is the *functional equivalent of C* to the underlying virtual assembly language, `nock`. From a certain viewpoint, the "language" essentially consists of *glorified Nock macros*. Glorified assembly macros are pretty much all a *low-level* [HLL](https://en.wikipedia.org/wiki/High-level_programming_language) essentially is, so the claim seems about right. + +Nock is a peculiar assembly language. According to the comments in [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), it is a *Turing-complete non-lambda automaton*. The instruction set is permanently frozen, as if it was a physical CPU chip. Opcodes are just natural numbers, 0 through 11, and it is very minimalistic. For example, there is not even a decrement opcode. This is because from an axiomatic viewpoint, decrement can be defined recursively via increment. At which point, every systems programmer objects, rightfully, that no one sane actually does so, because that costs `O(n)`. Indeed, the `hoon` standard library uses C FFI to take advantage of the physical processor's instruction set to perform arithmetic operations. Each piece of C code used for such acceleration purposes is termed a *jet*. + +Since - by the fact that the programmer called a particular standard library function - the system knows we want to compute a decrement (or a multiplication, a power, maybe some floating point operation, etc.), it can *accelerate* that particular operation by using the available hardware. + +The important point is, you *could* write out a `nock` macro that does the same thing, only it would be unbearably slow. In the axiomatic perspective - which is about proving programs correct - speed does not matter. At the same time, FFI gives speed for the real world. + +To summarize; as someone already put it, `hoon` offers a glimpse into an alternate universe of systems programming, where the functional camp won. It may also be a useful tool, or a source for further unconventional ideas - but to know for sure, I will have to read more about it. + +I think the perfect place to end this piece is to quote a few lines from the language definition [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), to give a flavor: + +``` +++ doos :: sleep until + |= hap=path ^- (unit ,@da) + (doze:(wink:(vent bud (dink (dint hap))) now 0 (beck ~)) now [hap ~]) +:: +++ hurl :: start loop no id + |= ovo=ovum + ^- [p=(list ovum) q=(list ,[p=@tas q=vase])] + (kick [[~ [[(dint p.ovo) ~] p.ovo ~] q.ovo] ~]) +:: +++ hymn :: start loop with id + |= [who=ship ovo=ovum] + ^- [p=(list ovum) q=(list ,[p=@tas q=vase])] + (kick [[[~ %iron who] [[(dint p.ovo) ~] p.ovo ~] q.ovo] ~]) +:: +++ kick :: complete loop + |= mor=(list move) + =| ova=(list ovum) + |- ^- [p=(list ovum) q=(list ,[p=@tas q=vase])] + ?~ mor + [(flop ova) fan] + :: ~& [%kick-move q.i.mor -.r.i.mor] + ?> ?=(^ q.i.mor) + ?~ t.q.i.mor + $(mor t.mor, ova [[i.q.i.mor r.i.mor] ova]) + ?> ?=(^ i.q.i.mor) + =- $(mor (weld p.nyx t.mor), fan q.nyx) + ^= nyx + =+ naf=fan + |- ^- [p=(list move) q=_fan] + ?~ naf [~ ~] + ?. =(i.i.q.i.mor p.i.naf) + =+ tuh=$(naf t.naf) + [p.tuh [i.naf q.tuh]] + =+ ven=(vent bud q.i.naf) + =+ win=(wink:ven now (shax now) (beck p.i.mor)) + =+ ^= yub + %- beat:win + [p.i.mor t.i.q.i.mor t.q.i.mor r.i.mor] + [p.yub [[p.i.naf ves:q.yub] t.naf]] +-- +``` + +The Lisp family (particularly the Common Lisp branch) has a reputation for silly terminology, but I think `hoon` deserves the crown. All control structures are punctuation-only ASCII digraphs, and almost every name is a monosyllabic nonsense word. Still, this Lewis-Carroll-esque naming convention of making words mean what you define them to mean makes at least as much sense as the standard naming convention in mathematics, naming theorems after their discoverers! (Or at least, [after someone else](https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy).) + +I actually like the phonetic base, making numbers sound like [*sorreg-namtyv*](https://urbit.org/docs/hoon/hoon-school/nouns/); that is 5 702 400 for the rest of us. And I think I will, quite seriously, adopt the verb *bunt*, meaning *to take the default value of*. That is such a common operation in programming that I find it hard to believe there is no standard abbreviation. I wonder what other discoveries await. + +Finally, in some way I cannot quite put a finger on, to me the style has echoes of [Jorge Luis Borges](https://en.wikipedia.org/wiki/Jorge_Luis_Borges). I can imagine `hoon` as the *official* programming language of *[Tlön](https://en.wikipedia.org/wiki/Tl%C3%B6n%2C_Uqbar%2C_Orbis_Tertius)*. + +So maybe there is a place for `unpythonic`, too. + + +**Links** + +- [Latest documentation for `hoon`](https://urbit.org/docs/hoon/) +- There is a [whole operating system](https://github.com/urbit/urbit) built on `hoon` and `nock`. +- [Wikipedia has an entry on it](https://en.wikipedia.org/wiki/Urbit). Deconstructing the client-server model sounds very [postmodern](https://en.wikipedia.org/wiki/Deconstructivism). + + +**Note on natural-number opcodes** + +Using natural numbers for the opcodes at first glance sounds like a [Gödel numbering](https://en.wikipedia.org/wiki/G%C3%B6del_numbering) for the program space; but actually, the input to [the VM](https://urbit.org/docs/nock/definition/) contains some linked-list structure, which is not represented that way. Also, **any** programming language imposes its own Gödel numbering on the program space. Just take, for example, the UTF-8 representation of the source code text (which, in Python terms, is a `bytes` object), and interpret those bytes as one single bignum. + +Obviously, any interesting programs correspond to very large numbers, and are few and far between, so decoding random numbers via a Gödel numbering is not a practical way to generate interesting programs. [Genetic programming](https://en.wikipedia.org/wiki/Genetic_programming) works much better, because unlike Gödel numbering, it was actually designed specifically to do that. GP takes advantage of the semantic structure present in the source code (or AST) representation. + +The purpose of the original Gödel numbering was to prove Gödel's incompleteness theorem. In the case of `nock`, my impression is that the opcodes are natural numbers just for flavoring purposes. If you are building an ab initio software stack, what better way to announce that than to use natural numbers as your virtual machine's opcodes? + + +# Common Lisp, Python, and productivity + +The various essays Paul Graham wrote near the turn of the millennium, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for the Lisp family of languages. So how does the programming world look in that light now, 20 years later? + +The base abstraction level of programming languages, even those in popular use, has increased. The trend was visible already then, and was indeed noted in the essays. The focus on low-level languages such as C++ has decreased. Java is still popular, but high-level FP languages that compile to JVM bytecode (Kotlin, Scala, Clojure) are rising. + +Python has become highly popular, and is now also closer to Lisp than it was 20 years ago, especially after `MacroPy` introduced syntactic macros to Python (in 2013, [according to the git log](https://github.com/lihaoyi/macropy/commits/python2/macropy/__init__.py)). Python was not bad as a Lisp replacement even back in 2000 - see Peter Norvig's essay [Python for Lisp Programmers](https://norvig.com/python-lisp.html). Some more historical background, specifically on lexically scoped closures (and the initial lack thereof), can be found in [PEP 3104](https://www.python.org/dev/peps/pep-3104/), [PEP 227](https://www.python.org/dev/peps/pep-0227/), and [Historical problems with closures in JavaScript and Python](http://giocc.com/problems-with-closures-in-javascript-and-python.html). + +In 2020, does it still make sense to learn [the legendary](https://xkcd.com/297/) Common Lisp? + +To know exactly what it has to offer, **yes**. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. Having more perspectives at one's disposal makes one a better programmer. + +But as a practical tool? Is CL hands-down better than Python? Maybe no. Python has already delivered on 90% of the productivity promise of Lisp. Both languages cut down significantly on [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet). Python has a huge library ecosystem. [`mcpyrate`](https://github.com/Technologicat/mcpyrate) and `unpythonic` are trying to push the language-level features a further 5%. (A full 100% is likely impossible when extending an existing language; if nothing else, there will be seams.) + +As for productivity, [it may be](https://medium.com/smalltalk-talk/lisp-smalltalk-and-the-power-of-symmetry-8bd96aaa0c0c) that a form of code-data equivalence (symmetry!), not macros specifically, is what makes Lisp powerful. If so, there may be other ways to reach that equivalence. For example Smalltalk, like Lisp, *runs in the same context it's written in*. All Smalltalk data are programs. Smalltalk [may be making a comeback](https://hackernoon.com/how-to-evangelize-a-programming-language-0p7p3y02), in the form of [Pharo](https://pharo.org/). + +Haskell aims at code-data equivalence from a third angle (memoized pure functions are in essence infinite lookup tables), but I have not used it in practice, so I do not have the experience to say whether this is enough to make it feel powerful in a similar way. + +Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world is not that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead - without restarting the whole app at each change. Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. In web applications, [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a small step in a somewhat similar direction (as long as one can restart the server app easily, to make it use the latest definitions). diff --git a/doc/features.md b/doc/features.md index e032dfbf..09c44236 100644 --- a/doc/features.md +++ b/doc/features.md @@ -7,6 +7,7 @@ - [REPL server](repl.md) - [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) +- [Essays](essays.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/macros.md b/doc/macros.md index 1117b574..34922890 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -7,6 +7,7 @@ - [REPL server](repl.md) - [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) +- [Essays](essays.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/readings.md b/doc/readings.md index eb838c22..4b3b1a66 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -7,6 +7,7 @@ - [REPL server](repl.md) - [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) +- [Essays](essays.md) - **Additional reading** - [Contribution guidelines](../CONTRIBUTING.md) @@ -182,49 +183,13 @@ The common denominator is programming. Some relate to language design, some to c - [Richard P. Gabriel, Kent M. Pitman (2001): Technical Issues of Separation in Function Cells and Value Cells](https://dreamsongs.com/Separation.html) - A discussion of [Lisp-1 vs. Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2). -- [`hoon`: The C of Functional Programming](https://github.com/cgyarvin/urbit/blob/master/doc/book/0-intro.markdown#hoon) - - *The above link points to an old version from 2013; see below for a link to the latest version. I have given the old link here first, because it explains the philosophy differently from the latest documentation.* - - Some days I wonder if this `unpythonic` endeavor even makes any sense, and then I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. From the doc linked above: - - *So we could describe Hoon as a pure, strict, higher-order typed functional language. But don't do this in front of a Haskell purist, unless you put quotes around "typed," "functional," and possibly even "language." We could also say "object-oriented," with the same scare quotes for the cult of Eiffel.* - - While I am not sure if I will ever *use* `hoon`, it is hard not to like a language that puts quotes around "language". Few languages go that far in shaking up preconceptions. Critically examining what we believe, and why, often leads to useful insights. - - The claim that `hoon` is not a language, but a "language", fully makes sense after reading some of the documentation. `hoon` is essentially an *ab initio* language with an axiomatic approach to defining its operational semantics, similarly to how *Arc* approaches defining Lisp. Furthermore, `hoon` is the *functional equivalent of C* to the underlying virtual assembly language, `nock`. From a certain viewpoint, the "language" essentially consists of *glorified Nock macros*. Glorified assembly macros are pretty much all a *low-level* [HLL](https://en.wikipedia.org/wiki/High-level_programming_language) essentially is, so the claim seems about right. - - Nock is a peculiar assembly language. According to the comments in [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), it is a *Turing-complete non-lambda automaton*. The instruction set is permanently frozen, as if it was a physical CPU chip. Opcodes are just natural numbers, 0 through 11, and it is very minimalistic. For example, there is not even a decrement opcode. This is because from an axiomatic viewpoint, decrement can be defined recursively via increment. At which point, every systems programmer objects, rightfully, that no one sane actually does so. Indeed, the `hoon` standard library uses C FFI to take advantage of the physical processor's instruction set to perform arithmetic operations. Each piece of C code used for such acceleration purposes is termed a *jet*. - - Since - by the fact that the programmer called a particular standard library function - the system knows we want to compute a decrement (or a multiplication, a power, maybe some floating point operation, etc.), it can *accelerate* that particular operation by using the available hardware. - - The important point is, you *could* write out a `nock` macro that does the same thing, only it would be unbearably slow. In the axiomatic perspective - which is about proving programs correct - speed does not matter. At the same time, FFI gives speed for the real world. - - To summarize; as someone already put it, `hoon` offers a glimpse into an alternate universe of systems programming, where the functional camp won. It may also be a useful tool, or a source for further unconventional ideas - but to know for sure, I will have to read more about it. - - *NOTE: Using natural numbers for the opcodes at first glance sounds like a [Gödel numbering](https://en.wikipedia.org/wiki/G%C3%B6del_numbering) for the program space; but actually, the input to the VM contains some linked-list structure, which is not represented that way. Also, **any** programming language imposes its own Gödel numbering on the program space. Just take, for example, the UTF-8 representation of the source code text (which, in Python terms, is a `bytes` object), and interpret those bytes as one single bignum.* - - *Obviously, any interesting programs correspond to very large numbers, and are few and far between, so decoding random numbers via a Gödel numbering is not a practical way to generate interesting programs. [Genetic programming](https://en.wikipedia.org/wiki/Genetic_programming) works much better, because unlike Gödel numbering, it was actually designed specifically to do that. GP takes advantage of the semantic structure present in the source code (or AST) representation.* - - *The purpose of the original Gödel numbering was to prove Gödel's incompleteness theorem. In the case of `nock`, my impression is that the opcodes are natural numbers just for flavoring purposes. If you are building an ab initio software stack, what better way to announce that than to use natural numbers as your virtual machine's opcodes?* - - - From the language definition, [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon): - ``` - ++ doos :: sleep until - |= hap=path ^- (unit ,@da) - (doze:(wink:(vent bud (dink (dint hap))) now 0 (beck ~)) now [hap ~]) - :: - ``` - The Lisp family (particularly the Common Lisp branch) has a reputation for silly terminology, but `hoon` takes that a step further. - - However, I think I will adopt the verb *bunt*, meaning *to take the default value of*. That is such a common operation in programming that I find it hard to believe there is no standard abbreviation. - - Judging by the docs, `hoon` is definitely ha-ha-only-serious, but I am not sure of whether it is serious-serious. It does advertise itself as the functional-programming equivalent of C. See the comments to the entry on Manuel Simoni's blog - some people do think `hoon` is actually useful. - - So maybe there is a place for `unpythonic`, too. - - - The development of `urbit` has [moved to a new repository](https://github.com/urbit/urbit). - - The [latest `hoon` docs](https://urbit.org/docs/hoon/). - - Interestingly, `hoon` has uniform support for *wide* and *tall* modes; it does not use parentheses, but uses a single space (in characteristic `hoon` fashion, termed an *ace*) versus multiple spaces (respectively, a *gap*). "Multiple spaces" allows also newlines, like in LaTeX. - - `hoon` does not have syntactic macros. The reason given in the docs is the same as sometimes heard in the Python community - having a limited number of standard control structures, you always know what you're looking at. +- [`hoon`: The C of Functional Programming](https://urbit.org/docs/hoon/) + - Interesting take on an alternative computing universe where the functional camp won systems programming. These people have built [a whole operating system](https://github.com/urbit/urbit) on a Turing-complete non-lambda automaton, Nock. + - For my take, see [the opinion piece in Essays](essays.md#hoon-the-c-of-functional-programming). + - Judging by the docs, `hoon` is definitely ha-ha-only-serious, but I am not sure of whether it is serious-serious. It does advertise itself as the functional-programming equivalent of C. See the comments to [the entry on Manuel Simoni's blog](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) - some people do think `hoon` is actually useful. + - Technical points: + - `hoon` does not have syntactic macros. The reason given in the docs is the same as sometimes heard in the Python community - having a limited number of standard control structures, you always know what you are looking at. + - Interestingly, `hoon` has uniform support for *wide* and *tall* modes; it does not use parentheses, but uses a single space (in characteristic `hoon` fashion, termed an *ace*) versus multiple spaces (respectively, a *gap*). "Multiple spaces" allows also newlines, like in LaTeX. So [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) is not alone. # Python-related FP resources diff --git a/doc/repl.md b/doc/repl.md index 6c101be6..d253e928 100644 --- a/doc/repl.md +++ b/doc/repl.md @@ -7,6 +7,7 @@ - **REPL server** - [Troubleshooting](troubleshooting.md) - [Design notes](design-notes.md) +- [Essays](essays.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index 15112ad7..63910a6d 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -7,6 +7,7 @@ - [REPL server](repl.md) - **Troubleshooting** - [Design notes](design-notes.md) +- [Essays](essays.md) - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) From 2aa338186f6ba6a1e06d8193540f9501f7a004ec Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 04:22:51 +0300 Subject: [PATCH 453/832] improve flow --- doc/essays.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/doc/essays.md b/doc/essays.md index e2f78af0..ab84e32c 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -156,12 +156,18 @@ Python has become highly popular, and is now also closer to Lisp than it was 20 In 2020, does it still make sense to learn [the legendary](https://xkcd.com/297/) Common Lisp? -To know exactly what it has to offer, **yes**. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. Having more perspectives at one's disposal makes one a better programmer. - -But as a practical tool? Is CL hands-down better than Python? Maybe no. Python has already delivered on 90% of the productivity promise of Lisp. Both languages cut down significantly on [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet). Python has a huge library ecosystem. [`mcpyrate`](https://github.com/Technologicat/mcpyrate) and `unpythonic` are trying to push the language-level features a further 5%. (A full 100% is likely impossible when extending an existing language; if nothing else, there will be seams.) +As a practical tool? Is CL hands-down better than Python? Maybe no. Python has already delivered on 90% of the productivity promise of Lisp. Both languages cut down significantly on [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet). Python has a huge library ecosystem. [`mcpyrate`](https://github.com/Technologicat/mcpyrate) and `unpythonic` are trying to push the language-level features a further 5%. (A full 100% is likely impossible when extending an existing language; if nothing else, there will be seams.) As for productivity, [it may be](https://medium.com/smalltalk-talk/lisp-smalltalk-and-the-power-of-symmetry-8bd96aaa0c0c) that a form of code-data equivalence (symmetry!), not macros specifically, is what makes Lisp powerful. If so, there may be other ways to reach that equivalence. For example Smalltalk, like Lisp, *runs in the same context it's written in*. All Smalltalk data are programs. Smalltalk [may be making a comeback](https://hackernoon.com/how-to-evangelize-a-programming-language-0p7p3y02), in the form of [Pharo](https://pharo.org/). Haskell aims at code-data equivalence from a third angle (memoized pure functions are in essence infinite lookup tables), but I have not used it in practice, so I do not have the experience to say whether this is enough to make it feel powerful in a similar way. Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world is not that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead - without restarting the whole app at each change. Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. In web applications, [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a small step in a somewhat similar direction (as long as one can restart the server app easily, to make it use the latest definitions). + +But to know exactly what Common Lisp has to offer, **yes**, it does make sense to learn it. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. + +Having more perspectives at one's disposal makes one a better programmer - and that is what ultimately counts. As [Alan Perlis said in 1982](https://en.wikiquote.org/wiki/Alan_Perlis): + +*A language that doesn't affect the way you think about programming, is not worth knowing.* + +In this sense, Common Lisp is very much worth knowing. Although, if you want a beautiful, advanced Lisp, maybe go for [Racket](https://racket-lang.org/) first; but that is an essay for another day. From 2eb635c4dfe8ab5f510f2c66441063ffbc043e4b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 9 Jun 2021 04:25:09 +0300 Subject: [PATCH 454/832] wording --- doc/essays.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/essays.md b/doc/essays.md index ab84e32c..7bcb263d 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -58,7 +58,7 @@ For general programming in the early 2020s, Python still has the ecosystem advan # `hoon`: The C of Functional Programming -Some days I wonder if this `unpythonic` endeavor even makes any sense. Then, turning the pages of the [book of sand](https://en.wikipedia.org/wiki/The_Book_of_Sand) that is the web, I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. +Some days I wonder if this whole `unpythonic` endeavor even makes any sense. Then, turning the pages of the [book of sand](https://en.wikipedia.org/wiki/The_Book_of_Sand) that is the web, I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. Its philosophy is best described by this gem from an [early version of its documentation](https://github.com/cgyarvin/urbit/blob/master/doc/book/0-intro.markdown#hoon): From 58d895519e6306645422b860a3682e3b987b17f0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:49:19 +0300 Subject: [PATCH 455/832] wording --- doc/dialects/lispython.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 161ef5f3..f94f2a5c 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -87,7 +87,7 @@ In terms of ``unpythonic.syntax``, we implicitly enable ``autoreturn``, ``tco``, - Note the analysis is performed at compile time, whence it does **not** care about the short-circuit behavior that occurs at run time. - The last item of a `do[]`. - The last item of an implicit `do[]` in a `let[]` where the body uses the extra bracket syntax. (All `let` constructs provided by `unpythonic.syntax` are supported.) - - For the gritty details, see the source code of `unpythonic.syntax.tailtools._transform_retexpr`. + - For the gritty details, see the syntax transformer `_transform_retexpr` in [`unpythonic.syntax.tailtools`](../../unpythonic/syntax/tailtools.py). - Multiple-expression lambdas, using bracket syntax, for example ``lambda x: [expr0, ...]``. - Brackets denote a multiple-expression lambda body. Technically, the brackets create a `do[]` environment. - If you want your lambda to have one expression that is a literal list, double the brackets: `lambda x: [[5 * x]]`. From c78c533dfcd0e7ec18de27dc384f137da9a003b9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:49:31 +0300 Subject: [PATCH 456/832] update comment on Borgesian flavor of `hoon` --- doc/essays.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/essays.md b/doc/essays.md index 7bcb263d..3c1ead0c 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -58,7 +58,7 @@ For general programming in the early 2020s, Python still has the ecosystem advan # `hoon`: The C of Functional Programming -Some days I wonder if this whole `unpythonic` endeavor even makes any sense. Then, turning the pages of the [book of sand](https://en.wikipedia.org/wiki/The_Book_of_Sand) that is the web, I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. +Some days I wonder if this whole `unpythonic` endeavor even makes any sense. Then, turning the pages of [the book of sand](https://en.wikipedia.org/wiki/The_Book_of_Sand) that is the web, I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. Its philosophy is best described by this gem from an [early version of its documentation](https://github.com/cgyarvin/urbit/blob/master/doc/book/0-intro.markdown#hoon): @@ -74,7 +74,7 @@ Since - by the fact that the programmer called a particular standard library fun The important point is, you *could* write out a `nock` macro that does the same thing, only it would be unbearably slow. In the axiomatic perspective - which is about proving programs correct - speed does not matter. At the same time, FFI gives speed for the real world. -To summarize; as someone already put it, `hoon` offers a glimpse into an alternate universe of systems programming, where the functional camp won. It may also be a useful tool, or a source for further unconventional ideas - but to know for sure, I will have to read more about it. +To summarize; as someone already put it, `hoon` offers a glimpse into an alternative universe of systems programming, where the functional camp won. It may also be a useful tool, or a source for further unconventional ideas - but to know for sure, I will have to read more about it. I think the perfect place to end this piece is to quote a few lines from the language definition [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), to give a flavor: @@ -125,9 +125,9 @@ The Lisp family (particularly the Common Lisp branch) has a reputation for silly I actually like the phonetic base, making numbers sound like [*sorreg-namtyv*](https://urbit.org/docs/hoon/hoon-school/nouns/); that is 5 702 400 for the rest of us. And I think I will, quite seriously, adopt the verb *bunt*, meaning *to take the default value of*. That is such a common operation in programming that I find it hard to believe there is no standard abbreviation. I wonder what other discoveries await. -Finally, in some way I cannot quite put a finger on, to me the style has echoes of [Jorge Luis Borges](https://en.wikipedia.org/wiki/Jorge_Luis_Borges). I can imagine `hoon` as the *official* programming language of *[Tlön](https://en.wikipedia.org/wiki/Tl%C3%B6n%2C_Uqbar%2C_Orbis_Tertius)*. +Finally, in some way I cannot quite put a finger on, to me the style has echoes of [Jorge Luis Borges](https://en.wikipedia.org/wiki/Jorge_Luis_Borges). Maybe it is that the `hoon` source code sounds like something out of [The Library of Babel](https://en.wikipedia.org/wiki/The_Library_of_Babel). The Borgesian flavor seems intentional, too; the company building the Urbit stack, which `hoon` is part of, is itself named *[Tlon](https://en.wikipedia.org/wiki/Tl%C3%B6n%2C_Uqbar%2C_Orbis_Tertius)*. Remaking the world by re-imagining it, indeed. -So maybe there is a place for `unpythonic`, too. +Maybe there is a place for `unpythonic`, too. **Links** From 155896a951959dcfe80da74a6618da1c9145f19d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:51:04 +0300 Subject: [PATCH 457/832] update namedlambda and continuations docs --- doc/macros.md | 72 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 27 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 34922890..5fd715e7 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -652,6 +652,8 @@ In the second example, returning ``x`` separately is redundant, because the assi ### ``namedlambda``: auto-name your lambdas +**Changed in v0.15.0.** *When `namedlambda` encounters a lambda definition it cannot infer a name for, it instead injects source location info into the name, provided that the AST node for that particular `lambda` has a line number for it. The result looks like ``.* + Who said lambdas have to be anonymous? ```python @@ -695,41 +697,49 @@ The naming is performed using the function ``unpythonic.misc.namelambda``, which - **Added in v0.15.0**: Named expressions (a.k.a. walrus operator, Python 3.8+), ``f := lambda ...: ...`` - Expression-assignment to an unpythonic environment, ``f << (lambda ...: ...)`` - - Env-assignments are processed lexically, just like regular assignments. + - Env-assignments are processed lexically, just like regular assignments. This should not cause problems, because left-shifting by a literal lambda most often makes no sense (whence, that syntax is *almost* guaranteed to mean an env-assignment). - - Let bindings, ``let[[f << (lambda ...: ...)] in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). + - Let bindings, ``let[[f << (lambda ...: ...)] in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). - **Added in v0.14.2**: Named argument in a function call, as in ``foo(f=lambda ...: ...)``. - **Added in v0.14.2**: In a dictionary literal ``{...}``, an item with a literal string key, as in ``{"f": lambda ...: ...}``. -Support for other forms of assignment may or may not be added in a future version. +Support for other forms of assignment may or may not be added in a future version. We will maintain a list here; but if you want the gritty details, see the `_namedlambda` syntax transformer in [`unpythonic.syntax.lambdatools`](../unpythonic/syntax/lambdatools.py). -### ``fn``: underscore notation (quick lambdas) for Python. +### ``fn``: underscore notation (quick lambdas) for Python -**Changed in v0.15.0.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves.* +**Changed in v0.15.0.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves. Note that the name of the construct is now `fn[]`.* -*The name is now `fn[]`. This was changed because `f` is often used as a function name in code examples, local temporaries, and similar. Also, `fn[]` is a less ambiguous abbreviation for a syntactic construct that means *function*, while remaining shorter than the equivalent `lambda`.* +The syntax ``fn[...]`` creates a lambda, where each underscore `_` in the ``...`` part introduces a new parameter: -*The underscore `_` is no longer a macro on its own. The `fn` macro treats the underscore magically, as `f` did before, but anywhere else the underscore is available to be used as a regular variable. If you use `fn[]`, change your import of this macro to `from unpythonic.syntax import macros, fn`.** +```python +from unpythonic.syntax import macros, fn +from unpythonic.syntax import _ # optional, makes IDEs happy -*The underscore does **not** need to be imported for `fn[]` to recognize it. But if you want to make your IDE happy, there is a symbol named `_` in `unpythonic.syntax` you can import to silence any "undefined name" errors regarding the use of `_`. It is a regular run-time object, not a macro.* +double = fn[_ * 2] # --> double = lambda x: x * 2 +mul = fn[_ * _] # --> mul = lambda x, y: x * y +``` -The syntax ``fn[...]`` creates a lambda, where each underscore in the ``...`` part introduces a new parameter. The macro does not descend into any nested ``fn[]``. +The macro does not descend into any nested ``fn[]``, to allow the macro expander itself to expand those separately. -Example: +We have named the construct `fn`, because `f` is often used as a function name in code examples, local temporaries, and similar. Also, `fn[]` is a less ambiguous abbreviation for a syntactic construct that means *function*, while remaining shorter than the equivalent `lambda`. -```python -func = fn[_ * _] # --> func = lambda x, y: x * y -``` +The underscore `_` itself is not a macro. The `fn` macro treats the underscore magically, just like MacroPy's `f`, but anywhere else the underscore is available to be used as a regular variable. -Since in `mcpyrate`, macros can be as-imported, you can rename `fn` at import time to have any name you want. The `quicklambda` block macro (see below) respects the as-import. Now you **must** import also the macro `fn` when you import the macro `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `fn` is currently bound to. +The underscore does not need to be imported for `fn[]` to recognize it, but if you want to make your IDE happy, there is a symbol named `_` in `unpythonic.syntax` you can import to silence any "undefined name" errors regarding the use of `_`. It is a regular run-time object, not a macro. + +(It *could* be made into a `@namemacro` that triggers a syntax error when it appears in an improper context, like starting with v0.15.0, many auxiliary constructs in similar roles already do. But it was decided that in this particular case, it is more valuable to have the name `_` available for other uses in other contexts, because it is a standard dummy name in Python. The lambdas created using `fn[]` are likely short enough that not automatically detecting misplaced underscores does not cause problems in practice.) + +Because in `mcpyrate`, macros can be as-imported, you can rename `fn` at import time to have any name you want. The `quicklambda` block macro (see below) respects the as-import. You **must** import also the macro `fn` if you use `quicklambda`, because `quicklambda` internally queries the expander to determine the name(s) the macro `fn` is currently bound to. If the `fn` macro is not bound to any name, `quicklambda` will do nothing. + +It is sufficient that `fn` has been macro-imported by the time when the `with quicklambda` expands. So it is possible, for example, for a dialect template to macro-import just `quicklambda` and inject an invocation for it, and leave macro-importing `fn` to the user code. The `Lispy` variant of the [Lispython dialect](dialects/lispython.md) does exactly this. ### ``quicklambda``: expand quick lambdas first To be able to transform correctly, the block macros in ``unpythonic.syntax`` that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all ``lambda`` definitions written with Python's standard ``lambda``. -However, the ``fn`` macro uses the syntax ``fn[...]``, which (to the analyzer) does not look like a lambda definition. This macro changes the expansion order, forcing any ``fn[...]`` lexically inside the block to expand before any other macros do. +However, the ``fn`` macro uses the syntax ``fn[...]``, which (to the analyzer) does not look like a lambda definition. The `quicklambda` block macro changes the expansion order, forcing any ``fn[...]`` lexically inside the block to expand before any other macros do. Any expression of the form ``fn[...]``, where ``fn`` is any name bound in the current macro expander to the macro `unpythonic.syntax.fn`, is understood as a quick lambda. (In plain English, this respects as-imports of the macro ``fn``.) @@ -1082,30 +1092,38 @@ See the docstring of ``unpythonic.syntax.tco`` for details. We provide **genuine multi-shot continuations for Python**. Compare generators and coroutines, which are resumable functions, or in other words, single-shot continuations. In single-shot continuations, once execution passes a certain point, it cannot be rewound. Multi-shot continuations [can be emulated](https://gist.github.com/yelouafi/858095244b62c36ec7ebb84d5f3e5b02), but this makes the execution time `O(n**2)`, because when we want to restart again at an already passed point, the execution must start from the beginning, replaying the history. In contrast, **we implement continuations that can natively resume execution multiple times from the same point.** -This feature has some limitations and is mainly intended for teaching continuations in a Python setting. +This feature has some limitations and is mainly intended for experimenting with, and teaching, multi-shot continuations in a Python setting. -- Especially, there are seams between continuation-enabled code and regular Python code. (This happens with any feature that changes the semantics of only a part of a program.) +- There are seams between continuation-enabled code and regular Python code. (This happens with any feature that changes the semantics of only a part of a program.) -- There's no [`dynamic-wind`](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28quote._~23~25kernel%29._dynamic-wind%29%29) (the generalization of `try/finally`, when control can jump back in to the block from outside it). +- There is no [`dynamic-wind`](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28quote._~23~25kernel%29._dynamic-wind%29%29) (the generalization of `try/finally`, when control can jump back in to the block from outside it). -- Interaction of continuations with exceptions isn't fully thought out. Interaction with async functions **is currently not even implemented**. This is quite simply because this feature is primarily for teaching, and the implementation is already quite complex. +- Interaction of continuations with exceptions is not fully thought out. -- The implicit `cc` parameter might not be a good idea in the long run, and it might or might not change in a future release. It suffers from the same lack of transparency as the implicit `this` in many languages (e.g. C++ and JavaScript). - - Because it's implicit, it's easy to forget that each function definition implicitly introduces its own `cc`. - - This introduces a bug when one introduces an inner function, and attempts to use the outer `cc` inside the inner function body, forgetting that inside the inner function the name `cc` points to **the inner function's** own `cc`. +- Interaction with async functions **is not even implemented**. An `async def` or `await` appearing inside a `with continuations` block is considered a syntax error. + +- The implicit `cc` parameter might not be a good idea in the long run. + - This design might or might not change in a future release. It suffers from the same lack of transparency, whence the same potential for bugs, as the implicit `this` in many languages (e.g. C++ and JavaScript). + - Because `cc` is *declared* implicitly, it is easy to forget that *every* function definition anywhere inside the `with continuations` block introduces its own `cc` parameter. + - This introduces a bug when one introduces an inner function, and attempts to use the outer `cc` inside the inner function body, forgetting that inside the inner function, the name `cc` points to **the inner function's** own `cc`. + - The correct pattern is to `outercc = cc` in the outer function, and then use `outercc` inside the inner function body. - Not introducing its own `this` [was precisely why](http://tc39wiki.calculist.org/es6/arrow-functions/) the arrow function syntax was introduced to JavaScript in ES6. - - Python gets `self` right in that while it's conveniently *passed* implicitly, it must be *declared* explicitly, eliminating the transparency issue. - - On the other hand, a semi-explicit `cc`, like Python's `self`, was tried in a previous release, and it led to a lot of boilerplate. It's especially bad that it effectively needs to be a keyword parameter, necessitating the user to write `def f(x, *, cc)`. + - Python gets `self` right in that while it is conveniently *passed* implicitly, it must be *declared* explicitly, eliminating the transparency issue. + - On the other hand, a semi-explicit `cc`, like Python's `self`, was tried in an early version of this continuations subsystem, and it led to a lot of boilerplate. It is especially bad that to avoid easily avoidable bugs regarding passing in the wrong arguments, `cc` effectively must be a keyword parameter, necessitating the user to write `def f(x, *, cc)`. Not having to type out the `, *, cc` is much nicer, albeit not as pythonic. #### General remarks on continuations -If you're new to continuations, see the [short and easy Python-based explanation](https://www.ps.uni-saarland.de/~duchier/python/continuations.html) of the basic idea. +If you are new to continuations, see the [short and easy Python-based explanation](https://www.ps.uni-saarland.de/~duchier/python/continuations.html) of the basic idea. -We provide a very loose pythonification of Paul Graham's continuation-passing macros, chapter 20 in [On Lisp](http://paulgraham.com/onlisp.html). +We essentially provide a very loose pythonification of Paul Graham's continuation-passing macros, chapter 20 in [On Lisp](http://paulgraham.com/onlisp.html). The approach differs from native continuation support (such as in Scheme or Racket) in that the continuation is captured only where explicitly requested with ``call_cc[]``. This lets most of the code work as usual, while performing the continuation magic where explicitly desired. -As a consequence of the approach, our continuations are [*delimited*](https://en.wikipedia.org/wiki/Delimited_continuation) in the very crude sense that the captured continuation ends at the end of the body where the *currently dynamically outermost* ``call_cc[]`` was used (and it returns a value). Hence, if porting some code that uses ``call/cc`` from Racket to Python, in the Python version the ``call_cc[]`` may be need to be placed further out to capture the relevant part of the computation. For example, see ``amb`` in the demonstration below; a Scheme or Racket equivalent usually has the ``call/cc`` placed inside the ``amb`` operator itself, whereas in Python we must place the ``call_cc[]`` at the call site of ``amb``. +As a consequence of the approach, our continuations are [*delimited*](https://en.wikipedia.org/wiki/Delimited_continuation) in the very crude sense that the captured continuation ends at the end of the body where the *currently dynamically outermost* ``call_cc[]`` was used. Notably, in `unpythonic`, a continuation eventually terminates and returns a value, without hijacking the rest of the whole-program execution. + +Hence, if porting some code that uses ``call/cc`` from Racket to Python, in the Python version the ``call_cc[]`` may be need to be placed further out to capture the relevant part of the computation. For example, see ``amb`` in the demonstration below; a Scheme or Racket equivalent usually has the ``call/cc`` placed inside the ``amb`` operator itself, whereas in Python we must place the ``call_cc[]`` at the call site of ``amb``. + +Observe that while our outermost `call_cc` already somewhat acts like a prompt (in the sense of delimited continuations), we are currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and terminate the capture there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. For various possible program topologies that continuations may introduce, see [these clarifying pictures](callcc_topology.pdf). From cee083d7aecea5f6cf7c852871f3c95f18e3a61b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:51:21 +0300 Subject: [PATCH 458/832] update readings; add Seibel 2005 --- doc/readings.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/readings.md b/doc/readings.md index 4b3b1a66..7f697d54 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -171,14 +171,16 @@ The common denominator is programming. Some relate to language design, some to c - [Pascal Costanza's Highly Opinionated Guide to Lisp (2013)](http://www.p-cos.net/lisp/guide.html) +- [Peter Seibel (2005): Practical Common Lisp](https://gigamonkeys.com/book/) + - This book is an excellent introduction that walks through Common Lisp, including some advanced features. It is also useful for non-lispers to take home interesting ideas from CL. + - R. Kent Dybvig, Simon Peyton Jones, Amr Sabry (2007). A Monadic Framework for Delimited Continuations. Journal of functional programming, 17(6), 687-730. Preprint [here](https://legacy.cs.indiana.edu/~dyb/pubs/monadicDC.pdf). - Particularly approachable explanation of delimited continuations. - - Could try building that for `unpythonic` in a future version. While our outermost `call_cc` already somewhat acts like a prompt, we're currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and terminate the capture there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. + - Could try building that for `unpythonic` in a future version. - [Wat: Concurrency and Metaprogramming for JS](https://github.com/manuel/wat-js) - [pywat: Interpreter of the Wat language written in Python](https://github.com/piokuc/pywat) - [Example of Wat in Manuel Simoni's blog (2013)](http://axisofeval.blogspot.com/2013/05/green-threads-in-browser-in-20-lines-of.html) - - This suggests building proper delimited continuations shouldn't be that hard in Python. - [Richard P. Gabriel, Kent M. Pitman (2001): Technical Issues of Separation in Function Cells and Value Cells](https://dreamsongs.com/Separation.html) - A discussion of [Lisp-1 vs. Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2). From b94cc56a5fa3172f274ad83348462a37198d3c07 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:51:41 +0300 Subject: [PATCH 459/832] wording --- doc/troubleshooting.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index 63910a6d..4dc742b2 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -34,7 +34,7 @@ On the other hand, `unpythonic` is a kitchen-sink language extension, and half o If you intend to **use** `unpythonic.syntax` or `unpythonic.dialects`, or if you intend to **develop** `unpythonic` (specifically: to be able to run its test suite), then you will need a macro expander. -As of v0.15.0, specifically you'll need [`mcpyrate`](https://github.com/Technologicat/mcpyrate). +As of v0.15.0, specifically you will need [`mcpyrate`](https://github.com/Technologicat/mcpyrate). ### Why `mcpyrate` and not MacroPy? @@ -46,7 +46,7 @@ Beside the advanced features, the reason we use `mcpyrate` is that the `unpython ### Cannot import the name `macros`? -In `mcpyrate`-based programs, there is no run-time object named `macros`, so failing to import that usually means that, for some reason, the macro expander was not active. +In `mcpyrate`-based programs, there is no run-time object named `macros`, so failing to import that usually means that, for some reason, the macro expander is not enabled. Macro-enabled, `mcpyrate`-based programs expect to be run with `macropython` (included in the [`mcpyrate` PyPI package](https://pypi.org/project/mcpyrate/)) instead of bare `python3`. @@ -70,13 +70,13 @@ This will force a recompile of the `.py` files the next time they are loaded. Th ### I'm hacking a macro inside a module in `unpythonic.syntax`, and my changes don't take? -This is also likely due to a stale bytecode cache. As of `mcpyrate` 3.4.0, macro re-exports, used by `unpythonic.syntax.__init__`, may confuse the macro-dependency analyzer that determines bytecode cache validity. +This is also likely due to a stale bytecode cache. As of `mcpyrate` 3.4.0, macro re-exports, used by `unpythonic.syntax.__init__`, are not seen by the macro-dependency analyzer that determines bytecode cache validity. -The thing to realize here is that as per macropythonic tradition, in `mcpyrate`, a function being a macro is a property of its **use site**, not of its definition site. So how do we re-export a macro? We simply re-export the macro function, like we would do for any other function. +The important point to realize here is that as per macropythonic tradition, in `mcpyrate`, a function being a macro is a property of its **use site**, not of its definition site. So how do we re-export a macro? We simply re-export the macro function, like we would do for any other function. -Importantly, the import to make that re-export happen does not look like a macro-import. This is the right way to do it, since we want to make the object (macro function) available for clients to import, **not** establish bindings in the macro expander *for compiling the module `unpythonic.syntax.__init__` itself*. (The latter is what a macro-import does - it establishes macro bindings *for the module it lexically appears in*.) +The import to make that re-export happen does not look like a macro-import. This is the right way to do it, since we want to make the object (macro function) available for clients to import, **not** establish bindings in the macro expander *for compiling the module `unpythonic.syntax.__init__` itself*. (The latter is what a macro-import does - it establishes macro bindings *for the module it lexically appears in*.) -The problem is, the macro-dependency analyzer only looks at the macro-import dependency graph, not the full dependency graph, so when analyzing the user program (e.g. a unit test module in `unpythonic.syntax.tests`), it doesn't notice that the macro definition has changed. +The problem is, the macro-dependency analyzer only looks at the macro-import dependency graph, not the full dependency graph, so when analyzing the user program (e.g. a unit test module in `unpythonic.syntax.tests`), it does not scan the re-export that points to the changed macro definition. I might modify the `mcpyrate` analyzer in the future, but doing so will make the dependency scan a lot slower than it needs to be in most circumstances, because a large majority of imports in Python have nothing to do with macros. From 3b3499163bf22ca9210d296a9c13ce74f08de424 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:51:45 +0300 Subject: [PATCH 460/832] oops, export Lispy, too --- unpythonic/dialects/lispython.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index b6a893b6..13dbc869 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -4,7 +4,7 @@ Powered by `mcpyrate` and `unpythonic`. """ -__all__ = ["Lispython"] +__all__ = ["Lispython", "Lispy"] __version__ = '2.0.0' From b67e9e3751ba973541fcf999bdf639d542b12f41 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:51:59 +0300 Subject: [PATCH 461/832] be more precise in comment --- unpythonic/dialects/tests/test_lispy.py | 4 ++-- unpythonic/dialects/tests/test_lispython.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/unpythonic/dialects/tests/test_lispy.py b/unpythonic/dialects/tests/test_lispy.py index eb58da92..8c5f8aed 100644 --- a/unpythonic/dialects/tests/test_lispy.py +++ b/unpythonic/dialects/tests/test_lispy.py @@ -69,8 +69,8 @@ def f(k, acc): test[a(3) == "odd"] # MacroPy #21; namedlambda must be in its own with block in the - # dialect implementation or this particular combination will fail - # (uncaught jump, __name__ not set). + # dialect implementation or the particular combination of macros + # invoked by Lispy will fail (uncaught jump, __name__ not set). # # With `mcpyrate` this shouldn't matter, but we're keeping the example. with testset("autonamed letrec lambdas, multiple-expression let body"): diff --git a/unpythonic/dialects/tests/test_lispython.py b/unpythonic/dialects/tests/test_lispython.py index e41913ec..4085b07f 100644 --- a/unpythonic/dialects/tests/test_lispython.py +++ b/unpythonic/dialects/tests/test_lispython.py @@ -100,8 +100,8 @@ def f(k, acc): test[a(3) == "odd"] # MacroPy #21; namedlambda must be in its own with block in the - # dialect implementation or this particular combination will fail - # (uncaught jump, __name__ not set). + # dialect implementation or the particular combination of macros + # invoked by Lispython will fail (uncaught jump, __name__ not set). # # With `mcpyrate` this shouldn't matter, but we're keeping the example. with testset("autonamed letrec lambdas, multiple-expression let body"): From eed0350e9b29d2579b3259090460a41ec03ac329 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:52:10 +0300 Subject: [PATCH 462/832] correct version number in comment --- unpythonic/syntax/lambdatools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 5127852e..1b089392 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -341,7 +341,7 @@ def transform(self, tree): else: tree.value = self.visit(tree.value) return tree - elif type(tree) is NamedExpr: # f := lambda ...: ... (Python 3.8+, added in unpythonic 0.15) + elif type(tree) is NamedExpr: # f := lambda ...: ... (Python 3.8+, added in unpythonic 0.15.0) tree.value, thelambda, match = nameit(getname(tree.target), tree.value) if match: thelambda.body = self.visit(thelambda.body) From 0a8006cbfa76c05f686c77fe51a9465375e28b68 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 10:52:21 +0300 Subject: [PATCH 463/832] refactor --- unpythonic/syntax/letdoutil.py | 56 +++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 21 deletions(-) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 8d630b30..ee1f6207 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -37,13 +37,36 @@ def _canonize_macroargs_node(macroargs): return macroargs.elts return [macroargs] # anything that doesn't have at least one comma at the top level -def canonize_bindings(elts, letsyntax_mode=False): # public as of v0.14.3+ - """Wrap a single binding without container into a length-1 `list`. +# For analysis of let-bindings and env-assignments. +def _isname(tree): + """Return whether `tree` is a lexical name. + + The actual `ast.Name` may be wrapped in a `mcpyrate.core.Done`, which is produced + by expanded `@namemacro`s; we accept a `Done` containing an `ast.Name`, too. + + We don't accept hygienic captures, since those correspond to values, not names. + """ + return type(tree) is Name or (isinstance(tree, Done) and _isname(tree.body)) +def _isbindingtarget(tree, letsyntax_mode): + """Return whether `tree` is a valid target for a let-binding or env-assignment. + + letsyntax_mode: used by let_syntax to allow template definitions. + This allows, beside a bare name `k`, the formats `k(a0, ...)` and `k[a0, ...]` + to appear in the variable-name position. + """ + return (_isname(tree) or + (letsyntax_mode and ((type(tree) is Call and _isname(tree.func)) or + (type(tree) is Subscript and _isname(tree.value))))) - Pass through multiple bindings as-is. +def canonize_bindings(elts, letsyntax_mode=False): # public as of v0.14.3+ + """Convert any `let` bindings format supported by `unpythonic` into a canonical format. Yell if the input format is invalid. + The canonical format is a `list` of `ast.Tuple`:: + + [Tuple(elts=[k0, v0]), ...] + elts: `list` of bindings, one of:: [(k0, v0), ...] # multiple bindings contained in a tuple [(k, v),] # single binding contained in a tuple also ok @@ -59,22 +82,10 @@ def canonize_bindings(elts, letsyntax_mode=False): # public as of v0.14.3+ This allows, beside a bare name `k`, the formats `k(a0, ...)` and `k[a0, ...]` to appear in the variable-name position. """ - def isname(tree): - # Note we don't accept hygienic captures. - # The `Done` may be produced by expanded `@namemacro`s. - return type(tree) is Name or (isinstance(tree, Done) and isname(tree.body)) - def isbindingtarget(tree): - return (isname(tree) or - (letsyntax_mode and ((type(tree) is Call and isname(tree.func)) or - (type(tree) is Subscript and isname(tree.value))))) def iskvpairbinding(lst): - return len(lst) == 2 and isbindingtarget(lst[0]) - def isenvassignbinding(tree): - if not (type(tree) is BinOp and type(tree.op) is LShift): - return False - return isbindingtarget(tree.left) + return len(lst) == 2 and _isbindingtarget(lst[0], letsyntax_mode) - if len(elts) == 1 and isenvassignbinding(elts[0]): # [k << v] + if len(elts) == 1 and isenvassign(elts[0], letsyntax_mode): # [k << v] return [Tuple(elts=[elts[0].left, elts[0].right])] if len(elts) == 2 and iskvpairbinding(elts): # [k, v] return [Tuple(elts=elts)] # TODO: `mcpyrate`: just `q[t[elts]]`? @@ -82,20 +93,23 @@ def isenvassignbinding(tree): return elts if all((type(b) is List and iskvpairbinding(b.elts)) for b in elts): # [[k0, v0], ...] return [Tuple(elts=b.elts) for b in elts] - if all((isenvassign(b) and isbindingtarget(b.left)) for b in elts): # [k0 << v0, ...] + if all(isenvassign(b, letsyntax_mode) for b in elts): # [k0 << v0, ...] return [Tuple(elts=[b.left, b.right]) for b in elts] raise SyntaxError("expected bindings to be `(k0, v0), ...`, `[k0, v0], ...`, or `k0 << v0, ...`, or a single `k, v`, or `k << v`") # pragma: no cover -def isenvassign(tree): +def isenvassign(tree, letsyntax_mode=False): """Detect whether tree is an unpythonic ``env`` assignment, ``name << value``. The only way this differs from a general left-shift is that the LHS must be an ``ast.Name``. + + letsyntax_mode: used by let_syntax to allow template definitions. + This allows, beside a bare name `k`, the formats `k(a0, ...)` and `k[a0, ...]` + to appear in the variable-name position. """ if not (type(tree) is BinOp and type(tree.op) is LShift): return False - # The `Done` may be produced by expanded `@namemacro`s. - return type(tree.left) is Name or (isinstance(tree.left, Done) and type(tree.body) is Name) + return _isbindingtarget(tree.left, letsyntax_mode) # TODO: This would benefit from macro destructuring in the expander. # TODO: See https://github.com/Technologicat/mcpyrate/issues/3 From 9bbe3d37f17ce53e38f4a487df334872118d6d1e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 10 Jun 2021 11:02:43 +0300 Subject: [PATCH 464/832] add another relevant LtU discussion --- doc/readings.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/readings.md b/doc/readings.md index 7f697d54..3c0e9d7d 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -193,6 +193,10 @@ The common denominator is programming. Some relate to language design, some to c - `hoon` does not have syntactic macros. The reason given in the docs is the same as sometimes heard in the Python community - having a limited number of standard control structures, you always know what you are looking at. - Interestingly, `hoon` has uniform support for *wide* and *tall* modes; it does not use parentheses, but uses a single space (in characteristic `hoon` fashion, termed an *ace*) versus multiple spaces (respectively, a *gap*). "Multiple spaces" allows also newlines, like in LaTeX. So [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) is not alone. +- [LtU: Why is there no widely accepted progress for 50 years?](http://lambda-the-ultimate.org/node/5590) + - Discussion on how programming languages *have* improved. + - Contains interesting viewpoints, such as dmbarbour's suggestion that much of modern hardware is essentially "compiled" from a hardware description language such as VHDL. + # Python-related FP resources From 7553d6b08e6863c714497161f679197bcf47ba35 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 00:27:59 +0300 Subject: [PATCH 465/832] Use star-re-exports for dialects. Dialects are just regular public API names. --- unpythonic/dialects/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/dialects/__init__.py b/unpythonic/dialects/__init__.py index 1326d62d..644a5cee 100644 --- a/unpythonic/dialects/__init__.py +++ b/unpythonic/dialects/__init__.py @@ -12,6 +12,6 @@ """ # re-exports -from .lispython import Lispython, Lispy # noqa: F401 -from .listhell import Listhell # noqa: F401 -from .pytkell import Pytkell # noqa: F401 +from .lispython import * # noqa: F401, F403 +from .listhell import * # noqa: F401, F403 +from .pytkell import * # noqa: F401, F403 From 2410f2995133c5de20012e2a55c36320e1bd3faa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 00:28:57 +0300 Subject: [PATCH 466/832] reorganize essays --- doc/essays.md | 64 +++++++++++++++++++++++++++++---------------------- 1 file changed, 36 insertions(+), 28 deletions(-) diff --git a/doc/essays.md b/doc/essays.md index 3c1ead0c..f865cf14 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -11,18 +11,22 @@ - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) +For now, essays are listed in chronological order, most recent last. + **Table of Contents** - [What Belongs in Python?](#what-belongs-in-python) -- [`hoon`: The C of Functional Programming](#hoon-the-c-of-functional-programming) - [Common Lisp, Python, and productivity](#common-lisp-python-and-productivity) +- [`hoon`: The C of Functional Programming](#hoon-the-c-of-functional-programming) # What Belongs in Python? +*Originally written in 2020; updated 9 June 2021.* + You may feel that [my hovercraft is full of eels](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html). It is because they come with the territory. Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). @@ -56,8 +60,39 @@ What I can say is, `unpythonic` is not meant for the average Python project, eit For general programming in the early 2020s, Python still has the ecosystem advantage, so it does not make sense to move to anything else, at least yet. So, let us empower what we have. Even if we have to build something that could be considered *unpythonic*. +# Common Lisp, Python, and productivity + +*Originally written in 2020; updated 9 June 2021.* + +The various essays Paul Graham wrote near the turn of the millennium, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for the Lisp family of languages. So how does the programming world look in that light now, 20 years later? + +The base abstraction level of programming languages, even those in popular use, has increased. The trend was visible already then, and was indeed noted in the essays. The focus on low-level languages such as C++ has decreased. Java is still popular, but high-level FP languages that compile to JVM bytecode (Kotlin, Scala, Clojure) are rising. + +Python has become highly popular, and is now also closer to Lisp than it was 20 years ago, especially after `MacroPy` introduced syntactic macros to Python (in 2013, [according to the git log](https://github.com/lihaoyi/macropy/commits/python2/macropy/__init__.py)). Python was not bad as a Lisp replacement even back in 2000 - see Peter Norvig's essay [Python for Lisp Programmers](https://norvig.com/python-lisp.html). Some more historical background, specifically on lexically scoped closures (and the initial lack thereof), can be found in [PEP 3104](https://www.python.org/dev/peps/pep-3104/), [PEP 227](https://www.python.org/dev/peps/pep-0227/), and [Historical problems with closures in JavaScript and Python](http://giocc.com/problems-with-closures-in-javascript-and-python.html). + +In 2020, does it still make sense to learn [the legendary](https://xkcd.com/297/) Common Lisp? + +As a practical tool? Is CL hands-down better than Python? Maybe no. Python has already delivered on 90% of the productivity promise of Lisp. Both languages cut down significantly on [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet). Python has a huge library ecosystem. [`mcpyrate`](https://github.com/Technologicat/mcpyrate) and `unpythonic` are trying to push the language-level features a further 5%. (A full 100% is likely impossible when extending an existing language; if nothing else, there will be seams.) + +As for productivity, [it may be](https://medium.com/smalltalk-talk/lisp-smalltalk-and-the-power-of-symmetry-8bd96aaa0c0c) that a form of code-data equivalence (symmetry!), not macros specifically, is what makes Lisp powerful. If so, there may be other ways to reach that equivalence. For example Smalltalk, like Lisp, *runs in the same context it's written in*. All Smalltalk data are programs. Smalltalk [may be making a comeback](https://hackernoon.com/how-to-evangelize-a-programming-language-0p7p3y02), in the form of [Pharo](https://pharo.org/). + +Haskell aims at code-data equivalence from a third angle (memoized pure functions are in essence infinite lookup tables), but I have not used it in practice, so I do not have the experience to say whether this is enough to make it feel powerful in a similar way. + +Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world is not that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead - without restarting the whole app at each change. Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. In web applications, [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a small step in a somewhat similar direction (as long as one can restart the server app easily, to make it use the latest definitions). + +But to know exactly what Common Lisp has to offer, **yes**, it does make sense to learn it. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. + +Having more perspectives at one's disposal makes one a better programmer - and that is what ultimately counts. As [Alan Perlis said in 1982](https://en.wikiquote.org/wiki/Alan_Perlis): + +*A language that doesn't affect the way you think about programming, is not worth knowing.* + +In this sense, Common Lisp is very much worth knowing. Although, if you want a beautiful, advanced Lisp, maybe go for [Racket](https://racket-lang.org/) first; but that is an essay for another day. + + # `hoon`: The C of Functional Programming +*9 June 2021* + Some days I wonder if this whole `unpythonic` endeavor even makes any sense. Then, turning the pages of [the book of sand](https://en.wikipedia.org/wiki/The_Book_of_Sand) that is the web, I [happen to run into something](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) like `hoon`. Its philosophy is best described by this gem from an [early version of its documentation](https://github.com/cgyarvin/urbit/blob/master/doc/book/0-intro.markdown#hoon): @@ -144,30 +179,3 @@ Using natural numbers for the opcodes at first glance sounds like a [Gödel numb Obviously, any interesting programs correspond to very large numbers, and are few and far between, so decoding random numbers via a Gödel numbering is not a practical way to generate interesting programs. [Genetic programming](https://en.wikipedia.org/wiki/Genetic_programming) works much better, because unlike Gödel numbering, it was actually designed specifically to do that. GP takes advantage of the semantic structure present in the source code (or AST) representation. The purpose of the original Gödel numbering was to prove Gödel's incompleteness theorem. In the case of `nock`, my impression is that the opcodes are natural numbers just for flavoring purposes. If you are building an ab initio software stack, what better way to announce that than to use natural numbers as your virtual machine's opcodes? - - -# Common Lisp, Python, and productivity - -The various essays Paul Graham wrote near the turn of the millennium, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for the Lisp family of languages. So how does the programming world look in that light now, 20 years later? - -The base abstraction level of programming languages, even those in popular use, has increased. The trend was visible already then, and was indeed noted in the essays. The focus on low-level languages such as C++ has decreased. Java is still popular, but high-level FP languages that compile to JVM bytecode (Kotlin, Scala, Clojure) are rising. - -Python has become highly popular, and is now also closer to Lisp than it was 20 years ago, especially after `MacroPy` introduced syntactic macros to Python (in 2013, [according to the git log](https://github.com/lihaoyi/macropy/commits/python2/macropy/__init__.py)). Python was not bad as a Lisp replacement even back in 2000 - see Peter Norvig's essay [Python for Lisp Programmers](https://norvig.com/python-lisp.html). Some more historical background, specifically on lexically scoped closures (and the initial lack thereof), can be found in [PEP 3104](https://www.python.org/dev/peps/pep-3104/), [PEP 227](https://www.python.org/dev/peps/pep-0227/), and [Historical problems with closures in JavaScript and Python](http://giocc.com/problems-with-closures-in-javascript-and-python.html). - -In 2020, does it still make sense to learn [the legendary](https://xkcd.com/297/) Common Lisp? - -As a practical tool? Is CL hands-down better than Python? Maybe no. Python has already delivered on 90% of the productivity promise of Lisp. Both languages cut down significantly on [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet). Python has a huge library ecosystem. [`mcpyrate`](https://github.com/Technologicat/mcpyrate) and `unpythonic` are trying to push the language-level features a further 5%. (A full 100% is likely impossible when extending an existing language; if nothing else, there will be seams.) - -As for productivity, [it may be](https://medium.com/smalltalk-talk/lisp-smalltalk-and-the-power-of-symmetry-8bd96aaa0c0c) that a form of code-data equivalence (symmetry!), not macros specifically, is what makes Lisp powerful. If so, there may be other ways to reach that equivalence. For example Smalltalk, like Lisp, *runs in the same context it's written in*. All Smalltalk data are programs. Smalltalk [may be making a comeback](https://hackernoon.com/how-to-evangelize-a-programming-language-0p7p3y02), in the form of [Pharo](https://pharo.org/). - -Haskell aims at code-data equivalence from a third angle (memoized pure functions are in essence infinite lookup tables), but I have not used it in practice, so I do not have the experience to say whether this is enough to make it feel powerful in a similar way. - -Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world is not that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead - without restarting the whole app at each change. Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. In web applications, [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a small step in a somewhat similar direction (as long as one can restart the server app easily, to make it use the latest definitions). - -But to know exactly what Common Lisp has to offer, **yes**, it does make sense to learn it. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. - -Having more perspectives at one's disposal makes one a better programmer - and that is what ultimately counts. As [Alan Perlis said in 1982](https://en.wikiquote.org/wiki/Alan_Perlis): - -*A language that doesn't affect the way you think about programming, is not worth knowing.* - -In this sense, Common Lisp is very much worth knowing. Although, if you want a beautiful, advanced Lisp, maybe go for [Racket](https://racket-lang.org/) first; but that is an essay for another day. From e1bbe7e7ae07369b5d4304b81e4050af01202ee4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 00:29:09 +0300 Subject: [PATCH 467/832] update readings on ab-initio programming language efforts. --- doc/readings.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/doc/readings.md b/doc/readings.md index 3c0e9d7d..4cbcbad6 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -188,10 +188,18 @@ The common denominator is programming. Some relate to language design, some to c - [`hoon`: The C of Functional Programming](https://urbit.org/docs/hoon/) - Interesting take on an alternative computing universe where the functional camp won systems programming. These people have built [a whole operating system](https://github.com/urbit/urbit) on a Turing-complete non-lambda automaton, Nock. - For my take, see [the opinion piece in Essays](essays.md#hoon-the-c-of-functional-programming). - - Judging by the docs, `hoon` is definitely ha-ha-only-serious, but I am not sure of whether it is serious-serious. It does advertise itself as the functional-programming equivalent of C. See the comments to [the entry on Manuel Simoni's blog](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) - some people do think `hoon` is actually useful. + - Judging by the docs, `hoon` is definitely ha-ha-only-serious, but I am not sure of whether it is serious-serious. See the comments to [the entry on Manuel Simoni's blog](http://axisofeval.blogspot.com/2015/07/what-i-learned-about-urbit-so-far.html) - some people do think `hoon` is actually useful. - Technical points: - `hoon` does not have syntactic macros. The reason given in the docs is the same as sometimes heard in the Python community - having a limited number of standard control structures, you always know what you are looking at. - - Interestingly, `hoon` has uniform support for *wide* and *tall* modes; it does not use parentheses, but uses a single space (in characteristic `hoon` fashion, termed an *ace*) versus multiple spaces (respectively, a *gap*). "Multiple spaces" allows also newlines, like in LaTeX. So [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) is not alone. + - Interestingly, `hoon` has uniform support for *wide* and *tall* modes; it does not use parentheses, but uses a single space (in characteristic `hoon` fashion, termed an *ace*) versus multiple spaces (respectively, a *gap*). "Multiple spaces" allows also newlines, like in LaTeX. So [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) is not the only attempt at a two-mode uniform grouping syntax. + +- *Ab initio* programming language efforts: + - `hoon`, see separate entry above. + - [Arc](http://www.paulgraham.com/arc.html) by Paul Graham and Robert Morris. + - [Discussion on](https://news.ycombinator.com/item?id=10535364) the Nile programming language developed by Ian Piumarta, Alan Kay, et al. + - Especially the low-level [Maru](https://www.piumarta.com/software/maru/) language by Ian Piumarta seems interesting. + - *Maru is a symbolic expression evaluator that can compile its own implementation language.* + - It compiles s-expressions to IA32 machine code, and has a metacircular evaluator implemented in less than 2k SLOC. It bootstraps from C. - [LtU: Why is there no widely accepted progress for 50 years?](http://lambda-the-ultimate.org/node/5590) - Discussion on how programming languages *have* improved. From babbac2fb63beff35e727655d81bff45d06e6bf3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 00:29:30 +0300 Subject: [PATCH 468/832] elaborate on how to use different features of Python in a lambda --- doc/design-notes.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index b7e1fbd0..25650cac 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -115,15 +115,23 @@ The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [ma The oft-quoted single-expression limitation of the Python ``lambda`` is ultimately a herring, as this library demonstrates. The real problem is the statement/expression dichotomy. In Python, the looping constructs (`for`, `while`), the full power of `if`, and `return` are statements, so they cannot be used in lambdas. (This observation has been earlier made by others, too; see e.g. the [Wikipedia page on anonymous functions](https://en.wikipedia.org/wiki/Anonymous_function#Python).) We can work around some of this: + - The expr macro `do[]` gives us sequencing, i.e. allows to use, in any expression position, multiple expressions that run in the specified order. - The expr macro ``cond[]`` gives us a general ``if``/``elif``/``else`` expression. - Without it, the expression form of `if` (that Python already has) could be used, but readability suffers if nested, since it has no ``elif``. Actually, [`and` and `or` are sufficient for full generality](https://www.ibm.com/developerworks/library/l-prog/), but readability suffers even more. - So we use macros to define a ``cond`` expression, essentially duplicating a feature the language already almost has. See [our macros](macros.md). - - Functional looping (with TCO, to boot) is possible. See the constructs in ``unpythonic.fploop``. + - Functional looping (with TCO) gives us equivalents of ``for`` and ``while``. See the constructs in ``unpythonic.fploop``, particularly ``looped`` and ``breakably_looped``. - ``unpythonic.ec.call_ec`` gives us ``return`` (the ec). - ``unpythonic.misc.raisef`` gives us ``raise``, and ``unpythonic.misc.tryf`` gives us ``try``/``except``/``else``/``finally``. - - A lambda can be named (``unpythonic.misc.namelambda``, with some practical limitations on the fully qualified name of nested lambdas). - - Even an anonymous function can recurse with some help (``unpythonic.fun.withself``). + - A lambda can be named, see ``unpythonic.misc.namelambda``. + - There are some practical limitations on the fully qualified name of nested lambdas. + - Note this does not bind the name to an identifier at the use site, so the name cannot be used to recurse. The point is that the name is available for inspection, and it will show in tracebacks. + - A lambda can recurse using ``unpythonic.fun.withself``. You will get a `self` argument that points to the lambda itself, and is passed implicitly, like `self` usually in Python. + - A lambda can define a class using the three-argument form of the builtin `type` function. For an example, see [Peter Corbett (2005): Statementless Python](https://gist.github.com/brool/1679908), a complete minimal Lisp interpreter implemented as a single Python expression. + - A lambda can import a module using the builtin `__import__`, or better, `importlib.import_module`. + - A lambda can assert by using an if-expression and then ``raisef`` to actually raise the ``AssertionError``. + - This can be packaged into a function ``assertf``, though that requires jumping through some hoops to produce a traceback that omits ``assertf`` itself. See ``equip_with_traceback``. - Context management (``with``) is currently **not** available for lambdas, even in ``unpythonic``. + - Aside from the `async` stuff, this is the last hold-out preventing full generality, so we will likely add an expression form of ``with`` in a future version. This is tracked in [issue #76](https://github.com/Technologicat/unpythonic/issues/76). Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. From ff530d237898b7344aabf885da022ba73f0efd16 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 02:08:50 +0300 Subject: [PATCH 469/832] almost-final README for 0.15.0 --- README.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 2d645c8d..ce516e20 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ The features of `unpythonic` are built out of, in increasing order of [magic](ht - Pure Python (e.g. batteries for `itertools`), - Macros driving a pure-Python core (`do`, `let`), - Pure macros (e.g. `continuations`, `lazify`, `dbg`). - - Whole-module transformations, a.k.a. dialects. + - Whole-module transformations, a.k.a. dialects (e.g. `Lispy`). This depends on the purpose of each feature, as well as ease-of-use considerations. See the design notes for more information. @@ -348,7 +348,7 @@ If this sounds a lot like an exception system, that's because conditions are the Roughly, a [symbol](https://stackoverflow.com/questions/8846628/what-exactly-is-a-symbol-in-lisp-scheme) is a guaranteed-[interned](https://en.wikipedia.org/wiki/String_interning) string. -A [gensym](http://clhs.lisp.se/Body/f_gensym.htm) is a guaranteed-unique string, which is useful as a nonce value. It's similar to the pythonic idiom `nonce = object()`, but with a nice repr, and object-identity-preserving pickle support. +A [gensym](http://clhs.lisp.se/Body/f_gensym.htm) is a guaranteed-*unique* string, which is useful as a nonce value. It's similar to the pythonic idiom `nonce = object()`, but with a nice repr, and object-identity-preserving pickle support. ```python from unpythonic import sym # lispy symbol @@ -559,7 +559,8 @@ with session("simple framework demo"): test[returns_normally(g(2, 3))] test[g(2, 3) == 6] # Use `the[]` (or several) in a `test[]` to declare what you want to inspect if the test fails. - test[counter() < the[counter()]] + # Implicit `the[]`: in comparison, the LHS; otherwise the whole expression. Used if no explicit `the[]`. + test[the[counter()] < the[counter()]] with testset("outer"): with testset("inner 1"): @@ -729,11 +730,11 @@ with continuations: # enables also TCO automatically The [dialects subsystem of `mcpyrate`](https://github.com/Technologicat/mcpyrate/blob/master/doc/dialects.md) makes Python into a language platform, à la [Racket](https://racket-lang.org/). We provide some example dialects based on `unpythonic`'s macro layer. See [documentation](doc/dialects.md). -
Lispython: The love child of Python and Scheme. +
Lispython: automatic TCO and an implicit return statement. [[docs](doc/dialects/lispython.md)] -Python with automatic tail-call optimization, an implicit return statement, and automatically named, multi-expression lambdas. +Also comes with automatically named, multi-expression lambdas. ```python from unpythonic.dialects import dialects, Lispython # noqa: F401 @@ -760,12 +761,10 @@ g = lambda x: [local[y << 2 * x], assert g(10) == 21 ```
-
Pytkell: Because it's good to have a kell. +
Pytkell: Automatic currying and implicitly lazy functions. [[docs](doc/dialects/pytkell.md)] -Python with automatic currying and implicitly lazy functions. - ```python from unpythonic.dialects import dialects, Pytkell # noqa: F401 @@ -786,12 +785,10 @@ assert my_prod(range(1, 5)) == 24 assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) ```
-
Listhell: It's not Lisp, it's not Python, it's not Haskell. +
Listhell: Prefix syntax for function calls, and automatic currying. [[docs](doc/dialects/listhell.md)] -Python with prefix syntax for function calls, and automatic currying. - ```python from unpythonic.dialects import dialects, Listhell # noqa: F401 From 7f1b7ea1899103d20128d1c6d9751a279f435b0d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 02:09:23 +0300 Subject: [PATCH 470/832] 0.15.0: improve let docs --- doc/features.md | 117 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 109 insertions(+), 8 deletions(-) diff --git a/doc/features.md b/doc/features.md index 09c44236..bab1687e 100644 --- a/doc/features.md +++ b/doc/features.md @@ -102,15 +102,18 @@ Tools to bind identifiers in ways not ordinarily supported by Python. ### ``let``, ``letrec``: local bindings in an expression -**NOTE**: This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use. Below is the documentation for the raw API. +**NOTE**: This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API. + +The `let` constructs introduce bindings local to an expression, like Scheme's ``let`` and ``letrec``. -Introduces bindings local to an expression, like Scheme's ``let`` and ``letrec``. For easy-to-use versions of these constructs that look almost like normal Python, see [our macros](macros.md). +#### ``let`` In ``let``, the bindings are independent (do not see each other). A binding is of the form ``name=value``, where ``name`` is a Python identifier, and ``value`` is any expression. Use a `lambda e: ...` to supply the environment to the body: ```python +# These six are the constructs covered in this section of documentation. from unpythonic import let, letrec, dlet, dletrec, blet, bletrec u = lambda lst: let(seen=set(), @@ -125,6 +128,8 @@ Generally speaking, `body` is a one-argument function, which takes in the enviro *Let over lambda*. Here the inner ``lambda`` is the definition of the function ``counter``: ```python +from unpythonic import let, begin + counter = let(x=0, body=lambda e: lambda: @@ -134,6 +139,21 @@ counter() # --> 1 counter() # --> 2 ``` +For comparison, with the macro API, this becomes: + +```python +from unpythonic.syntax import macros, let, do + +counter = let[[x << 0] in + (lambda: + do[x << x + 1, + x])] +counter() # --> 1 +counter() # --> 2 +``` + +(*The parentheses around the lambda are just to make the expression into syntactically valid Python. You can also use brackets instead, denoting a multiple-expression `let` body - which is also valid even if there is just one expression. The `do` makes a multiple-expression `lambda` body. For more, see the [macro documentation](macros.md).*) + Compare the sweet-exp [Racket](http://racket-lang.org/) (see [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) and [sweet](https://docs.racket-lang.org/sweet/)): ```racket @@ -146,9 +166,13 @@ counter() ; --> 1 counter() ; --> 2 ``` +#### ``dlet``, ``blet`` + *Let over def* decorator ``@dlet``, to *let over lambda* more pythonically: ```python +from unpythonic import dlet + @dlet(x=0) def counter(*, env=None): # named argument "env" filled in by decorator env.x += 1 @@ -157,9 +181,28 @@ counter() # --> 1 counter() # --> 2 ``` -In `letrec`, bindings may depend on ones above them in the same `letrec`, by using `lambda e: ...` (**Python 3.6+**): +For comparison, with the macro API, this becomes: ```python +from unpythonic.syntax import macros, dlet + +@dlet(x << 0) +def counter(): + x << x + 1 + return x +counter() # --> 1 +counter() # --> 2 +``` + +The ``@blet`` decorator is otherwise the same as ``@dlet``, but instead of decorating a function definition in the usual manner, it runs the `def` block immediately, and upon exit, replaces the function definition with the return value. The name ``blet`` is an abbreviation of *block let*, since the role of the `def` is just a code block to be run immediately. + +#### ``letrec`` + +In `letrec`, bindings may depend on ones above them in the same `letrec`, by using `lambda e: ...`: + +```python +from unpythonic import letrec + x = letrec(a=1, b=lambda e: e.a + 1, @@ -167,13 +210,27 @@ x = letrec(a=1, e.b) # --> 2 ``` -In `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent any misunderstandings in the environment initialization procedure. +The ordering of the definitions is respected, because Python 3.6 and later preserve the ordering of named arguments passed in a function call. See [PEP 468](https://www.python.org/dev/peps/pep-0468/). + +For comparison, with the macro API, this becomes: + +```python +from unpythonic.syntax import macros, letrec + +x = letrec[[a << 1, + b << a + 1] in + b] +``` + +In the non-macro `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent any misunderstandings in the environment initialization procedure. In a non-callable ``valexpr``, trying to depend on a binding below it raises ``AttributeError``. -A callable ``valexpr`` may depend on any bindings (also later ones) in the same `letrec`. Mutually recursive functions: +A callable ``valexpr`` may depend on any bindings (also later ones) in the same `letrec`. For example, here is a pair of mutually recursive functions: ```python +from unpythonic import letrec + letrec(evenp=lambda e: lambda x: (x == 0) or e.oddp(x - 1), @@ -184,9 +241,24 @@ letrec(evenp=lambda e: e.evenp(42)) # --> True ``` +For comparison, with the macro API, this becomes: + +```python +from unpythonic.syntax import macros, letrec + +letrec[[evenp << (lambda x: + (x == 0) or oddp(x - 1)), + oddp << (lambda x: + (x != 0) and evenp(x - 1))] in + evenp(42)] # --> True +``` + + Order-preserving list uniqifier: ```python +from unpythonic import letrec, begin + u = lambda lst: letrec(seen=set(), see=lambda e: lambda x: @@ -196,11 +268,22 @@ u = lambda lst: letrec(seen=set(), [e.see(x) for x in lst if x not in e.seen]) ``` -**CAUTION**: in Pythons older than 3.6, bindings are **initialized in an arbitrary order**, also in `letrec`. This is a limitation of the kwargs abuse. Hence mutually recursive functions are possible, but a non-callable `valexpr` cannot depend on other bindings in the same `letrec`. +For comparison, with the macro API, this becomes: -Trying to access `e.foo` from `e.bar` arbitrarily produces either the intended value of `e.foo`, or the uninitialized `lambda e: ...`, depending on whether `e.foo` has been initialized or not at the point of time when `e.bar` is being initialized. +```python +from unpythonic.syntax import macros, letrec, do + +u = lambda lst: letrec[[seen << set(), + see << (lambda x: + do[seen.add(x), + x])] in + [[see(x) for x in lst if x not in seen]]] +``` + +(*The double brackets around the `letrec` body are needed because brackets denote a multiple-expression `letrec` body. So it is a multiple-expression body that contains just one expression, which is a list comprehension.*) + +The decorators ``@dletrec`` and ``@bletrec`` work otherwise exactly like ``@dlet`` and ``@blet``, respectively, but the bindings are scoped like in ``letrec`` (mutually recursive scope). -This has been fixed in Python 3.6, see [PEP 468](https://www.python.org/dev/peps/pep-0468/). #### Lispylet: alternative syntax @@ -233,6 +316,24 @@ letrec((("evenp", lambda e: The syntax is `let(bindings, body)` (respectively `letrec(bindings, body)`), where `bindings` is `((name, value), ...)`, and `body` is like in the default variants. The same rules concerning `name` and `value` apply. +For comparison, with the macro API, the above becomes: + +```python +from unpythonic.syntax import macros, letrec + +letrec[[a << 1, + b << a + 1] in + b] + +letrec[[evenp << (lambda x: + (x == 0) or oddp(x - 1)), + oddp << (lambda x: + (x != 0) and evenp(x - 1))] in + evenp(42)] # --> True +``` + +(*The transformations made by the macros may be the most apparent when comparing these examples. Note that the macros scope the `let` bindings lexically, automatically figuring out which `let` environment, if any, to refer to.*) + ### ``env``: the environment From ee2068d988edd6c80d9b6a0e26695824fe17c516 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 02:35:05 +0300 Subject: [PATCH 471/832] 0.15.0: improve assignonce docs --- doc/features.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index bab1687e..54434e87 100644 --- a/doc/features.md +++ b/doc/features.md @@ -380,6 +380,8 @@ When the `with` block exits, the environment clears itself. The environment inst ### ``assignonce`` +*As of v0.15.0, `assignonce` is mostly a standalone curiosity that has never been integrated with the rest of `unpythonic`. But anything that works with arbitrary subclasses of `env`, for example `mogrify`, works with it, too.* + In Scheme terms, make `define` and `set!` look different: ```python @@ -392,7 +394,7 @@ with assignonce() as e: e.foo = "quux" # AttributeError, e.foo already defined. ``` -It's a subclass of ``env``, so it shares most of the same [features](#env-the-environment) and allows similar usage. +The `assignonce` construct is a subclass of ``env``, so it shares most of the same [features](#env-the-environment) and allows similar usage. #### Historical note From 33a75f3e4c86b945202a61eb07b8252ae192b36d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 02:35:22 +0300 Subject: [PATCH 472/832] 0.15.0: improve dynamic assignment docs --- doc/features.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/doc/features.md b/doc/features.md index 54434e87..58b5dff3 100644 --- a/doc/features.md +++ b/doc/features.md @@ -403,9 +403,13 @@ The fact that in Python creating bindings and updating (rebinding) them look the ### ``dyn``: dynamic assignment -([As termed by Felleisen.](https://groups.google.com/forum/#!topic/racket-users/2Baxa2DxDKQ) Other names seen in the wild for variants of this feature include *parameters* (not to be confused with function parameters), *special variables*, *fluid variables*, *fluid let*, and even the misnomer *"dynamic scoping"*.) +**Changed in v0.14.2.** *To bring this in line with [SRFI-39](https://srfi.schemers.org/srfi-39/srfi-39.html), `dyn` now supports rebinding, using assignment syntax such as `dyn.x = 42`, and the function `dyn.update(x=42, y=17, ...)`.* + +([As termed by Felleisen.](https://groups.google.com/forum/#!topic/racket-users/2Baxa2DxDKQ) Other names seen in the wild for variants of this feature include *parameters* ([Scheme](https://srfi.schemers.org/srfi-39/srfi-39.html) and [Racket](https://docs.racket-lang.org/reference/parameters.html); not to be confused with function parameters), *special variables* (Common Lisp), *fluid variables*, *fluid let* (e.g. Emacs Lisp), and even the misnomer *"dynamic scoping"*.) + +The feature itself is *dynamic assignment*; the things it creates are *dynamic variables* (a.k.a. *dynvars*). -Like global variables, but better-behaved. Useful for sending some configuration parameters through several layers of function calls without changing their API. Best used sparingly. +Dynvars are like global variables, but better-behaved. Useful for sending some configuration parameters through several layers of function calls without changing their API. Best used sparingly. There's a singleton, `dyn`: @@ -435,32 +439,30 @@ def g(): g() ``` -Dynamic variables (a.k.a. *dynvars*) are created using `with dyn.let(k0=v0, ...)`. The syntax is in line with the nature of the assignment, which is in effect *for the dynamic extent* of the `with`. Exiting the `with` block pops the dynamic environment stack. Inner dynamic environments shadow outer ones. +Dynvars are created using `with dyn.let(k0=v0, ...)`. The syntax is in line with the nature of the assignment, which is in effect *for the dynamic extent* of the `with`. Exiting the `with` block pops the dynamic environment stack. Inner dynamic environments shadow outer ones. -The point of dynamic assignment is that dynvars are seen also by code that is outside the lexical scope where the `with dyn.let` resides. The use case is to avoid a function parameter definition cascade, when you need to pass some information through several layers that don't care about it. This is especially useful for passing "background" information, such as plotter settings in scientific visualization, or the macro expander instance in metaprogramming. +The point of dynamic assignment is that dynvars are seen also by code that is *outside the lexical scope* where the `with dyn.let` resides. The use case is to avoid a function parameter definition cascade, when you need to pass some information through several layers that do not care about it. This is especially useful for passing "background" information, such as plotter settings in scientific visualization, or the macro expander instance in metaprogramming. To give a dynvar a top-level default value, use ``make_dynvar(k0=v0, ...)``. Usually this is done at the top-level scope of the module for which that dynvar is meaningful. Each dynvar, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). To rebind existing dynvars, use `dyn.k = v`, or `dyn.update(k0=v0, ...)`. Rebinding occurs in the closest enclosing dynamic environment that has the target name bound. If the name is not bound in any dynamic environment (including the top-level one), ``AttributeError`` is raised. -**CAUTION**: Use rebinding of dynvars carefully, if at all. Stealth updates of dynvars defined in an enclosing dynamic extent can destroy any chance of statically reasoning about the code. +**CAUTION**: Use rebinding of dynvars carefully, if at all. Stealth updates of dynvars defined in an enclosing dynamic extent can destroy any chance of statically reasoning about your code. There is no `set` function or `<<` operator, unlike in the other `unpythonic` environments. -**Changed in v0.14.2.** *To bring this in line with [SRFI-39](https://srfi.schemers.org/srfi-39/srfi-39.html), `dyn` now supports rebinding, using assignment syntax such as `dyn.x = 42`, and the function `dyn.update(x=42, y=17, ...)`.* -
Each thread has its own dynamic scope stack. There is also a global dynamic scope for default values, shared between threads. A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. -The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, PRs welcome!) +The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, a PR is welcome!) -Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (after the global definition has been executed). +Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (at any time after the global definition has been executed).
For more details, see the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a *live view* to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. ``dyn`` also provides the ``collections.abc.Mapping`` API. -To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, so you can do things like: +To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, to allow things like: ```python print(tuple((k, dyn[k]) for k in dyn)) @@ -472,9 +474,9 @@ For some more details, see [the unit tests](../unpythonic/tests/test_dynassign.p ### Relation to similar features in Lisps -This is essentially [SRFI-39: Parameter objects](https://srfi.schemers.org/srfi-39/), using the MzScheme approach in the presence of multiple threads. +This is essentially [SRFI-39: Parameter objects](https://srfi.schemers.org/srfi-39/) for Python, using the MzScheme approach in the presence of multiple threads. -[Racket](http://racket-lang.org/)'s [`parameterize`](https://docs.racket-lang.org/guide/parameterize.html) behaves similarly. However, Racket seems to be the state of the art in many lispy language design related things, so its take on the feature may have some finer points I haven't thought of. +[Racket](http://racket-lang.org/)'s [`parameterize`](https://docs.racket-lang.org/guide/parameterize.html) behaves similarly. However, Racket seems to be the state of the art in many lispy language design related things, so its take on the feature may have some finer points I have not thought of. On Common Lisp's special variables, see [Practical Common Lisp by Peter Seibel](http://www.gigamonkeys.com/book/variables.html), especially footnote 10 in the linked chapter, for a definition of terms. Similarly, dynamic variables in our `dyn` have *indefinite scope* (because `dyn` is implemented as a module-level global, accessible from anywhere), but *dynamic extent*. From e8d8b891f284cbc419d35a976b28f115a3584b7e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 02:36:00 +0300 Subject: [PATCH 473/832] 0.15.0: improve container docs --- doc/features.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/features.md b/doc/features.md index 58b5dff3..aee48501 100644 --- a/doc/features.md +++ b/doc/features.md @@ -485,13 +485,15 @@ So what we have in `dyn` is almost exactly like Common Lisp's special variables, ## Containers -We provide some additional containers. +We provide some additional low-level containers beyond those provided by Python itself. The class names are lowercase, because these are intended as low-level utility classes in principle on par with the builtins. The immutable containers are hashable. All containers are pickleable (if their contents are). ### ``frozendict``: an immutable dictionary -Given the existence of ``dict`` and ``frozenset``, this one is oddly missing from the standard library. +**Changed in 0.14.2**. *[A bug in `frozendict` pickling](https://github.com/Technologicat/unpythonic/issues/55) has been fixed. Now also the empty `frozendict` pickles and unpickles correctly.* + +Given the existence of ``dict`` and ``frozenset``, this one is oddly missing from the language. ```python from unpythonic import frozendict @@ -529,7 +531,7 @@ assert fd == {1: 2, 3: 4} **The usual caution** concerning immutable containers in Python applies: the container protects only the bindings against changes. If the values themselves are mutable, the container cannot protect from mutations inside them. -All the usual read-access stuff works: +All the usual read-access features work: ```python d7 = frozendict({1:2, 3:4}) @@ -559,7 +561,7 @@ assert hash(d7) == hash(frozendict({1:2, 3:4})) assert hash(d7) != hash(frozendict({1:2})) ``` -The abstract superclasses are virtual, just like for ``dict`` (i.e. they do not appear in the MRO). +The abstract superclasses are virtual, just like for ``dict``. We mean *virtual* in the sense of [`abc.ABCMeta`](https://docs.python.org/3/library/abc.html#abc.ABCMeta), i.e. a virtual superclass does not appear in the MRO. Finally, ``frozendict`` obeys the empty-immutable-container singleton invariant: @@ -567,8 +569,6 @@ Finally, ``frozendict`` obeys the empty-immutable-container singleton invariant: assert frozendict() is frozendict() ``` -**Changed in 0.14.2**. *[A bug in `frozendict` pickling](https://github.com/Technologicat/unpythonic/issues/55) has been fixed. Now also the empty `frozendict` pickles and unpickles correctly.* - ### `cons` and friends: pythonic lispy linked lists From 59933a9387a6874371c6902662ceab1192ed28d0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 03:14:25 +0300 Subject: [PATCH 474/832] update docstring --- unpythonic/collections.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/collections.py b/unpythonic/collections.py index 9d544168..3eaf65b7 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -810,7 +810,7 @@ def in_slice(i, s, length=None): (if ``s.start`` or ``s.stop`` is ``None``). If ``length is None``, negative or missing ``s.start`` or ``s.stop`` may raise - ValueError. (A negative ``s.step`` by itself does not need ``l``.) + ValueError. (A negative ``s.step`` by itself does not need ``length``.) """ if not isinstance(s, (slice, int)): raise TypeError(f"s must be slice or int, got {type(s)} with value {s}") From 220bf7449a068149c29f2828c59106bf5d94112e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 03:14:39 +0300 Subject: [PATCH 475/832] document nil as Singleton (actually already changed in 0.14.2) --- doc/features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/features.md b/doc/features.md index aee48501..315816af 100644 --- a/doc/features.md +++ b/doc/features.md @@ -574,6 +574,8 @@ assert frozendict() is frozendict() *Laugh, it's funny.* +**Changed in v0.14.2.** *`nil` is now a `Singleton`, so it is treated correctly by `pickle`. The `nil` instance refresh code inside the `cons` class has been removed, so the previous caveat about pickling a standalone `nil` value no longer applies.* + ```python from unpythonic import (cons, nil, ll, llist, car, cdr, caar, cdar, cadr, cddr, From f7da9d7ec16292d67d4cc78acf846b52a19c9adc Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 03:15:03 +0300 Subject: [PATCH 476/832] 0.15.0: update cons/llist docs --- doc/features.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/doc/features.md b/doc/features.md index 315816af..2795efdc 100644 --- a/doc/features.md +++ b/doc/features.md @@ -611,9 +611,9 @@ assert lzip(ll(1, 2, 3), ll(4, 5, 6)) == ll(ll(1, 4), ll(2, 5), ll(3, 6)) Cons cells are immutable à la Racket (no `set-car!`/`rplaca`, `set-cdr!`/`rplacd`). Accessors are provided up to `caaaar`, ..., `cddddr`. -Although linked lists are created with ``ll`` or ``llist``, the data type (for e.g. ``isinstance``) is ``cons``. +Although linked lists are created with the functions ``ll`` or ``llist``, the data type (for e.g. ``isinstance``) is ``cons``. -Iterators are supported to walk over linked lists (this also gives sequence unpacking support). When ``next()`` is called, we return the car of the current cell the iterator points to, and the iterator moves to point to the cons cell in the cdr, if any. When the cdr is not a cons cell, it is the next (and last) item returned; except if it `is nil`, then iteration ends without returning the `nil`. +Iterators are supported, to walk over linked lists. This also gives sequence unpacking support. When ``next()`` is called, we return the `car` of the current cell the iterator points to, and the iterator moves to point to the cons cell in the `cdr`, if any. When the `cdr` is not a cons cell, it is the next (and last) item returned; except if it `is nil`, then iteration ends without returning the `nil`. Python's builtin ``reversed`` can be applied to linked lists; it will internally ``lreverse`` the list (which is O(n)), then return an iterator to that. The ``llist`` constructor is special-cased so that if the input is ``reversed(some_ll)``, it just returns the internal already reversed list. (This is safe because cons cells are immutable.) @@ -639,7 +639,7 @@ For more, see the ``llist`` submodule. There is no ``copy`` method or ``lcopy`` function, because cons cells are immutable; which makes cons structures immutable. -(However, for example, it is possible to ``cons`` a new item onto an existing linked list; that's fine because it produces a new cons structure - which shares data with the original, just like in Racket.) +However, for example, it is possible to ``cons`` a new item onto an existing linked list; that is fine, because it produces a new cons structure - which shares data with the original, just like in Racket. In general, copying cons structures can be error-prone. Given just a starting cell it is impossible to tell if a given instance of a cons structure represents a linked list, or something more general (such as a binary tree) that just happens to locally look like one, along the path that would be traversed if it was indeed a linked list. @@ -649,8 +649,6 @@ We provide a ``JackOfAllTradesIterator`` as a compromise that understands both t ``cons`` has no ``collections.abc`` virtual superclasses (except the implicit ``Hashable`` since ``cons`` provides ``__hash__`` and ``__eq__``), because general cons structures do not fit into the contracts represented by membership in those classes. For example, size cannot be known without iterating, and depends on which iteration scheme is used (e.g. ``nil`` dropping, flattening); which scheme is appropriate depends on the content. -**Caution**: the ``nil`` singleton is freshly created in each session; newnil is not oldnil, so don't pickle a standalone ``nil``. The unpickler of ``cons`` automatically refreshes any ``nil`` instances inside a pickled cons structure, so that **cons structures** support the illusion that ``nil`` is a special value like ``None`` or ``...``. After unpickling, ``car(c) is nil`` and ``cdr(c) is nil`` still work as expected, even though ``id(nil)`` has changed between sessions. - ### ``box``: a mutable single-item container From 21745defee2c8dbffd6b01abd6497706eb00e35d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 11 Jun 2021 03:15:49 +0300 Subject: [PATCH 477/832] 0.15.0: improve box family docs --- doc/features.md | 55 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/doc/features.md b/doc/features.md index 2795efdc..13e7cb3c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -660,7 +660,9 @@ We provide a ``JackOfAllTradesIterator`` as a compromise that understands both t **Changed in v0.14.2**. *Accessing the `.x` attribute of a `box` directly is now deprecated. It will continue to work with `box` at least until 0.15, but it does not and cannot work with `ThreadLocalBox`, which must handle things differently due to implementation reasons. Use the API mentioned above; it supports both kinds of boxes with the same syntax.* -No doubt anyone programming in an imperative language has run into the situation caricatured by this highly artificial example: +#### ``box`` + +Consider this highly artificial example: ```python animal = "dog" @@ -672,7 +674,7 @@ f(animal) assert animal == "dog" ``` -Many solutions exist. Common pythonic ones are abusing a ``list`` to represent a box (and then trying to manually remember that it is supposed to hold only a single item), or (if the lexical structure of the particular piece of code allows it) using the ``global`` or ``nonlocal`` keywords to tell Python, on assignment, to overwrite a name that already exists in a surrounding scope. +Many solutions exist. Common pythonic ones are abusing a ``list`` to represent a box (and then trying to remember that it is supposed to hold only a single item), or (if the lexical structure of the particular piece of code allows it) using the ``global`` or ``nonlocal`` keywords to tell Python, on assignment, to overwrite a name that already exists in a surrounding scope. As an alternative to the rampant abuse of lists, we provide a rackety ``box``, which is a minimalistic mutable container that holds exactly one item. Any code that has a reference to the box can update the data in it: @@ -741,11 +743,21 @@ assert "fox" in box3 The expression ``item in b`` has the same meaning as ``unbox(b) == item``. Note ``box`` is a **mutable container**, so it is **not hashable**. -The expression `unbox(b)` has the same meaning as `b.get()`, but because it is a function (instead of a method), it additionally sanity checks that `b` is a box, and if not, raises `TypeError`. +The expression `unbox(b)` has the same meaning as `b.get()`, but because it is a function (instead of a method), it additionally sanity-checks that `b` is a box, and if not, raises `TypeError`. The expression `b << newitem` has the same meaning as `b.set(newitem)`. In both cases, the new value is returned as a convenience. -`ThreadLocalBox` is otherwise exactly like `box`, but it's magic: its contents are thread-local. It also holds a default object, which is set initially when the `ThreadLocalBox` is instantiated. The default object is seen by threads that have not placed any object into the box. +#### ``Some`` + +We also provide an **immutable** box, `Some`. This can be useful to represent optional data. + +The idea is that the value, when present, is placed into a `Some`, such as `Some(42)`, `Some("cat")`, `Some(myobject)`. Then, the situation where the value is absent can be represented as a bare `None`. So specifically, `Some(None)` means that a value is present and this value is `None`, whereas a bare `None` means that there is no value. + +(It is like the `Some` constructor of a `Maybe` monad, but with no monadic magic. In this interpretation, the bare constant `None` plays the role of `Nothing`.) + +#### ``ThreadLocalBox`` + +`ThreadLocalBox` is otherwise exactly like `box`, but magical: its contents are thread-local. It also holds a default object, which is set initially when the `ThreadLocalBox` is instantiated. The default object is seen by threads that have not placed any object into the box. ```python from unpythonic import ThreadLocalBox, unbox @@ -802,16 +814,14 @@ tlb.clear() # When we clear the box in this thread... assert unbox(tlb) == "cat" # ...this thread sees the current default object again. ``` -We also provide an **immutable** box, `Some`. This can be useful for optional data. The idea is that the value, when present, is placed into a `Some`, such as `Some(42)`, `Some("cat")`, `Some(myobject)`. Then, the situation where the value is absent can be represented as a bare `None`. So specifically, `Some(None)` means that a value is present and this value is `None`, whereas a bare `None` means that there is no value. - ### ``Shim``: redirect attribute accesses **Added in v0.14.2**. -A `Shim` is an attribute access proxy. The shim holds a `box` (or a `ThreadLocalBox`), and redirects attribute accesses on the shim to whatever object happens to currently be in the box. The point is that the object in the box can be replaced with a different one later (by sending another object into the box), and the code accessing the proxied object through the shim doesn't need to be aware that anything has changed. +A `Shim` is an *attribute access proxy*. The shim holds a `box` (or a `ThreadLocalBox`; your choice), and redirects attribute accesses on the shim to whatever object happens to currently be in the box. The point is that the object in the box can be replaced with a different one later (by sending another object into the box), and the code accessing the proxied object through the shim does not need to be aware that anything has changed. -For example, this can combo with `ThreadLocalBox` to redirect standard output only in particular threads. Place the stream object in a `ThreadLocalBox`, shim that box, then replace `sys.stdout` with the shim. See the source code of `unpythonic.net.server` for an example that actually does (and cleanly undoes) this. +For example, `Shim` can combo with `ThreadLocalBox` to redirect standard output only in particular threads. Place the stream object in a `ThreadLocalBox`, shim that box, then replace `sys.stdout` with the shim. See the source code of `unpythonic.net.server` for an example that actually does (and cleanly undoes) this. Since deep down, attribute access is the whole point of objects, `Shim` is essentially a transparent object proxy. (For example, a method call is an attribute read (via a descriptor), followed by a function call.) @@ -844,9 +854,9 @@ assert s.getme() == 42 assert not hasattr(s, "y") # The new TestTarget instance doesn't have "y". ``` -A shim can have an optional fallback object. It can be either any object, or a box if you want to replace the fallback later. **For attribute reads** (i.e. `__getattr__`), if the object in the primary box does not have the requested attribute, `Shim` will try to get it from the fallback. If `fallback` is boxed, the attribute read takes place on the object in the box. If it is not boxed, the attribute read takes place directly on `fallback`. +A shim can have an optional fallback object. It can be either any object, or a `box` (or `ThreadLocalBox`) if you want to replace the fallback later. **For attribute reads** (i.e. `__getattr__`), if the object in the primary box does not have the requested attribute, `Shim` will try to get it from the fallback. If `fallback` is boxed, the attribute read takes place on the object in the box. If it is not boxed, the attribute read takes place directly on `fallback`. -Any **attribute writes** (i.e. `__setattr__`, binding or rebinding an attribute) always take place on the object in the primary box. +Any **attribute writes** (i.e. `__setattr__`, binding or rebinding an attribute) always take place on the object in the **primary** box. That is, binding or rebinding of attributes is never performed on the fallback object. ```python from unpythonic import Shim, box, unbox @@ -889,9 +899,34 @@ assert s.y == "hi from Wai" assert s.z == "hi from Zee" ``` +Or, since the operation takes just one `elt` and an `acc`, we can also use `reducer` instead of `foldr`, shortening this by one line: + +```python +from unpythonic import Shim, box, unbox, reducer + +class Ex: + x = "hi from Ex" +class Wai: + x = "hi from Wai" + y = "hi from Wai" +class Zee: + x = "hi from Zee" + y = "hi from Zee" + z = "hi from Zee" + + # There will be tried from left to right. +boxes = [box(obj) for obj in (Ex(), Wai(), Zee())] +s = reducer(Shim, boxes) # Shim(box, fallback) <-> op(elt, acc) +assert s.x == "hi from Ex" +assert s.y == "hi from Wai" +assert s.z == "hi from Zee" +``` + ### Container utilities +**Changed in v0.15.0.** *The sequence length argument in `in_slice`, `index_in_slice` is now named `length`, not `l` (ell). This avoids an E741 warning in `flake8`, and is more descriptive.* + **Inspect the superclasses** that a particular container type has: ```python From a8cacf6251bb529d584ac11329553a4cf76729b8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 01:34:25 +0300 Subject: [PATCH 478/832] styling --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 13e7cb3c..46a8dad6 100644 --- a/doc/features.md +++ b/doc/features.md @@ -102,7 +102,7 @@ Tools to bind identifiers in ways not ordinarily supported by Python. ### ``let``, ``letrec``: local bindings in an expression -**NOTE**: This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API. +**NOTE**: *This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API.* The `let` constructs introduce bindings local to an expression, like Scheme's ``let`` and ``letrec``. From 78957aaff77e06dcc263a9efff1b853fc4cce7d2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 01:34:37 +0300 Subject: [PATCH 479/832] 0.15.0: improve do/do0 docs --- doc/features.md | 144 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 114 insertions(+), 30 deletions(-) diff --git a/doc/features.md b/doc/features.md index 46a8dad6..efd337f0 100644 --- a/doc/features.md +++ b/doc/features.md @@ -955,41 +955,53 @@ An optional length argument can be given to interpret negative indices. See the Sequencing refers to running multiple expressions, in sequence, in place of one expression. -Keep in mind the only reason to ever need multiple expressions: *side effects.* (Assignment is a side effect, too; it modifies the environment. In functional style, intermediate named definitions to increase readability are perhaps the most useful kind of side effect.) +Keep in mind the only reason to ever need multiple expressions: *side effects.* Assignment is a side effect, too; it modifies the environment. In functional style, intermediate named definitions to increase readability are perhaps the most useful kind of side effect. See also ``multilambda`` in [macros](macros.md). ### ``begin``: sequence side effects -**CAUTION**: the `begin` family of forms are provided **for use in pure-Python projects only** (and are a permanent part of the `unpythonic` API for that purpose). If your project uses macros, prefer the `do[]` and `do0[]` macros; these are the only sequencing constructs understood by other macros in `unpythonic.syntax` that need to perform tail-position analysis (e.g. `tco`, `autoreturn`, `continuations`). The `do[]` and `do0[]` macros also provide some convenience features, such as expression-local variables. +**CAUTION**: the `begin` family of forms are provided **for use in pure-Python projects only**, and are a permanent part of the `unpythonic` API for that purpose. They are somewhat simpler and less flexible than the `do` family, described further below. + +*If your project uses macros, prefer the `do[]` and `do0[]` macros; those are the only sequencing constructs understood by other macros in `unpythonic.syntax` that need to perform tail-position analysis (e.g. `tco`, `autoreturn`, `continuations`). The `do[]` and `do0[]` macros also provide some convenience features, such as expression-local variables.* ```python from unpythonic import begin, begin0 f1 = lambda x: begin(print("cheeky side effect"), - 42*x) + 42 * x) f1(2) # --> 84 -f2 = lambda x: begin0(42*x, +f2 = lambda x: begin0(42 * x, print("cheeky side effect")) f2(2) # --> 84 ``` -Actually a tuple in disguise. If worried about memory consumption, use `lazy_begin` and `lazy_begin0` instead, which indeed use loops. The price is the need for a lambda wrapper for each expression to delay evaluation, see [`unpythonic.seq`](../unpythonic/seq.py) for details. +The `begin` and `begin0` forms are actually tuples in disguise; evaluation of all items occurs before the `begin` or `begin0` form gets control. Items are evaluated left-to-right due to Python's argument passing rules. + +We provide also `lazy_begin` and `lazy_begin0`, which use loops. The price is the need for a lambda wrapper for each expression to delay evaluation, see [`unpythonic.seq`](../unpythonic/seq.py) for details. ### ``do``: stuff imperative code into an expression -**NOTE**: This is primarily a code generation target API for the ``do[]`` [macro](macros.md), which makes the construct easier to use. Below is the documentation for the raw API. +**NOTE**: *This is primarily a code generation target API for the ``do[]`` and ``do0[]`` [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API.* + +Basically, the ``do`` family is a more advanced and flexible variant of the ``begin`` family. + + - ``do`` can bind names to intermediate results and then use them in later items. + + - ``do`` is effectively a ``let*`` (technically, ``letrec``) where making a binding is optional, so that some items can have only side effects if so desired. There is no semantically distinct ``body``; all items play the same role. -No monadic magic. Basically, ``do`` is: + - Despite the name, there is no monadic magic. - - An improved ``begin`` that can bind names to intermediate results and then use them in later items. +Like in ``letrec``, use ``lambda e: ...`` to access the environment, and to wrap callable values (to prevent misinterpretation by the machinery). - - A ``let*`` (technically, ``letrec``) where making a binding is optional, so that some items can have only side effects if so desired. No semantically distinct ``body``; all items play the same role. +Unlike ``begin`` (and ``begin0``), there is no separate ``lazy_do`` (``lazy_do0``), because using a ``lambda e: ...`` wrapper for an item will already delay its evaluation; and the main point of ``do``/``do0`` is that there is an environment that holds local definitions. If you want a lazy variant, just wrap each item with a ``lambda e: ...``, also those that don't otherwise need it. -Like in ``letrec`` (see below), use ``lambda e: ...`` to access the environment, and to wrap callable values (to prevent misunderstandings). +#### ``do`` + +Like ``begin`` and ``lazy_begin``, the ``do`` form evaluates all items in order, and then returns the value of the **last** item. ```python from unpythonic import do, assign @@ -1002,7 +1014,7 @@ y = do(assign(x=17), # create and set e.x assert y == 42 y = do(assign(x=17), - assign(z=lambda e: 2*e.x), + assign(z=lambda e: 2 * e.x), lambda e: e.z) assert y == 34 @@ -1013,16 +1025,91 @@ y = do(assign(x=5), assert y == 25 ``` -If you need to return the first value instead of the last one, use this trick: +For comparison, with the macro API, this becomes: + +```python +from unpythonic.syntax import macros, do, local + +y = do[local[x << 17], # create and set an x local to the environment + print(x), + x << 23, # overwrite x + print(x), + 42] # return value +assert y == 42 + +y = do[local[x << 17], + local[z << 2 * x], + z] +assert y == 34 + +y = do[local[x << 5], + local[f << (lambda x: x**2)], + print("hello from 'do'"), + f(x)] +assert y == 25 +``` + +*In the macro version, all items are delayed automatically; that is, **every** item has an implicit ``lambda e: ...``.* + +*Note that instead of the `assign` function, the macro version uses the syntax ``local[name << value]`` to **create** an expression-local variable. Updating an existing variable in the `do` environment is just ``name << value``. Finally, there is also ``delete[name]`.* + +When using the raw API, beware of this pitfall: + +```python +from unpythonic import do + +do(lambda e: print("hello 2 from 'do'"), # delayed because lambda e: ... + print("hello 1 from 'do'"), # Python prints immediately before do() + "foo") # gets control, because technically, it is + # **the return value** that is an argument + # for do(). +``` + +The above pitfall also applies to using escape continuations inside a ``do``. To do that, wrap the ec call into a ``lambda e: ...`` to delay its evaluation until the ``do`` actually runs: ```python +from unpythonic import call_ec, do, assign + +call_ec( + lambda ec: + do(assign(x=42), + lambda e: ec(e.x), # IMPORTANT: must delay this! + lambda e: print("never reached"))) # and this (as above) +``` + +This way, any assignments made in the ``do`` (which occur only after ``do`` gets control), performed above the line with the ``ec`` call, will have been performed when the ``ec`` is called. + +For comparison, with the macro API, the last example becomes: + +```python +from unpythonic.syntax import macros, do, local +from unpythonic import call_ec + +call_ec( + lambda ec: + do[local[x << 42], + ec(x), + print("never reached")]) +``` + +*In the macro version, all items are delayed automatically, so there ``do``/``do0`` gets control before any items are evaluated. The `ec` fires when the `do` evaluates that item, and the `print` is indeed never reached.* + +#### ``do0`` + +Like ``begin0`` and ``lazy_begin0``, the ``do0`` form evaluates all items in order, and then returns the value of the **first** item. + +It effectively does this internally: + +```python +from unpythonic import do, assign + y = do(assign(result=17), print("assigned 'result' in env"), lambda e: e.result) # return value assert y == 17 ``` -Or use ``do0``, which does it for you: +So we can write: ```python from unpythonic import do0, assign @@ -1038,30 +1125,27 @@ y = do0(assign(x=17), # the first item of do0 can be an assignment, too assert y == 17 ``` -Beware of this pitfall: +For comparison, with the macro API, this becomes: ```python -do(lambda e: print("hello 2 from 'do'"), # delayed because lambda e: ... - print("hello 1 from 'do'"), # Python prints immediately before do() - "foo") # gets control, because technically, it is - # **the return value** that is an argument - # for do(). -``` +from unpythonic.syntax import macros, do, local -Unlike ``begin`` (and ``begin0``), there is no separate ``lazy_do`` (``lazy_do0``), because using a ``lambda e: ...`` wrapper will already delay evaluation of an item. If you want a lazy variant, just wrap each item (also those which don't otherwise need it). +y = do[local[result << 17], + print("assigned 'result' in env"), + result] +assert y == 17 -The above pitfall also applies to using escape continuations inside a ``do``. To do that, wrap the ec call into a ``lambda e: ...`` to delay its evaluation until the ``do`` actually runs: +y = do0[17, + local[x << 42], + print(x), + print("hello from 'do0'")] +assert y == 17 -```python -call_ec( - lambda ec: - do(assign(x=42), - lambda e: ec(e.x), # IMPORTANT: must delay this! - lambda e: print("never reached"))) # and this (as above) +y = do0[local[x << 17], + print(x)] +assert y == 17 ``` -This way, any assignments made in the ``do`` (which occur only after ``do`` gets control), performed above the line with the ``ec`` call, will have been performed when the ``ec`` is called. - ### ``pipe``, ``piped``, ``lazy_piped``: sequence functions From c87614008dc63334d019373f46ced7d7b80d10f0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 03:18:16 +0300 Subject: [PATCH 480/832] 0.15.0: update pipe system docs --- doc/features.md | 91 +++++++++++++++++++++++++++++++++++++++-------- unpythonic/seq.py | 2 +- 2 files changed, 78 insertions(+), 15 deletions(-) diff --git a/doc/features.md b/doc/features.md index efd337f0..2e2e044a 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1153,12 +1153,24 @@ assert y == 17 *The variants `pipe` and `pipec` now expect a `Values` initial value if you want to unpack it into the args and kwargs of the first function in the pipe. Otherwise, the initial value is sent as a single positional argument (notably tuples too).* -*The variants `piped` and `lazy_piped` pack the initial arguments automatically into a `Values`.* +*The variants `piped` and `lazy_piped` automatically pack the initial arguments into a `Values`.* -Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/). A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It's just function composition, but with an emphasis on data flow, which helps improve readability: +**Changed in v0.14.2**. *Both `getvalue` and `runpipe`, used in the shell-like syntax, are now known by the single unified name `exitpipe`. This is just a rename, with no functionality changes. The old names are deprecated in 0.14.2 and 0.14.3, and have been removed in 0.15.0.* + +Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/), but no macros. A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It is just function composition, but with an emphasis on data flow, which helps improve readability. + +Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with ``1``, and they are slightly faster than the general versions. The use case is one-argument functions that return one value. + +In the n-to-m versions, when a function returns a `Values`, it is unpacked to the args and kwargs of the next function in the pipeline. When a pipe exits, the `Values` wrapper (if any) around the final result is discarded if it contains only one positional value. The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as the args/kwargs of each output `Values` can be accepted as input by the next function in the pipe). + +Additional examples can be found in [the unit tests](../unpythonic/tests/test_seq.py). + +#### ``pipe`` + +The function `pipe` represents a self-contained pipeline that starts from a given value (or values), applies some operations in sequence, and then exits: ```python -from unpythonic import pipe +from unpythonic import pipe, Values double = lambda x: 2 * x inc = lambda x: x + 1 @@ -1167,11 +1179,43 @@ x = pipe(42, double, inc) assert x == 85 ``` -We also provide ``pipec``, which curries the functions before applying them. Useful with passthrough (see below on ``curry``). +To pass several positional values and/or named values, use a `Values` object: + +```python +from unpythonic import pipe, Values + +a, b = pipe(Values(2, 3), + lambda x, y: Values(x=(x + 1), y=(2 * y)), + lambda x, y: Values(x * 2, y + 1)) +assert (a, b) == (6, 7) +``` + +In this example, we pass the initial values positionally into the first function in the pipeline; that function passes its return values by name; and the second function in the pipeline passes the final results positionally. Because there are only positional values in the final `Values` object, it can be unpacked like a tuple. + +#### ``pipec`` + +The function ``pipec`` is otherwise exactly like ``pipe``, but it curries the functions before applying them. This is useful with the passthrough feature of ``curry``. + +With ``pipec`` you can do things like: + +```python +from unpythonic import pipec, Values + +a, b = pipec(Values(1, 2), + lambda x: x + 1, # extra values passed through by curry (positionals on the right) + lambda x, y: Values(x * 2, y + 1)) +assert (a, b) == (4, 3) +``` + +For more on passthrough, see the section on ``curry``. + +#### ``piped`` + +We also provide a **shell-like syntax**, with purely functional updates. -Optional **shell-like syntax**, with purely functional updates. +To set up a pipeline for use with the shell-like syntax, call ``piped`` to load the initial value(s). It is possible to provide both positional and named values. Each use of the pipe operator applies the given function, but keeps the result inside the pipeline, ready to accept another function. -**Changed in v0.14.2**. *Both `getvalue` and `runpipe` are now known by the single unified name `exitpipe`. This is just a rename, with no functionality changes. The old names are now deprecated, and will be removed in 0.15.0.* +When done, pipe into the sentinel ``exitpipe`` to exit the pipeline and return the current value(s): ```python from unpythonic import piped, exitpipe @@ -1184,9 +1228,33 @@ assert p | inc | exitpipe == 85 assert p | exitpipe == 84 # p itself is never modified by the pipe system ``` -Set up a pipe by calling ``piped`` for the initial value. Pipe into the sentinel ``exitpipe`` to exit the pipe and return the current value. +Multiple values work like in `pipe`, except the initial value(s) passed to ``piped`` are automatically packed into a `Values`. The pipe system then automatically unpacks a `Values` object into the args/kwargs of the next function in the pipeline. + +To return multiple positional values and/or named values, return a `Values` object from your function. + +When ``exitpipe`` is applied, if the last function returned anything other than one positional value, you will get a ``Values`` object. + +```python +from unpythonic import piped, exitpipe, Values + +f = lambda x, y: Values(2 * x, y + 1) +g = lambda x, y: Values(x + 1, 2 * y) +x = piped(2, 3) | f | g | exitpipe # --> (5, 8) +assert x == Values(5, 8) +``` + +Unpacking works also here, because in the final result, there are only positional values: -**Lazy pipes**, useful for mutable initial values. To perform the planned computation, pipe into the sentinel ``exitpipe``: +```python +from unpythonic import piped, exitpipe + +a, b = piped(2, 3) | f | g | exitpipe # --> (5, 8) +assert (a, b) == (5, 8) +``` + +#### ``lazy_piped`` + +Lazy pipes are useful when you have mutable initial values. To perform the planned computation, pipe into the sentinel ``exitpipe``: ```python from unpythonic import lazy_piped1, exitpipe @@ -1216,15 +1284,10 @@ def nextfibo(a, b): # multiple arguments allowed p = lazy_piped(1, 1) # load initial state for _ in range(10): # set up pipeline p = p | nextfibo -p | exitpipe -assert (p | exitpipe) == Values(a=89, b=144) # final state +assert (p | exitpipe) == Values(a=89, b=144) # run; check final state assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] ``` -Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with ``1``. The use case is one-argument functions that return one value (which may also be a tuple). - -In the n-to-m versions, when a function returns a `Values`, it is unpacked to the args and kwargs of the next function in the pipe. At ``exitpipe`` time, the `Values` wrapper (if any) around the final result is discarded if it contains only one positional value. The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as the args/kwargs of each output `Values` can be accepted as input by the next function in the pipe). - ## Batteries diff --git a/unpythonic/seq.py b/unpythonic/seq.py index d2b79cb2..712ed8d4 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -431,7 +431,7 @@ def nextfibo(a, b): # now two arguments p = lazy_piped(1, 1) for _ in range(10): p = p | nextfibo - assert p | exitpipe == Values(a=89, b=144) # final state + assert p | exitpipe == Values(a=89, b=144) # run; check final state assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] """ def __init__(self, *xs, _funcs=None, **kws): From af20a83ff87408298ed2279e309581b4ac7d2293 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 12:46:12 +0300 Subject: [PATCH 481/832] fix borked link --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 2e2e044a..0d7cf8cf 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4073,7 +4073,7 @@ assert f(1, 2, 3) == Values(1, 2, 3) ## Numerical tools -We briefly introduce the functions below. More details and examples can be found in the docstrings and [the unit tests](../unpythonic/tests/test_numutil.py**. +We briefly introduce the functions below. More details and examples can be found in the docstrings and [the unit tests](../unpythonic/tests/test_numutil.py). **CAUTION** for anyone new to numerics: From cce01e3b7ebfa0c1db1eafc91b16cb403d0e7d5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 12:46:25 +0300 Subject: [PATCH 482/832] 0.15.0: improve memoize and curry docs --- doc/features.md | 242 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 178 insertions(+), 64 deletions(-) diff --git a/doc/features.md b/doc/features.md index 0d7cf8cf..c5c7f3cb 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1295,28 +1295,9 @@ Things missing from the standard library. ### Batteries for functools - - `memoize`: - - Caches also exceptions à la Racket. If the memoized function is called again with arguments with which it raised an exception the first time, the same exception instance is raised again. - - Works also on instance methods, with results cached separately for each instance. - - This is essentially because ``self`` is an argument, and custom classes have a default ``__hash__``. - - Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. - - For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)). + - `memoize`, with exception caching. - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - - `curry`, with some extra features: - - **Changed in v0.15.0.** `curry` supports both positional and named arguments, and binds arguments to function parameters like Python itself does. The call triggers when all parameters are bound, regardless of whether they were passed by position or by name, and at which step of the currying process they were passed. - - **Changed in v0.15.0.** `unpythonic`'s multiple-dispatch system (`@generic`, `@typed`) is supported. `curry` looks for an exact match first, then a match with extra args/kwargs, and finally a partial match. If there is still no match, this implies that at least one parameter would get a binding that fails the type check. In such a case `TypeError` regarding failed multiple dispatch is raised. - - **Changed in v0.15.0.** If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`. - - Passthrough for args/kwargs that are incompatible with the target function's call signature (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket). - - Here *incompatible* means too many positional args, or named args that have no corresponding parameter. Note that if the function has a `**kwargs` parameter, then all named args are considered compatible, because it absorbs anything. - - Multiple return values (both positional and named) are denoted using `Values` (which see). A standard return value is considered to consist of one positional return value only. - - Positional args are passed through **on the right**. Any positional return values of the curried function are prepended, on the left. - - If the first positional return value of an intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining args and kwargs, after merging the rest of the return values into the args and kwargs. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). - - If more args/kwargs are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. - - To override, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. - - To set the dynvar, `from unpythonic import dyn`, and then `with dyn.let(curry_context=...):`. - - Can be used both as a decorator and as a regular function. - - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. - - **Caution**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python you have (and whether CPython or PyPy3). + - `curry`, with passthrough like in Haskell. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* - Any number of positional and keyword arguments are supported, with the same rules as in the pipe system. Multiple return values, or named return values, represented as a `Values`, are automatically unpacked to the args and kwargs of the next function in the chain. @@ -1335,36 +1316,14 @@ Things missing from the standard library. - `identity`, `const` which sometimes come in handy when programming with higher-order functions. - `fix`: detect and break infinite recursion cycles. **Added in v0.14.2.** -Examples (see also the next section): +We will discuss `memoize` and `curry` in more detail shortly; first, we will give some examples of the other utilities. Note that as always, more examples can be found in [the unit tests](../unpythonic/tests/test_fun.py). ```python -from operator import add, mul from typing import NoReturn -from unpythonic import (memoize, fix, andf, orf, flatmap, rotate, curry, dyn, - zipr, rzip, foldl, foldr, composer, to1st, cons, nil, ll, +from unpythonic import (fix, andf, orf, rotate, + zipr, rzip, foldl, foldr, withself) -# memoize: cache the results of pure functions (arguments must be hashable) -ncalls = 0 -@memoize # <-- important part -def square(x): - global ncalls - ncalls += 1 - return x**2 -assert square(2) == 4 -assert ncalls == 1 -assert square(3) == 9 -assert ncalls == 2 -assert square(3) == 9 -assert ncalls == 2 # called only once for each unique set of arguments -assert square(x=3) == 9 -assert ncalls == 2 # only the resulting bindings matter, not how you pass the args - - # "memoize lambda": classic evaluate-at-most-once thunk -thunk = memoize(lambda: print("hi from thunk")) -thunk() # the message is printed only the first time -thunk() - # detect and break infinite recursion cycles: # a(0) -> b(1) -> a(2) -> b(0) -> a(1) -> b(2) -> a(0) -> ... @fix() @@ -1375,6 +1334,7 @@ def b(k): return a((k + 1) % 3) assert a(0) is NoReturn # the call does return, saying the original function wouldn't. +# andf, orf: short-circuiting predicate combinators isint = lambda x: isinstance(x, int) iseven = lambda x: x % 2 == 0 isstr = lambda s: isinstance(s, str) @@ -1400,12 +1360,151 @@ assert myzipr((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) # zip and reverse don't commute for inputs with different lengths assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) # zip first assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # reverse first +``` -# curry with passthrough (positionals passed through on the right) -# final result is a tuple of the result(s) and the leftover args -double = lambda x: 2 * x -with dyn.let(curry_context=["whatever"]): # set a context to allow passthrough to the top level - assert curry(double, 2, "foo") == (4, "foo") # arity of double is 1 + +#### ``memoize`` + +The ``memoize`` decorator is meant for use with [pure functions](https://en.wikipedia.org/wiki/Pure_function). It caches the return value, so that *for each unique set of arguments*, the original function will be evaluated only once. All arguments must be hashable. + +Our ``memoize`` caches also exceptions, à la the [Mischief package in Racket](https://docs.racket-lang.org/mischief/memoize.html). If the memoized function is called again with arguments with which it raised an exception the first time, **that same exception instance** is raised again. + +The decorator **works also on instance methods**, with results cached separately for each instance. This is essentially because ``self`` is an argument, and custom classes have a default ``__hash__``. Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. (This approach does have the expected problem: if lots of instances are created and destroyed, and a memoized method is called for each, the memo will grow without bound.) + +*For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)).* + +There are some **important differences** to the nearest equivalents in the standard library, [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache) (Python 3.9+) and [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache): + + - `memoize` **binds arguments** like Python itself does, so given this definition: + + ```python + from unpythonic import memoize + + @memoize + def f(a, b): + return a + b + ``` + + the calls `f(1, 2)`, `f(1, b=2)`, `f(a=1, b=2)`, and `f(b=2, a=1)` all hit **the same cache key**. + + As of Python 3.9, in `functools.lru_cache` this is not so; see the internal function `functools._make_key` in [`functools.py`](https://github.com/python/cpython/blob/main/Lib/functools.py), where the comments explicitly say so. + + - `memoize` **caches exceptions**, too. A pure function that crashed for some combination of arguments, if given the same inputs again, will just crash again with the same error, so there is no reason to run it again. + + - `memoize` has **no** maximum cache size or hit/miss statistics counting. + + - `memoize` does **not** have a `typed` mode to treat `42` and `42.0` as different keys to the memo. The function arguments are hashed, and both an `int` and an equal `float` happen to hash to the same value. + + What the `typed` mode of the standard library functions is doing is actually a form of dispatch. Hence, you can use `@generic` (which see), and `@memoize` each individual multimethod: + + ```python + from unpythonic import generic, memoize + + @generic + @memoize + def thrice(x: int): + return 3 * x + + @generic + @memoize + def thrice(x: float): + return 3.0 * x + ``` + + Without using ``@generic``, The essential idea is: + + ```python + from unpythonic import memoize + + def thrice(x): # the dispatcher + if isinstance(x, int): + return thrice_int(x) + elif isinstance(x, float): + return thrice_float(x) + raise TypeError(type(x)) + + @memoize + def thrice_int(x): + return 3 * x + + @memoize + def thrice_float(x): + return 3.0 * x + ``` + + Observe that we memoize **each implementation**, not the dispatcher. + + This solution keeps dispatching and memoization orthogonal. + +Examples: + +```python +from unpythonic import memoize + +ncalls = 0 +@memoize # <-- important part +def square(x): + global ncalls + ncalls += 1 + return x**2 +assert square(2) == 4 +assert ncalls == 1 +assert square(3) == 9 +assert ncalls == 2 +assert square(3) == 9 +assert ncalls == 2 # called only once for each unique set of arguments +assert square(x=3) == 9 +assert ncalls == 2 # only the resulting bindings matter, not how you pass the args + +# "memoize lambda": classic evaluate-at-most-once thunk +# See also the `lazy[]` macro. +thunk = memoize(lambda: print("hi from thunk")) +thunk() # the message is printed only the first time +thunk() +``` + + +#### `curry` + +**Changed in v0.15.0.** *`curry` supports both positional and named arguments, and binds arguments to function parameters like Python itself does. The call triggers when all parameters are bound, regardless of whether they were passed by position or by name, and at which step of the currying process they were passed.* + +*`unpythonic`'s multiple-dispatch system (`@generic`, `@typed`) is supported. `curry` looks for an exact match first, then a match with extra args/kwargs, and finally a partial match. If there is still no match, this implies that at least one parameter would get a binding that fails the type check. In such a case `TypeError` regarding failed multiple dispatch is raised.* + +*If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`.* + +[Currying](https://en.wikipedia.org/wiki/Currying) is a technique in functional programming, where a function that takes multiple arguments is converted to a sequence of nested one-argument functions, each one *specializing* (fixing the value of) the leftmost remaining positional parameter. + +Some languages, such as Haskell, curry all functions natively. In languages that do not, like Python or [Racket](https://docs.racket-lang.org/reference/procedures.html#%28def._%28%28lib._racket%2Ffunction..rkt%29._curry%29%29), when currying is implemented as a library function, this is often done as a form of [partial application](https://en.wikipedia.org/wiki/Partial_application), which is a subtly different concept, but encompasses the curried behavior as a special case. + +Our ``curry`` can be used both as a decorator and as a regular function. As a decorator, `curry` takes no decorator arguments. As a regular function, `curry` itself is curried à la Racket. If any args or kwargs are given (beside the function to be curried), they are the first step. This helps eliminate many parentheses. + +**CAUTION**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python is used (and whether it is CPython or PyPy3). + +Like Haskell, and [`spicy` for Racket](https://github.com/Technologicat/spicy), our `curry` supports *passthrough*; but we pass through **both positional and named arguments**. + +Any args and/or kwargs that are incompatible with the target function's call signature, are *passed through* in the sense that the function is called, and then its return value is merged with the remaining args and kwargs. + +If the *first positional return value* of the result of passthrough is callable, it is (curried and) invoked on the remaining args and kwargs, after the merging. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). + +Some finer points concerning the passthrough feature: + + - *Incompatible* means too many positional args, or named args that have no corresponding parameter. Note that if the function has a `**kwargs` parameter, then all named args are considered compatible, because it absorbs anything. + + - Multiple return values (both positional and named) are denoted using `Values` (which see). A standard return value is considered to consist of *one positional return value* only (even if it is a `tuple`). + + - Extra positional args are passed through **on the right**. Any positional return values of the curried function are prepended, on the left. + + - Extra named args are passed through by name. They may be overridden by named return values (with the same name) from the curried function. + + - If more args/kwargs are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. + - To override this behavior, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. + - To set the dynvar, `from unpythonic import dyn`, and then `with dyn.let(curry_context=["whatever"]):`. + +Examples: + +```python +from operator import add, mul +from unpythonic import curry, foldl, foldr, composer, to1st, cons, nil, ll, dyn, Values mysum = curry(foldl, add, 0) myprod = curry(foldl, mul, 1) @@ -1417,6 +1516,15 @@ append_many = lambda *lsts: foldr(append_two, nil, lsts) # see unpythonic.lappe assert mysum(append_many(a, b, c)) == 21 assert myprod(b) == 12 +# curry with passthrough +double = lambda x: 2 * x +with dyn.let(curry_context=["whatever"]): # set a context to allow passthrough to the top level + # positionals are passed through on the right + assert curry(double, 2, "foo") == Values(4, "foo") # arity of double is 1 + # named args are passed through by name + assert curry(double, 2, nosucharg="foo") == Values(4, nosucharg="foo") + +# actual use case for passthrough map_one = lambda f: curry(foldr, composer(cons, to1st(f)), nil) doubler = map_one(double) assert doubler((1, 2, 3)) == ll(2, 4, 6) @@ -1424,9 +1532,11 @@ assert doubler((1, 2, 3)) == ll(2, 4, 6) assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) ``` -*Minor detail*: We could also write the last example as: +We could also write the last example as: ```python +from unpythonic import curry, foldl, composer, const, to1st, nil, lreverse + double = lambda x: 2 * x rmap_one = lambda f: curry(foldl, composer(cons, to1st(f)), nil) # essentially reversed(map(...)) map_one = lambda f: composer(rmap_one(f), lreverse) @@ -1435,33 +1545,37 @@ assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) which may be a useful pattern for lengthy iterables that could overflow the call stack (although not in ``foldr``, since our implementation uses a linear process). -In ``rmap_one``, we can use either ``curry`` or ``functools.partial``. In this case it doesn't matter which, since we want just one partial application anyway. We provide two arguments, and the minimum arity of ``foldl`` is 3, so ``curry`` will trigger the call as soon as (and only as soon as) it gets at least one more argument. +In the example, in ``rmap_one``, we can use either ``curry`` or ``partial``. In this case it does not matter which, since we want just one partial application anyway. We provide two arguments, and the minimum arity of ``foldl`` is 3, so ``curry`` will trigger the call as soon as (and only as soon as) it gets at least one more argument. -The final ``curry`` uses both of the extra features. It invokes passthrough, since ``map_one`` has arity 1. It also invokes a call to the callable returned from ``map_one``, with the remaining arguments (in this case just one, the ``ll(1, 2, 3)``). +The final ``curry`` in the example uses the passthrough features. The function ``map_one`` has arity 1, but two positional arguments are given. It also invokes a call to the callable returned by ``map_one``, with the remaining arguments (in this case just one, the ``ll(1, 2, 3)``). Yet another way to write ``map_one`` is: ```python +from unpythonic import curry, foldr, composer, cons, nil + mymap = lambda f: curry(foldr, composer(cons, curry(f)), nil) ``` The curried ``f`` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; these two values then end up as the arguments to ``cons``. -Using a currying compose function (name suffixed with ``c``), the inner curry can be dropped: +Using a **currying compose function** (name suffixed with ``c``), we can drop the inner curry: ```python +from unpythonic import curry, foldr, composerc, cons, nil + mymap = lambda f: curry(foldr, composerc(cons, f), nil) myadd = lambda a, b: a + b assert curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9) ``` -This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy)) as we're gonna get in Python. +This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy)) as we're gonna get in pure Python. Notice how the last two versions accept multiple input iterables; this is thanks to currying ``f`` inside the composition. An element from each of the iterables is taken by the processing function ``f``. Being the last argument, ``acc`` is passed through on the right. The output from the processing function - one new item - and ``acc`` then become two arguments, passed into cons. -Finally, keep in mind this exercise is intended as a feature demonstration. In production code, the builtin ``map`` is much better. It produces a lazy iterable, and does not care which kind of actual data structure the items will be stored in (once computed). +Finally, keep in mind the `mymap` example is intended as a feature demonstration. In production code, the builtin ``map`` is much better. It produces a lazy iterable, so it does not care which kind of actual data structure the items will be stored in (once they are computed). In other words, a lazy iterable is a much better model for a process that produces a sequence of values; how, and whether, to store that sequence is an orthogonal concern. -The example we have here evaluates all items immediately, and specifically produces a linked list. It's just a nice example of function composition involving incompatible arities, thus demonstrating the kind of situation where the passthrough feature of `curry` is useful. It is taken from a paper by [John Hughes (1984)](https://www.cse.chalmers.se/~rjmh/Papers/whyfp.html). +The example we have here evaluates all items immediately, and specifically produces a linked list. It is just a nice example of function composition involving incompatible positional arities, thus demonstrating the kind of situation where the passthrough feature of `curry` is useful. It is taken from a paper by [John Hughes (1984)](https://www.cse.chalmers.se/~rjmh/Papers/whyfp.html). #### ``curry`` and reduction rules @@ -1494,7 +1608,7 @@ As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the - Note we keep track of which arguments were passed positionally and which by name. To avoid subtle errors, they are eventually passed to `f` the same way they were passed to `curry`. (Positional args are passed positionally, and kwargs are passed by name.) - If there are no unbound parameters, and no args/kwargs are left over, we have an exact match. Call `f` and return its result, like a normal function call. - Any sequence of curried calls that ends up binding all parameters of `f` triggers the call. - - As before, beware when working with variadic functions. Particularly, keep in mind that `*args` matches **zero or more** positional arguments (as the [Kleene star](https://en.wikipedia.org/wiki/Kleene_star)-ish notation indeed suggests). + - Beware when working with variadic functions. Particularly, keep in mind that `*args` matches **zero or more** positional arguments (as the [Kleene star](https://en.wikipedia.org/wiki/Kleene_star)-ish notation indeed suggests). - If there are no unbound parameters, but there are args/kwargs left over, arrange passthrough for the leftover args/kwargs (that were rejected by the call signature of `f`), and call `f`. Any leftover positional arguments are passed through **on the right**. - Merge the return value of `f` with the leftover args/kwargs, thus forming updated leftover args/kwargs. - If the return value of `f` is a `Values`: prepend positional return values into the leftover args (i.e. insert them **on the left**), and update the leftover kwargs with the named return values. (I.e. a key name conflict causes an overwrite in the leftover kwargs.) @@ -1507,9 +1621,9 @@ As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the - First, try for an exact match that passes the type check. **If any such match is found**, pick that multimethod. Call it and return its result (as above). - Then, try for a match that passes the type check, but has extra args/kwargs. **If any such match is found**, pick that multimethod. Arrange passthrough... (as above). - Then, try for a partial match that passes the type check. **If any such match is found**, keep currying. - - If none of the above match, it implies that no matter which multimethod we pick, at least one parameter would get a binding that fails the type check. Raise `TypeError`. + - If none of the above match, it implies that no matter which multimethod we pick, at least one parameter will get a binding that fails the type check. Raise `TypeError`. -(If *really* interested in the gritty details, look at the source code of `unpythonic.fun.curry`. It calls some functions from `unpythonic.dispatch` for its `@generic` support, but otherwise it's pretty much self-contained.) +If interested in the gritty details, see [the source code](../unpythonic/fun.py) of `unpythonic.fun.curry`. It calls some functions from `unpythonic.dispatch` for its `@generic` support, but otherwise it is pretty much self-contained. Getting back to the simple case, in the above example: @@ -1517,13 +1631,13 @@ Getting back to the simple case, in the above example: curry(mapl_one, double, ll(1, 2, 3)) ``` -the callable ``mapl_one`` takes one argument, which is a function. It yields another function, let us call it ``g``. We are left with: +the callable ``mapl_one`` takes one argument, which is a function. It returns another function, let us call it ``g``. We are left with: ```python curry(g, ll(1, 2, 3)) ``` -The argument is then passed into ``g``; we obtain a result, and reduction is complete. +The remaining argument is then passed into ``g``; we obtain a result, and reduction is complete. A curried function is also a curry context: @@ -1533,7 +1647,7 @@ a2 = curry(add2) a2(a, b, c) # same as curry(add2, a, b, c); reduces to (a + b, c) ``` -so on the last line, we don't need to say +so on the last line, we do not need to say ```python curry(a2, a, b, c) From a4670684c5f3b107793af4aa866f5d2d6a8d7008 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 12:49:59 +0300 Subject: [PATCH 483/832] wording fixes --- doc/features.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index c5c7f3cb..52441862 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1395,7 +1395,7 @@ There are some **important differences** to the nearest equivalents in the stand - `memoize` does **not** have a `typed` mode to treat `42` and `42.0` as different keys to the memo. The function arguments are hashed, and both an `int` and an equal `float` happen to hash to the same value. - What the `typed` mode of the standard library functions is doing is actually a form of dispatch. Hence, you can use `@generic` (which see), and `@memoize` each individual multimethod: + The `typed` mode of the standard library functions is actually a form of dispatch. Hence, you can use `@generic` (which see), and `@memoize` each individual multimethod: ```python from unpythonic import generic, memoize @@ -1411,7 +1411,7 @@ There are some **important differences** to the nearest equivalents in the stand return 3.0 * x ``` - Without using ``@generic``, The essential idea is: + Without using ``@generic``, the essential idea is: ```python from unpythonic import memoize From 05394461759f54cb741ee8db2f16445b2bc13c51 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 12:51:42 +0300 Subject: [PATCH 484/832] add TOC links --- doc/features.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/features.md b/doc/features.md index 52441862..cb4c21d0 100644 --- a/doc/features.md +++ b/doc/features.md @@ -37,9 +37,14 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``begin``: sequence side effects](#begin-sequence-side-effects) - [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) **[M]** - [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) + - [``pipe``](#pipe) + - [``piped``](#piped) + - [``lazy_piped``](#lazy_piped) [**Batteries**](#batteries) missing from the standard library. - [**Batteries for functools**](#batteries-for-functools): `memoize`, `curry`, `compose`, `withself`, `fix` and more. + - [``memoize``](#memoize): a detailed explanation of the memoizer. + - [``curry``](#curry): a detailed explanation of the curry utility. - [``curry`` and reduction rules](#curry-and-reduction-rules): we provide some extra features for bonus Haskellness. - [``fix``: break infinite recursion cycles](#fix-break-infinite-recursion-cycles) - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. From 57b8e3f96b91a9f86e34ef5f4e6122207dc24067 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 12:53:27 +0300 Subject: [PATCH 485/832] add more missing TOC links Maybe we need to migrate also the main docs to the use a generated TOC. --- doc/features.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/features.md b/doc/features.md index cb4c21d0..8e520d35 100644 --- a/doc/features.md +++ b/doc/features.md @@ -30,6 +30,9 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``frozendict``: an immutable dictionary](#frozendict-an-immutable-dictionary) - [`cons` and friends: pythonic lispy linked lists](#cons-and-friends-pythonic-lispy-linked-lists) - [``box``: a mutable single-item container](#box-a-mutable-single-item-container) + - [``box``](#box) + - [``Some``](#some): immutable box, to explicitly indicate the presence of a value. + - [``ThreadLocalBox``](#threadlocalbox) - [``Shim``: redirect attribute accesses](#shim-redirect-attribute-accesses) - [Container utilities](#container-utilities): ``get_abcs``, ``in_slice``, ``index_in_slice`` From 2e6bb5555d4210576f25b4c8e2edfa2b2824adb4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 12:58:41 +0300 Subject: [PATCH 486/832] remove duplicate note --- doc/features.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 8e520d35..be24e830 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1588,10 +1588,6 @@ The example we have here evaluates all items immediately, and specifically produ #### ``curry`` and reduction rules -**Changed in v0.15.0.** *`curry` now supports kwargs, too, and binds parameters like Python itself does. Also, `@generic` and `@typed` functions are supported.* - -*For advanced examples, see [the unit tests](../unpythonic/tests/test_fun.py).* - Our ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. Let's consider a simple example with positional arguments only. When we say: ```python From bcadb50f512d7aabcfced86d2b994a531b2bc7bf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:02:14 +0300 Subject: [PATCH 487/832] add yet more missing links --- doc/features.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index be24e830..6f4d4ee2 100644 --- a/doc/features.md +++ b/doc/features.md @@ -21,6 +21,9 @@ The exception are the features marked **[M]**, which are primarily intended as a [**Bindings**](#bindings) - [``let``, ``letrec``: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) **[M]** + - [``let``](#let) + - [``dlet``, ``blet``](#dlet-blet): *let-over-def*, like the classic let-over-lambda. + - [``letrec``](#letrec) - [Lispylet: alternative syntax](#lispylet-alternative-syntax) **[M]** - [``env``: the environment](#env-the-environment) - [``assignonce``](#assignonce), a relative of ``env``. @@ -1687,7 +1690,7 @@ because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by desi - For run-time type checking, consider `@typed` or `@generic` right here in `unpythonic`. -- You can also just use Python's type annotations; `unpythonic`'s `curry` type-checks the arguments before accepting the curried function. The annotations work if the stdlib function `typing.get_type_hints` can find them. +- You can also just use Python's type annotations; `unpythonic`'s `curry` type-checks the arguments before accepting the curried function. The annotations work if the stdlib function [`typing.get_type_hints`](https://docs.python.org/3/library/typing.html#typing.get_type_hints) can find them. #### ``fix``: break infinite recursion cycles From c7cb73015cc4951e42c5f52d07175ea017023f0e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:08:40 +0300 Subject: [PATCH 488/832] improve letrec doc --- doc/features.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/features.md b/doc/features.md index 6f4d4ee2..69367e27 100644 --- a/doc/features.md +++ b/doc/features.md @@ -209,7 +209,9 @@ The ``@blet`` decorator is otherwise the same as ``@dlet``, but instead of decor #### ``letrec`` -In `letrec`, bindings may depend on ones above them in the same `letrec`, by using `lambda e: ...`: +The name of this construct comes from the Scheme family of Lisps, and stands for *let (mutually) recursive*. The "[mutually recursive](https://en.wikipedia.org/wiki/Mutual_recursion)" refers to the kind of scoping between the bindings in the same `letrec`. + +In plain English, in `letrec`, bindings may depend on ones above them in the same `letrec`. The raw API in `unpythonic` uses a `lambda e: ...` to provide the environment: ```python from unpythonic import letrec @@ -233,11 +235,11 @@ x = letrec[[a << 1, b] ``` -In the non-macro `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent any misunderstandings in the environment initialization procedure. +In the non-macro `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent misinterpretation by the machinery when the environment initialization procedure runs. In a non-callable ``valexpr``, trying to depend on a binding below it raises ``AttributeError``. -A callable ``valexpr`` may depend on any bindings (also later ones) in the same `letrec`. For example, here is a pair of mutually recursive functions: +A callable ``valexpr`` may depend on any bindings (**also later ones**) in the same `letrec`. For example, here is a pair of [mutually recursive](https://en.wikipedia.org/wiki/Mutual_recursion) functions: ```python from unpythonic import letrec From 245d98f3b7d1fa155c45123da335318cfb72c468 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:09:33 +0300 Subject: [PATCH 489/832] fix incorrect statement --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 69367e27..04ebdf64 100644 --- a/doc/features.md +++ b/doc/features.md @@ -211,7 +211,7 @@ The ``@blet`` decorator is otherwise the same as ``@dlet``, but instead of decor The name of this construct comes from the Scheme family of Lisps, and stands for *let (mutually) recursive*. The "[mutually recursive](https://en.wikipedia.org/wiki/Mutual_recursion)" refers to the kind of scoping between the bindings in the same `letrec`. -In plain English, in `letrec`, bindings may depend on ones above them in the same `letrec`. The raw API in `unpythonic` uses a `lambda e: ...` to provide the environment: +In plain English, in `letrec`, the value of a binding may depend on other bindings in the same `letrec`. The raw API in `unpythonic` uses a `lambda e: ...` to provide the environment: ```python from unpythonic import letrec From 83dd96342ebb4ba27ab437a8fe3e7248665f8925 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:11:28 +0300 Subject: [PATCH 490/832] styling --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 04ebdf64..d304dfab 100644 --- a/doc/features.md +++ b/doc/features.md @@ -300,7 +300,7 @@ The decorators ``@dletrec`` and ``@bletrec`` work otherwise exactly like ``@dlet #### Lispylet: alternative syntax -**NOTE**: This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use. Below is the documentation for the raw API. +**NOTE**: *This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use. Below is the documentation for the raw API.* The `lispylet` module was originally created to allow guaranteed left-to-right initialization of `letrec` bindings in Pythons older than 3.6, hence the positional syntax and more parentheses. The only difference is the syntax; the behavior is identical with the other implementation. As of 0.15, the main role of `lispylet` is to act as the run-time backend for the `let` family of macros. From 3c15d16c5f346d0bd5330db193ee23e657a598a7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:14:59 +0300 Subject: [PATCH 491/832] fix borkage; combine paragraphs --- doc/features.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/doc/features.md b/doc/features.md index d304dfab..1c735d90 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1062,9 +1062,7 @@ y = do[local[x << 5], assert y == 25 ``` -*In the macro version, all items are delayed automatically; that is, **every** item has an implicit ``lambda e: ...``.* - -*Note that instead of the `assign` function, the macro version uses the syntax ``local[name << value]`` to **create** an expression-local variable. Updating an existing variable in the `do` environment is just ``name << value``. Finally, there is also ``delete[name]`.* +*In the macro version, all items are delayed automatically; that is, **every** item has an implicit ``lambda e: ...``. Note that instead of the `assign` function, the macro version uses the syntax ``local[name << value]`` to **create** an expression-local variable. Updating an existing variable in the `do` environment is just ``name << value``. Finally, there is also ``delete[name]``.* When using the raw API, beware of this pitfall: From 631bf622aefb4e1762d3b84d0fa3d450d5d3472c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:15:08 +0300 Subject: [PATCH 492/832] add some missing TOC links --- doc/features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/features.md b/doc/features.md index 1c735d90..c3328b47 100644 --- a/doc/features.md +++ b/doc/features.md @@ -42,6 +42,8 @@ The exception are the features marked **[M]**, which are primarily intended as a [**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a ``lambda``). - [``begin``: sequence side effects](#begin-sequence-side-effects) - [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) **[M]** + - [``do``](#do) + - [``do0``](#do0) - [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) - [``pipe``](#pipe) - [``piped``](#piped) From 29353979f63cb5296fd75893d7a74c3e39259880 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:16:54 +0300 Subject: [PATCH 493/832] ordering --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index c3328b47..a5948c0d 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1309,8 +1309,8 @@ Things missing from the standard library. ### Batteries for functools - `memoize`, with exception caching. - - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `curry`, with passthrough like in Haskell. + - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* - Any number of positional and keyword arguments are supported, with the same rules as in the pipe system. Multiple return values, or named return values, represented as a `Values`, are automatically unpacked to the args and kwargs of the next function in the chain. From 8a670fae6e26ead75c0c8a9b9c9af49ab406bdf1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:23:44 +0300 Subject: [PATCH 494/832] ordering --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index a5948c0d..4bfc169c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1310,6 +1310,7 @@ Things missing from the standard library. - `memoize`, with exception caching. - `curry`, with passthrough like in Haskell. + - `fix`: detect and break infinite recursion cycles. **Added in v0.14.2.** - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* @@ -1327,7 +1328,6 @@ Things missing from the standard library. - `rotate`: a cousin of `flip`. Permute the order of positional arguments in a cycle. - `to1st`, `to2nd`, `tokth`, `tolast`, `to` to help inserting 1-in-1-out functions into m-in-n-out compose chains. (Currying can eliminate the need for these.) - `identity`, `const` which sometimes come in handy when programming with higher-order functions. - - `fix`: detect and break infinite recursion cycles. **Added in v0.14.2.** We will discuss `memoize` and `curry` in more detail shortly; first, we will give some examples of the other utilities. Note that as always, more examples can be found in [the unit tests](../unpythonic/tests/test_fun.py). From c658f652f6444eddc10cbeefa30774d2b204ac13 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 13:55:29 +0300 Subject: [PATCH 495/832] 0.15.0: improve batteries for functools docs --- doc/features.md | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/doc/features.md b/doc/features.md index 4bfc169c..0de2f57c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -50,10 +50,9 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``lazy_piped``](#lazy_piped) [**Batteries**](#batteries) missing from the standard library. -- [**Batteries for functools**](#batteries-for-functools): `memoize`, `curry`, `compose`, `withself`, `fix` and more. +- [**Batteries for functools**](#batteries-for-functools): `curry`, `compose`, `withself`, and more. - [``memoize``](#memoize): a detailed explanation of the memoizer. - - [``curry``](#curry): a detailed explanation of the curry utility. - - [``curry`` and reduction rules](#curry-and-reduction-rules): we provide some extra features for bonus Haskellness. + - [``curry``](#curry): a detailed explanation of the curry utility and its haskelly extra features. - [``fix``: break infinite recursion cycles](#fix-break-infinite-recursion-cycles) - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. - [**Batteries for network programming**](#batteries-for-network-programming): message protocol, PTY/socket proxy, etc. @@ -1329,7 +1328,7 @@ Things missing from the standard library. - `to1st`, `to2nd`, `tokth`, `tolast`, `to` to help inserting 1-in-1-out functions into m-in-n-out compose chains. (Currying can eliminate the need for these.) - `identity`, `const` which sometimes come in handy when programming with higher-order functions. -We will discuss `memoize` and `curry` in more detail shortly; first, we will give some examples of the other utilities. Note that as always, more examples can be found in [the unit tests](../unpythonic/tests/test_fun.py). +We will discuss `memoize`, `curry` and `fix` in more detail shortly; but first, we will give some examples of the other utilities. Note that as always, more examples can be found in [the unit tests](../unpythonic/tests/test_fun.py). ```python from typing import NoReturn @@ -1378,7 +1377,7 @@ assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # re #### ``memoize`` -The ``memoize`` decorator is meant for use with [pure functions](https://en.wikipedia.org/wiki/Pure_function). It caches the return value, so that *for each unique set of arguments*, the original function will be evaluated only once. All arguments must be hashable. +[*Memoization*](https://en.wikipedia.org/wiki/Memoization) is a functional programming technique, meant to be used with [pure functions](https://en.wikipedia.org/wiki/Pure_function). It caches the return value, so that *for each unique set of arguments*, the original function will be evaluated only once. All arguments must be hashable. Our ``memoize`` caches also exceptions, à la the [Mischief package in Racket](https://docs.racket-lang.org/mischief/memoize.html). If the memoized function is called again with arguments with which it raised an exception the first time, **that same exception instance** is raised again. @@ -1485,7 +1484,7 @@ thunk() *If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`.* -[Currying](https://en.wikipedia.org/wiki/Currying) is a technique in functional programming, where a function that takes multiple arguments is converted to a sequence of nested one-argument functions, each one *specializing* (fixing the value of) the leftmost remaining positional parameter. +[*Currying*](https://en.wikipedia.org/wiki/Currying) is a technique in functional programming, where a function that takes multiple arguments is converted to a sequence of nested one-argument functions, each one *specializing* (fixing the value of) the leftmost remaining positional parameter. Some languages, such as Haskell, curry all functions natively. In languages that do not, like Python or [Racket](https://docs.racket-lang.org/reference/procedures.html#%28def._%28%28lib._racket%2Ffunction..rkt%29._curry%29%29), when currying is implemented as a library function, this is often done as a form of [partial application](https://en.wikipedia.org/wiki/Partial_application), which is a subtly different concept, but encompasses the curried behavior as a special case. @@ -1591,7 +1590,7 @@ Finally, keep in mind the `mymap` example is intended as a feature demonstration The example we have here evaluates all items immediately, and specifically produces a linked list. It is just a nice example of function composition involving incompatible positional arities, thus demonstrating the kind of situation where the passthrough feature of `curry` is useful. It is taken from a paper by [John Hughes (1984)](https://www.cse.chalmers.se/~rjmh/Papers/whyfp.html). -#### ``curry`` and reduction rules +##### ``curry`` and reduction rules Our ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. Let's consider a simple example with positional arguments only. When we say: @@ -1697,10 +1696,14 @@ because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by desi #### ``fix``: break infinite recursion cycles -The name `fix` comes from the *least fixed point* with respect to the definedness relation, which is related to Haskell's `fix` function. However, this `fix` is not that function. Our `fix` breaks recursion cycles in strict functions - thus causing some non-terminating strict functions to return. (Here *strict* means that the arguments are evaluated eagerly.) +The name `fix` comes from the *least fixed point* with respect to the definedness relation, which is related to Haskell's `fix` function. However, this `fix` is **not** that function. Our `fix` breaks recursion cycles in strict functions - thus causing some non-terminating strict functions to return. (Here [*strict*](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) means that the arguments are evaluated eagerly.) **CAUTION**: Worded differently, this function solves a small subset of the halting problem. This should be hint enough that it will only work for the advertised class of special cases - i.e., a specific kind of recursion cycles. +If you need `fix` for code that uses TCO, use `fixtco`. The implementations of recursion cycle breaking and TCO must interact in a very particular way to work properly; this is done by `fixtco`. + +For examples, see [the unit tests](../unpythonic/tests/test_fix.py). + Usage: ```python @@ -1723,11 +1726,11 @@ If no recursion cycle occurs, `f` returns normally. If a cycle occurs, the call - In the latter example, the name `"f"` and the offending args are returned. -**A cycle is detected when** `f` is called again with a set of args that have already been previously seen in the current call chain. Infinite mutual recursion is detected too, at the point where any `@fix`-instrumented function is entered again with a set of args already seen during the current call chain. +**A cycle is detected when** `f` is called again with a set of args that have already been previously seen in the current call chain. Infinite *mutual recursion* is detected too, at the point where any `@fix`-instrumented function is entered again with a set of args already seen during the current call chain. -**CAUTION**: The infinitely recursive call sequence `f(0) → f(1) → ... → f(k+1) → ...` contains no cycles in the sense detected by `fix`. The `fix` function will not catch all cases of infinite recursion, but only those where a previously seen set of arguments is seen again. (If `f` is pure, the same arguments appearing again implies the call will not return, so we can terminate it.) +**CAUTION**: The infinitely recursive call sequence `f(0) → f(1) → ... → f(k+1) → ...` contains no cycles in the sense detected by `fix`. The `fix` function will **not** catch all cases of infinite recursion, but only those where a previously seen set of arguments is seen again. If `f` is [pure](https://en.wikipedia.org/wiki/Pure_function), the same arguments appearing again during recursion implies the call will not return, so we can terminate it. -**CAUTION**: If we have a function `g(a, b)`, the argument lists of the invocations `g(1, 2)` and `g(a=1, b=2)` are in principle different. This is a Python gotcha that was originally noticed by the author of the `wrapt` library, and mentioned in [its documentation](https://wrapt.readthedocs.io/en/latest/decorators.html#processing-function-arguments). However, once arguments are bound to the formal parameters of `g`, the result is the same. We consider the *resulting bindings*, not the exact way the arguments were passed. +**CAUTION**: If we have a function `g(a, b)`, the argument lists of the invocations `g(1, 2)` and `g(a=1, b=2)` are in principle different. However, we bind arguments like Python itself does, and consider the *resulting bindings* only. It does not matter how the arguments were passed. We can use `fix` to find the (arithmetic) fixed point of `cos`: @@ -1772,7 +1775,7 @@ c = fixpoint(cos, x0=1) assert c == cos(c) ``` -**NOTE**: But see `unpythonic.fixpoint`, which is meant specifically for finding *arithmetic* fixed points, and `unpythonic.iterate1`, which produces a generator that iterates `f` without needing recursion. +**NOTE**: *See `unpythonic.fixpoint`, which is meant specifically for finding arithmetic fixed points, and `unpythonic.iterate1`, which produces a generator that iterates `f` without needing recursion.* **Notes**: @@ -1792,15 +1795,15 @@ assert c == cos(c) - `bottom` can be a callable, in which case the function name and args at the point where the cycle was detected are passed to it, and its return value becomes the final return value. This is useful e.g. for debug logging. - - The `memo` flag controls whether to memoize also intermediate results. It adds some additional function call layers between function entries from recursive calls; if that is a problem (due to causing Python's call stack to blow up faster), use `memo=False`. You can still memoize the final result if you want; just put `@memoize` on the outside. + The function name is provided, because we catch also infinite *mutual recursion*; so it can be a useful piece of information *which function* it was that was first called with already-seen arguments. -**NOTE**: If you need `fix` for code that uses TCO, use `fixtco` instead. The implementations of recursion cycle breaking and TCO must interact in a very particular way to work properly; this is done by `fixtco`. + - The `memo` flag controls whether to memoize intermediate results. It adds some additional function call layers between function entries from recursive calls; if that is a problem (due to causing Python's call stack to blow up faster), use `memo=False`. You can still memoize the final result if you want; just put `@memoize` on the outside. ##### Real-world use and historical note This kind of `fix` is sometimes helpful in recursive pattern-matching definitions for parsers. When the pattern matcher gets stuck in an infinite left-recursion, it can return a customizable special value instead of not terminating. Being able to not care about non-termination may simplify definitions. -This `fix` can also be used to find fixed points of functions, as in the above examples. +This `fix` can also be used to find arithmetic fixed points of functions, as in the above examples. The idea comes from Matthew Might's article on [parsing with (Brzozowski's) derivatives](http://matt.might.net/articles/parsing-with-derivatives/), where it was a utility implemented in Racket as the `define/fix` form. It was originally ported to Python [by Per Vognsen](https://gist.github.com/pervognsen/8dafe21038f3b513693e) (linked from the article). The `fix` in `unpythonic` is a redesign with kwargs support, thread safety, and TCO support. From 8c6f7cb4db1d743eb34ff62f552f7a358a9df65c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 14:12:35 +0300 Subject: [PATCH 496/832] add link --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 0de2f57c..aa9f78cf 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1819,7 +1819,7 @@ A simple way to explain Haskell's `fix` is: fix f = let x = f x in x ``` -so anywhere the argument is referred to in the definition of `f`, it is replaced by another application of `f`, recursively. This obviously yields a notation useful for corecursively defining infinite lazy lists. +so anywhere the argument is referred to in the definition of `f`, it is replaced by another application of `f`, recursively. This obviously yields a notation useful for [corecursively](https://en.wikipedia.org/wiki/Corecursion) defining infinite lazy lists. For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[2]](https://www.vex.net/~trebla/haskell/fix.xhtml) [[3]](https://stackoverflow.com/questions/4787421/how-do-i-use-fix-and-how-does-it-work) [[4]](https://medium.com/@cdsmithus/fixpoints-in-haskell-294096a9fc10) [[5]](https://en.wikibooks.org/wiki/Haskell/Fix_and_recursion). From 46fe654dfefdcde2650ef21b08b8884406d3f904 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 12 Jun 2021 14:13:53 +0300 Subject: [PATCH 497/832] wording --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index aa9f78cf..fd93d978 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1830,7 +1830,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - Return the first ``n`` items and the ``k``th tail, in a tuple. Default is ``k = n``. - Use ``k > n`` to fast-forward, consuming the skipped items. Works by `drop`. - Use ``k < n`` to peek without permanently extracting an item. Works by [tee](https://docs.python.org/3/library/itertools.html#itertools.tee)ing; plan accordingly. - - *folds, scans, unfold*: + - *fold, scan, unfold*: - `foldl`, `foldr` with support for multiple input iterables, like in Racket. - Like in Racket, `op(elt, acc)`; general case `op(e1, e2, ..., en, acc)`. Note Python's own `functools.reduce` uses the ordering `op(acc, elt)` instead. - No sane default for multi-input case, so the initial value for `acc` must be given. From 02715bfcc0abe2775215a3ca0ab92731156dce1a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 00:33:24 +0300 Subject: [PATCH 498/832] update `unfold` to use `Values` This is a breaking change, part of 0.15.0. --- CHANGELOG.md | 1 + README.md | 6 +++--- doc/dialects/lispython.md | 2 +- doc/features.md | 14 +++++++++----- unpythonic/fold.py | 20 +++++++++++++------- unpythonic/tests/test_fold.py | 9 ++++++--- unpythonic/tests/test_fpnumerics.py | 3 ++- 7 files changed, 35 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b2d2f32..8bbdb224 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -155,6 +155,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `curry` - `pipe` family - `compose` family + - `unfold` - All multiple-return-values in code using the `with continuations` macro. (The continuations system essentially composes continuation functions.) - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.fn` (note name change!), because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **API differences.** diff --git a/README.md b/README.md index ce516e20..1c47997c 100644 --- a/README.md +++ b/README.md @@ -152,7 +152,7 @@ Scan and fold accept multiple iterables, like in Racket. ```python from operator import add -from unpythonic import scanl, foldl, unfold, take +from unpythonic import scanl, foldl, unfold, take, Values assert tuple(scanl(add, 0, range(1, 5))) == (0, 1, 3, 6, 10) @@ -160,8 +160,8 @@ def op(e1, e2, acc): return acc + e1 * e2 assert foldl(op, 0, (1, 2), (3, 4)) == 11 -def nextfibo(a, b): # *oldstates - return (a, b, a + b) # value, *newstates +def nextfibo(a, b): + return Values(a, a=b, b=a + b) assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) ```
diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index f94f2a5c..1d0d2d56 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -122,7 +122,7 @@ In the `Lispython` variant, we implicitly import some macros and functions to se - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax``. - The underscore: e.g. `fn[_ * 3]` becomes `lambda x: x * 3`, and `fn[_ * _]` becomes `lambda x, y: x * y`. - ``dyn``, for dynamic assignment. - - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, the `pipe` family, the `compose` family, and the `with continuations` macro.) + - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, `unfold`, the `pipe` family, the `compose` family, and the `with continuations` macro.) For detailed documentation of the language features, see [``unpythonic.syntax``](../macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. diff --git a/doc/features.md b/doc/features.md index fd93d978..0c8eec92 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1848,8 +1848,9 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `rscanl`, `rscanl1` reverse each input and then left-scan. This syncs the **right** ends. - `unfold1`, `unfold`: generate a sequence [corecursively](https://en.wikipedia.org/wiki/Corecursion). The counterpart of `foldl`. - `unfold1` is for 1-in-2-out functions. The input is `state`, the return value must be `(value, newstate)` or `None`. - - `unfold` is for n-in-(1+n)-out functions. The input is `*states`, the return value must be `(value, *newstates)` or `None`. - - Unfold returns a generator yielding the collected values. The output can be finite or infinite; to signify that a finite sequence ends, the user function must return `None`. + - `unfold` is for n-in-(1+n)-out functions. + - **Changed in v0.15.0.** *The initial args/kwargs are unpacked to the args/kwargs of the user function. The function must return a `Values` object, where the first positional return value is the value to be yielded, and anything else is unpacked to the args/kwargs of the user function at the next iteration.* + - Unfold returns a generator yielding the collected values. The output can be finite or infinite; to signify that a finite sequence ends, the user function must return `None`. (Beside a `Values` object, a bare `None` is the only other allowed return value from the user function.) - *mapping and zipping*: - `map_longest`: the final missing battery for `map`. - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by ``itertools``. @@ -1914,7 +1915,8 @@ from unpythonic import (scanl, scanr, foldl, foldr, s, inn, iindex, window, subset, powerset, - allsame) + allsame, + Values) assert tuple(scanl(add, 0, range(1, 5))) == (0, 1, 3, 6, 10) assert tuple(scanr(add, 0, range(1, 5))) == (0, 4, 7, 9, 10) @@ -1929,7 +1931,9 @@ def step2(k): # x0, x0 + 2, x0 + 4, ... assert tuple(take(10, unfold1(step2, 10))) == (10, 12, 14, 16, 18, 20, 22, 24, 26, 28) def nextfibo(a, b): - return (a, b, a + b) # value, *newstates + # First positional is value; everything else is newstate, + # to be unpacked to `nextfibo`'s args/kwargs at the next iteration. + return Values(a, a=b, b=a + b) assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) def fibos(): @@ -4116,7 +4120,7 @@ Most of the time, returning a tuple to denote multiple-return-values and unpacki But the distinction is critically important in function composition, so that positional return values can be automatically mapped into positional arguments to the next function in the chain, and named return values into named arguments. -Accordingly, various parts of `unpythonic` that deal with function composition use the `Values` abstraction; particularly `curry`, and the `compose` and `pipe` families, and the `with continuations` macro. +Accordingly, various parts of `unpythonic` that deal with function composition use the `Values` abstraction; particularly `curry`, `unfold`, the `compose` and `pipe` families, and the `with continuations` macro. #### Behavior diff --git a/unpythonic/fold.py b/unpythonic/fold.py index 3f6a0cb1..1897f3aa 100644 --- a/unpythonic/fold.py +++ b/unpythonic/fold.py @@ -23,6 +23,7 @@ from operator import mul #from collections import deque +from .funutil import Values #from .it import first, last, rev from .it import last, rev @@ -297,29 +298,34 @@ def step2(k): # x0, x0 + 2, x0 + 4, ... value, state = result yield value -def unfold(proc, *inits): +def unfold(proc, *inits, **kwinits): """Like unfold1, but for n-in-(1+n)-out proc. The current state is unpacked to the argument list of ``proc``. - It must return either ``(value, *newstates)``, or ``None`` to signify - that the sequence ends. + It must return either a ``Values`` object where the first positional + return value is the ``value`` to be yielded at this iteration, and + anything else is state to be unpacked to the args/kwargs of ``proc`` + at the next iteration; or a bare ``None`` to signify that the sequence ends. If your state is something simple such as one number, see ``unfold1``. Example:: def fibo(a, b): - return (a, b, a + b) + return Values(a, a=b, b=a + b) assert (tuple(take(10, unfold(fibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55)) """ - states = inits + state = Values(*inits, **kwinits) while True: - result = proc(*states) + result = proc(*state.rets, **state.kwrets) if result is None: break - value, *states = result + if not isinstance(result, Values): + raise TypeError(f"Expected `None` (to terminate) or a `Values` (to continue), got {type(result)} with value {repr(result)}") + value, *rets = result.rets # unpack the first positional return value, keep the rest + state = Values(*rets, **result.kwrets) yield value # This is **not** how to make a right map; the result is exactly the same diff --git a/unpythonic/tests/test_fold.py b/unpythonic/tests/test_fold.py index 7e442960..e059005a 100644 --- a/unpythonic/tests/test_fold.py +++ b/unpythonic/tests/test_fold.py @@ -10,6 +10,7 @@ foldl, foldr, reducel, reducer, rreducel, rfoldl, unfold, unfold1, prod, running_minmax, minmax) from ..fun import curry, composer, composerc, composel, to1st, rotate +from ..funutil import Values from ..llist import cons, nil, ll, lreverse from ..it import take, tail @@ -182,15 +183,17 @@ def step2(k): # x0, x0 + 2, x0 + 4, ... return (k, k + 2) # (value, newstate) def fibo(a, b): - return (a, b, a + b) # (value, *newstates) + # First positional is value; everything else is newstate, + # to be unpacked to `fibo`'s args/kwargs at the next iteration. + return Values(a, a=b, b=a + b) def myiterate(f, x): # x0, f(x0), f(f(x0)), ... - return (x, f, f(x)) + return Values(x, f=f, x=f(x)) def zip_two(As, Bs): if len(As) and len(Bs): (A0, *moreAs), (B0, *moreBs) = As, Bs - return ((A0, B0), moreAs, moreBs) + return Values((A0, B0), As=moreAs, Bs=moreBs) test[tuple(take(10, unfold1(step2, 10))) == (10, 12, 14, 16, 18, 20, 22, 24, 26, 28)] test[tuple(take(10, unfold(fibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55)] diff --git a/unpythonic/tests/test_fpnumerics.py b/unpythonic/tests/test_fpnumerics.py index 4c530a07..a6751ae6 100644 --- a/unpythonic/tests/test_fpnumerics.py +++ b/unpythonic/tests/test_fpnumerics.py @@ -13,6 +13,7 @@ from math import sin, pi, log2 from ..fun import curry +from ..funutil import Values from ..it import unpack, drop, take, tail, first, second, last, iterate1, within from ..fold import scanl, scanl1, unfold from ..mathseq import gmathify, imathify @@ -132,7 +133,7 @@ def nats(start=0): @gmathify def fibos(): def nextfibo(a, b): - return a, b, a + b + return Values(a, a=b, b=a + b) return unfold(nextfibo, 1, 1) @gmathify def pows(): From 1b02a58662857d7d1b0ce60bbb723bd8dfcca4a0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 00:34:41 +0300 Subject: [PATCH 499/832] improve currying general explanation --- doc/features.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index 0c8eec92..0f4f19c2 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1484,9 +1484,9 @@ thunk() *If the function being curried is `@generic` or `@typed`, or has type annotations on its parameters, the parameters being passed in are type-checked. A type mismatch immediately raises `TypeError`. This helps support [fail-fast](https://en.wikipedia.org/wiki/Fail-fast) in code using `curry`.* -[*Currying*](https://en.wikipedia.org/wiki/Currying) is a technique in functional programming, where a function that takes multiple arguments is converted to a sequence of nested one-argument functions, each one *specializing* (fixing the value of) the leftmost remaining positional parameter. +[*Currying*](https://en.wikipedia.org/wiki/Currying) is a technique in functional programming, where a function that takes multiple arguments is converted to a sequence of nested one-argument functions, each one *specializing* (fixing the value of) the leftmost remaining positional parameter. Each such function returns another function that takes the next parameter. The last function, when no more parameters remain, then performs the actual computation and returns the result. -Some languages, such as Haskell, curry all functions natively. In languages that do not, like Python or [Racket](https://docs.racket-lang.org/reference/procedures.html#%28def._%28%28lib._racket%2Ffunction..rkt%29._curry%29%29), when currying is implemented as a library function, this is often done as a form of [partial application](https://en.wikipedia.org/wiki/Partial_application), which is a subtly different concept, but encompasses the curried behavior as a special case. +Some languages, such as Haskell, curry all functions natively. In languages that do not, like Python or [Racket](https://docs.racket-lang.org/reference/procedures.html#%28def._%28%28lib._racket%2Ffunction..rkt%29._curry%29%29), when currying is implemented as a library function, this is often done as a form of [partial application](https://en.wikipedia.org/wiki/Partial_application), which is a subtly different concept, but encompasses the curried behavior as a special case. In practice this means that you can pass several arguments in a single step, and the original function will be called when all parameters have been bound. Our ``curry`` can be used both as a decorator and as a regular function. As a decorator, `curry` takes no decorator arguments. As a regular function, `curry` itself is curried à la Racket. If any args or kwargs are given (beside the function to be curried), they are the first step. This helps eliminate many parentheses. From 5d90a91a323d274cf650b2ca1f8212f248665ec6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 01:18:45 +0300 Subject: [PATCH 500/832] update `iterate` to use `Values` This is a breaking change, part of 0.15.0. --- CHANGELOG.md | 1 + doc/dialects/lispython.md | 2 +- doc/features.md | 6 ++++-- unpythonic/it.py | 24 +++++++++++++++++------- unpythonic/tests/test_it.py | 4 ++-- 5 files changed, 25 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bbdb224..8bbb1ad5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -156,6 +156,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - `pipe` family - `compose` family - `unfold` + - `iterate` - All multiple-return-values in code using the `with continuations` macro. (The continuations system essentially composes continuation functions.) - The lazy evaluation tools `lazy`, `Lazy`, and the quick lambda `f` (underscore notation for Python) are now provided by `unpythonic` as `unpythonic.syntax.lazy`, `unpythonic.lazyutil.Lazy`, and `unpythonic.syntax.fn` (note name change!), because they used to be provided by `macropy`, and `mcpyrate` does not provide them. - **API differences.** diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 1d0d2d56..21ccf599 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -122,7 +122,7 @@ In the `Lispython` variant, we implicitly import some macros and functions to se - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax``. - The underscore: e.g. `fn[_ * 3]` becomes `lambda x: x * 3`, and `fn[_ * _]` becomes `lambda x, y: x * y`. - ``dyn``, for dynamic assignment. - - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, `unfold`, the `pipe` family, the `compose` family, and the `with continuations` macro.) + - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, `unfold`, `iterate`, the `pipe` family, the `compose` family, and the `with continuations` macro.) For detailed documentation of the language features, see [``unpythonic.syntax``](../macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. diff --git a/doc/features.md b/doc/features.md index 0f4f19c2..43b6f870 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1887,7 +1887,9 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `within`: yield items from iterable until successive iterates are close enough. Useful with [Cauchy sequences](https://en.wikipedia.org/wiki/Cauchy_sequence). **Added in v0.14.2.** - `prod`: like the builtin `sum`, but compute the product. Oddly missing from the standard library. - `iterate1`, `iterate`: return an infinite generator that yields `x`, `f(x)`, `f(f(x))`, ... - - `iterate1` is for 1-to-1 functions; `iterate` for n-to-n, unpacking the return value to the argument list of the next call. + - `iterate1` is for 1-to-1 functions. + - `iterate` is for n-to-n, unpacking the return value to the args/kwargs of the next call. + - **Changed in v0.15.0.** *Now the function must return a `Values` object in the same shape as it accepts args and kwargs.* - *miscellaneous*: - `uniqify`, `uniq`: remove duplicates (either all or consecutive only, respectively), preserving the original ordering of the items. - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. @@ -4120,7 +4122,7 @@ Most of the time, returning a tuple to denote multiple-return-values and unpacki But the distinction is critically important in function composition, so that positional return values can be automatically mapped into positional arguments to the next function in the chain, and named return values into named arguments. -Accordingly, various parts of `unpythonic` that deal with function composition use the `Values` abstraction; particularly `curry`, `unfold`, the `compose` and `pipe` families, and the `with continuations` macro. +Accordingly, various parts of `unpythonic` that deal with function composition use the `Values` abstraction; particularly `curry`, `unfold`, `iterate`, the `compose` and `pipe` families, and the `with continuations` macro. #### Behavior diff --git a/unpythonic/it.py b/unpythonic/it.py index a83bd825..b53caf47 100644 --- a/unpythonic/it.py +++ b/unpythonic/it.py @@ -35,6 +35,8 @@ from itertools import tee, islice, zip_longest, starmap, chain, filterfalse, groupby, takewhile from collections import deque +from .funutil import Values + def rev(iterable): """Reverse an iterable. @@ -562,18 +564,26 @@ def iterate1(f, x): yield x x = f(x) -def iterate(f, *args): +def iterate(f, *args, **kwargs): """Multiple-argument version of iterate1. - The function ``f`` should return a tuple or list of as many elements as it - takes positional arguments; this will be unpacked to the argument list in - the next call. + The initial ``args`` and ``kwargs`` are packed into a ``Values`` object, + which we will below denote as ``x``. When calling ``f``, ``x`` is unpacked + to its args/kwargs. + + The function ``f`` must return a ``Values`` object in the same shape + as it takes args and kwargs; this then becomes the new ``x``. - Or in other words, yield args, f(*args), f(*f(*args)), ... + Using this notation, this function behaves exactly like ``iterate1``: + the return value of ``iterate`` is an infinite generator that yields + x, f(x), f(f(x)), ... """ + x = Values(*args, **kwargs) while True: - yield args - args = f(*args) + yield x + x = f(*x.rets, **x.kwrets) + if not isinstance(x, Values): + raise TypeError(f"Expected a `Values`, got {type(x)} with value {repr(x)}") def partition(pred, iterable): """Partition an iterable to entries satifying and not satisfying a predicate. diff --git a/unpythonic/tests/test_it.py b/unpythonic/tests/test_it.py index 7b7f0fce..50fe546a 100644 --- a/unpythonic/tests/test_it.py +++ b/unpythonic/tests/test_it.py @@ -351,9 +351,9 @@ def primes(): # it doesn't matter where you start, the fixed point of cosine # remains the same. def cos3(a, b, c): - return cos(a), cos(b), cos(c) + return Values(cos(a), cos(b), cos(c)) fp = 0.7390851332151607 - test[the[last(take(100, iterate(cos3, 1.0, 2.0, 3.0)))] == (the[fp], fp, fp)] + test[the[last(take(100, iterate(cos3, 1.0, 2.0, 3.0)))] == Values(the[fp], fp, fp)] # within() - terminate a Cauchy sequence after a tolerance is reached. # The condition is `abs(a - b) <= tol` **for the last two yielded items**. From ea8d8b3bba66d6bd123a36e2f9e7b4a01a600ed0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:07:22 +0300 Subject: [PATCH 501/832] fix example borkage in features.md --- doc/features.md | 67 +++++++++++++++++++++++++++++++------------------ 1 file changed, 43 insertions(+), 24 deletions(-) diff --git a/doc/features.md b/doc/features.md index 43b6f870..a9b77768 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1333,8 +1333,9 @@ We will discuss `memoize`, `curry` and `fix` in more detail shortly; but first, ```python from typing import NoReturn from unpythonic import (fix, andf, orf, rotate, - zipr, rzip, foldl, foldr, - withself) + foldl, foldr, + withself, + composel) # detect and break infinite recursion cycles: # a(0) -> b(1) -> a(2) -> b(0) -> a(1) -> b(2) -> a(0) -> ... @@ -1369,9 +1370,16 @@ myzipr = curry(foldr, zipper, ()) assert myzipl((1, 2, 3), (4, 5, 6), (7, 8)) == ((1, 4, 7), (2, 5, 8)) assert myzipr((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) -# zip and reverse don't commute for inputs with different lengths -assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) # zip first -assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # reverse first +# composel: compose functions, applying the leftmost first +with_n = lambda *args: (partial(f, n) for n, f in args) +clip = lambda n1, n2: composel(*with_n((n1, drop), (n2, take))) +assert tuple(clip(5, 10)(range(20))) == tuple(range(5, 15)) +``` + +In the last example, essentially we just want to `clip 5 10 (range 20)`, the grouping of the parentheses being pretty much an implementation detail. Using the passthrough in ``curry`` (more on which in the section on ``curry``, below), we can rewrite the last line as: + +```python +assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) ``` @@ -1907,14 +1915,19 @@ Examples: ```python from functools import partial +from itertools import count, takewhile +from operator import add, mul from unpythonic import (scanl, scanr, foldl, foldr, - mapr, zipr, + mapr, zipr, rmap, rzip, identity, uniqify, uniq, flatten1, flatten, flatten_in, flatmap, take, drop, unfold, unfold1, + unpack, cons, nil, ll, curry, - s, inn, iindex, + imemoize, gmemoize, + s, inn, iindex, find, + partition, partition_int, window, subset, powerset, allsame, @@ -1965,8 +1978,9 @@ assert not inn(1337, primes()) iseven = lambda x: x % 2 == 0 assert [tuple(it) for it in partition(iseven, range(10))] == [(1, 3, 5, 7, 9), (0, 2, 4, 6, 8)] +# CAUTION: not to be confused with: # partition_int: split a small positive integer, in all possible ways, into smaller integers that sum to it -assert tuple(partition_int(4)) == ((1, 1, 1, 1), (1, 1, 2), (1, 2, 1), (1, 3), (2, 1, 1), (2, 2), (3, 1), (4,)) +assert tuple(partition_int(4)) == ((4,), (3, 1), (2, 2), (2, 1, 1), (1, 3), (1, 2, 1), (1, 1, 2), (1, 1, 1, 1)) assert all(sum(terms) == 10 for terms in partition_int(10)) # iindex: find index of item in iterable (mostly only makes sense for memoized input) @@ -2009,16 +2023,31 @@ def msqrt(x): # multivalued sqrt return (s, -s) assert tuple(flatmap(msqrt, (0, 1, 4, 9))) == (0., 1., -1., 2., -2., 3., -3.) -# zipr reverses, then iterates. -assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) +# **CAUTION**: zip and reverse do NOT commute for inputs with different lengths: +assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) # zip first +assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # reverse first + +# zipr syncs *left* ends, then iterates *from the right*. +assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) + +# so does mapr. +zipr2 = partial(mapr, identity) +assert tuple(zipr2((1, 2, 3), (4, 5, 6), (7, 8))) == (Values(2, 5, 8), Values(1, 4, 7)) + +# rzip syncs *right* ends, then iterates from the right. +assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) -zipr2 = partial(mapr, identity) # mapr works the same way. -assert tuple(zipr2((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) +# so does rmap. +rzip2 = partial(rmap, identity) +assert tuple(rzip2((1, 2, 3), (4, 5, 6), (7, 8))) == (Values(3, 6, 8), Values(2, 5, 7)) -# foldr doesn't; it walks from the left, but collects results from the right: +# foldr syncs *left* ends, then collects results from the right: +def zipper(*args): + *rest, acc = args + return acc + (tuple(rest),) zipr1 = curry(foldr, zipper, ()) assert zipr1((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) -# so the result is reversed(zip(...)), whereas zipr gives zip(*(reversed(s) for s in ...)) +# so the result is tuple(rev(zip(...))), whereas rzip gives tuple(zip(*(rev(s) for s in ...))) assert tuple(uniqify((1, 1, 2, 2, 2, 1, 2, 2, 4, 3, 4, 3, 3))) == (1, 2, 4, 3) # all assert tuple(uniq((1, 1, 2, 2, 2, 1, 2, 2, 4, 3, 4, 3, 3))) == (1, 2, 1, 2, 4, 3, 4, 3) # consecutive @@ -2032,16 +2061,6 @@ assert tuple(flatten((((1, 2), (3, 4)), (5, 6)), is_nested)) == ((1, 2), (3, 4), data = (((1, 2), ((3, 4), (5, 6)), 7), ((8, 9), (10, 11))) assert tuple(flatten(data, is_nested)) == (((1, 2), ((3, 4), (5, 6)), 7), (8, 9), (10, 11)) assert tuple(flatten_in(data, is_nested)) == (((1, 2), (3, 4), (5, 6), 7), (8, 9), (10, 11)) - -with_n = lambda *args: (partial(f, n) for n, f in args) -clip = lambda n1, n2: composel(*with_n((n1, drop), (n2, take))) -assert tuple(clip(5, 10)(range(20))) == tuple(range(5, 15)) -``` - -In the last example, essentially we just want to `clip 5 10 (range 20)`, the grouping of the parentheses being pretty much an implementation detail. With ``curry``, we can rewrite the last line as: - -```python -assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) ``` ### Batteries for network programming From 0350ce3bb47ec8be1728d8208ec45caaa73c47de Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:07:34 +0300 Subject: [PATCH 502/832] add comment --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index a9b77768..f99e8989 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1966,7 +1966,7 @@ assert inn(42, evens()) assert not inn(41, evens()) @gmemoize -def primes(): +def primes(): # FP sieve of Eratosthenes yield 2 for n in count(start=3, step=2): if not any(n % p == 0 for p in takewhile(lambda x: x*x <= n, primes())): From 1501d9bfc64a05f2c3a19201203e8ca6ba438c45 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:07:42 +0300 Subject: [PATCH 503/832] wording --- doc/features.md | 5 +++-- unpythonic/tests/test_fold.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index f99e8989..e0e9fb6b 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1946,8 +1946,9 @@ def step2(k): # x0, x0 + 2, x0 + 4, ... assert tuple(take(10, unfold1(step2, 10))) == (10, 12, 14, 16, 18, 20, 22, 24, 26, 28) def nextfibo(a, b): - # First positional is value; everything else is newstate, - # to be unpacked to `nextfibo`'s args/kwargs at the next iteration. + # First positional return value is the value to yield. + # Everything else is newstate, to be unpacked to `nextfibo`'s + # args/kwargs at the next iteration. return Values(a, a=b, b=a + b) assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) diff --git a/unpythonic/tests/test_fold.py b/unpythonic/tests/test_fold.py index e059005a..d12ce6ea 100644 --- a/unpythonic/tests/test_fold.py +++ b/unpythonic/tests/test_fold.py @@ -183,8 +183,9 @@ def step2(k): # x0, x0 + 2, x0 + 4, ... return (k, k + 2) # (value, newstate) def fibo(a, b): - # First positional is value; everything else is newstate, - # to be unpacked to `fibo`'s args/kwargs at the next iteration. + # First positional return value is the value to yield. + # Everything else is newstate, to be unpacked to `fibo`'s + # args/kwargs at the next iteration. return Values(a, a=b, b=a + b) def myiterate(f, x): # x0, f(x0), f(f(x0)), ... From 4fb1da385635e9f9860313dbabca4e42389f9e3a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:07:55 +0300 Subject: [PATCH 504/832] improve docstring of CountingIterator --- unpythonic/misc.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/unpythonic/misc.py b/unpythonic/misc.py index 757db587..d13bbfd0 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -253,7 +253,10 @@ def __next__(self): class CountingIterator: """Iterator that counts how many elements it has yielded. - The count stops updating when the original iterable raises StopIteration. + Wraps the original iterator of `iterable`. Simply use + `CountingIterator(iterable)` in place of `iter(iterable)`. + + The count stops updating when the original iterator raises StopIteration. """ def __init__(self, iterable): self._it = iter(iterable) From ad74151c923044538bd8e83dc6c9451b0255d978 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:08:29 +0300 Subject: [PATCH 505/832] wording --- doc/features.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index e0e9fb6b..048144ca 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1857,11 +1857,11 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `unfold1`, `unfold`: generate a sequence [corecursively](https://en.wikipedia.org/wiki/Corecursion). The counterpart of `foldl`. - `unfold1` is for 1-in-2-out functions. The input is `state`, the return value must be `(value, newstate)` or `None`. - `unfold` is for n-in-(1+n)-out functions. - - **Changed in v0.15.0.** *The initial args/kwargs are unpacked to the args/kwargs of the user function. The function must return a `Values` object, where the first positional return value is the value to be yielded, and anything else is unpacked to the args/kwargs of the user function at the next iteration.* + - **Changed in v0.15.0.** *The initial args/kwargs are unpacked to the args/kwargs of the user function. The function must return a `Values` object, where the first positional return value is the value to yield, and anything else is unpacked to the args/kwargs of the user function at the next iteration.* - Unfold returns a generator yielding the collected values. The output can be finite or infinite; to signify that a finite sequence ends, the user function must return `None`. (Beside a `Values` object, a bare `None` is the only other allowed return value from the user function.) - *mapping and zipping*: - `map_longest`: the final missing battery for `map`. - - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by ``itertools``. + - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by ``itertools``, but it's convenient to have a named shorthand to do that. - `rmap`, `rzip`, `rmap_longest`, `rzip_longest`: reverse each input, then map/zip. For multiple inputs, syncs the **right** ends. - `mapr`, `zipr`, `mapr_longest`, `zipr_longest`: map/zip, then reverse the result. For multiple inputs, syncs the **left** ends. - `map`: curry-friendly wrapper for the builtin, making it mandatory to specify at least one iterable. **Added in v0.14.2.** @@ -1881,7 +1881,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - *extracting items, subsequences*: - `take`, `drop`, `split_at`: based on `itertools` [recipes](https://docs.python.org/3/library/itertools.html#itertools-recipes). - Especially useful for testing generators. - - `islice` is maybe more pythonic than `take` and `drop`. We provide a utility that supports the slice syntax. + - `islice` is maybe more pythonic than `take` and `drop`; it enables slice syntax for any iterable. - `tail`: return the tail of an iterable. Same as `drop(1, iterable)`; common use case. - `butlast`, `butlastn`: return a generator that yields from iterable, dropping the last `n` items if the iterable is finite. Inspired by a similar utility in PG's [On Lisp](http://paulgraham.com/onlisp.html). - Works by using intermediate storage. **Do not** use the original iterator after a call to `butlast` or `butlastn`. @@ -1897,7 +1897,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `iterate1`, `iterate`: return an infinite generator that yields `x`, `f(x)`, `f(f(x))`, ... - `iterate1` is for 1-to-1 functions. - `iterate` is for n-to-n, unpacking the return value to the args/kwargs of the next call. - - **Changed in v0.15.0.** *Now the function must return a `Values` object in the same shape as it accepts args and kwargs.* + - **Changed in v0.15.0.** *In the n-to-n version, now the user function must return a `Values` object in the same shape as it accepts args and kwargs. This `Values` object is the `x` that is yielded at each iteration.* - *miscellaneous*: - `uniqify`, `uniq`: remove duplicates (either all or consecutive only, respectively), preserving the original ordering of the items. - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. From a8d0bb2a83f16c15d80c134aef4ecde01edbf760 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:08:35 +0300 Subject: [PATCH 506/832] mention etymology for the name `scons` --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 048144ca..b32f3bca 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1901,7 +1901,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - *miscellaneous*: - `uniqify`, `uniq`: remove duplicates (either all or consecutive only, respectively), preserving the original ordering of the items. - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. - - `scons`: prepend one element to the start of an iterable, return new iterable. ``scons(x, iterable)`` is lispy shorthand for ``itertools.chain((x,), iterable)``, allowing to omit the one-item tuple wrapper. + - `scons`: prepend one element to the start of an iterable, return new iterable. ``scons(x, iterable)`` is lispy shorthand for ``itertools.chain((x,), iterable)``, allowing to omit the one-item tuple wrapper. The name is an abbreviation of [`stream-cons`](https://docs.racket-lang.org/reference/streams.html). - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. - Only applicable to monotonic divergent inputs (such as ``primes``). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. From 923ceaffc34d192087a2c108998df7c42b615ad8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 13 Jun 2021 02:08:55 +0300 Subject: [PATCH 507/832] improve explanation of CountingIterator --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index b32f3bca..4b113c2e 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1905,7 +1905,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. - Only applicable to monotonic divergent inputs (such as ``primes``). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. - - `CountingIterator`: count how many items have been yielded, as a side effect. The count is stored in the `.count` attribute. **Added in v0.14.2.** + - `CountingIterator`: use `CountingIterator(iterable)` instead of `iter(iterable)` to produce an iterator that, as a side effect, counts how many items have been yielded. The count is stored in the `.count` attribute. **Added in v0.14.2.** - `slurp`: extract all items from a `queue.Queue` (until it is empty) to a list, returning that list. **Added in v0.14.2.** - `subset`: test whether an iterable is a subset of another. **Added in v0.14.3.** - `powerset`: yield the power set (set of all subsets) of an iterable. Works also for potentially infinite iterables, if only a finite prefix is ever requested. (But beware, both runtime and memory usage are exponential in the input size.) **Added in v0.14.2.** From d46ae1cb39ad01757d5b733220184e5217e5d7bd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 00:27:11 +0300 Subject: [PATCH 508/832] better idea for assert in lambda: use the test[] macro --- doc/design-notes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 25650cac..80015eb4 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -129,7 +129,8 @@ The oft-quoted single-expression limitation of the Python ``lambda`` is ultimate - A lambda can define a class using the three-argument form of the builtin `type` function. For an example, see [Peter Corbett (2005): Statementless Python](https://gist.github.com/brool/1679908), a complete minimal Lisp interpreter implemented as a single Python expression. - A lambda can import a module using the builtin `__import__`, or better, `importlib.import_module`. - A lambda can assert by using an if-expression and then ``raisef`` to actually raise the ``AssertionError``. - - This can be packaged into a function ``assertf``, though that requires jumping through some hoops to produce a traceback that omits ``assertf`` itself. See ``equip_with_traceback``. + - Or use the `test[]` macro, which also shows the source code for the asserted expression if the assertion fails. + - Technically, `test[]` will `signal` the `TestFailure` (part of the public API of `unpythonic.test.fixtures`), not raise it, but essentially, `test[]` is a more convenient assert that optionally hooks into a testing framework. The error signal, if unhandled, will automatically chain into raising a `ControlError` exception, which is often just fine. - Context management (``with``) is currently **not** available for lambdas, even in ``unpythonic``. - Aside from the `async` stuff, this is the last hold-out preventing full generality, so we will likely add an expression form of ``with`` in a future version. This is tracked in [issue #76](https://github.com/Technologicat/unpythonic/issues/76). From 3fbe93e56eaa8a0ce0e70060588d5707acf99f47 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 02:17:59 +0300 Subject: [PATCH 509/832] improve infinite replacements in fup/fupdate/ShadowedSequence --- CHANGELOG.md | 4 +++ doc/features.md | 31 +++++++++++++++++ unpythonic/collections.py | 51 ++++++++++++++++++++++++---- unpythonic/gmemo.py | 11 ++++++ unpythonic/tests/test_collections.py | 12 +++++++ unpythonic/tests/test_fup.py | 21 +++++++++++- unpythonic/tests/test_slicing.py | 7 +++- 7 files changed, 129 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bbb1ad5..30309e5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,6 +119,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Positional passthrough works as before. Named passthrough added. - Any remaining arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result (if any), and then outward on the curry context stack as a `Values`. Since `curry` in this role is essentially a function-composition utility, the receiving curried function instance unpacks the `Values` into args and kwargs. - If any extra arguments (positional or named) remain when the top-level curry context exits, then by default, `TypeError` is raised. To override, use `with dyn.let(curry_context=["whatever"])`, just like before. Then you'll get a `Values` object. + - The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Note that they do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose. + - `fup`/`fupdate`/`ShadowedSequence` can now walk the start of a memoized infinite replacement backwards. (Use `imemoize` on the original iterable, instantiate the generator, and use that generator instance as the replacement.) - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). @@ -200,6 +202,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Fix bug in `with namedlambda`. Due to incorrect function arguments in the analyzer, already named lambdas were not detected correctly. +- Fix bug: `fup`/`fupdate`/`ShadowedSequence` now actually accept an infinite-length iterable as a replacement sequence (under the obvious usage limitations), as the documentation has always claimed. + --- diff --git a/doc/features.md b/doc/features.md index 4b113c2e..20073ff8 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2154,6 +2154,8 @@ Inspired by Python itself. ### `gmemoize`, `imemoize`, `fimemoize`: memoize generators +**Changed in v0.15.0.** *The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Note that they do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose.* + Make generator functions (gfunc, i.e. a generator definition) which create memoized generators, similar to how streams behave in Racket. Memoize iterables; like `itertools.tee`, but no need to know in advance how many copies of the iterator will be made. Provided for both iterables and for factory functions that make iterables. @@ -2248,6 +2250,8 @@ The only differences are the name of the decorator and ``return`` vs. ``yield fr ### ``fup``: Functional update; ``ShadowedSequence`` +**Changed in 0.15.0.** *Bug fixed: Now an infinite replacement sequence to pull items from is actually ok, as the documentation has always claimed.* + We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). It supports slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. See the docstring of ``ShadowedSequence`` for details. The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy, of the same type as the given input sequence, when it finalizes its output. @@ -2302,6 +2306,33 @@ When ``fupdate`` constructs its output, the replacement occurs by walking *the i The replacement sequence must have at least as many items as the slice requires (when applied to the original input). Any extra items in the replacement sequence are simply ignored (so e.g. an infinite ``repeat`` is fine), but if the replacement is too short, ``IndexError`` is raised. +Note that the replacement must have `__len__` and `__getitem__` methods if the replacement specification requires reading it backwards, and/or if you plan to iterate over the `ShadowedSequence` multiple times. If the replacement only needs to be read forwards, **AND** you only plan to iterate over the `ShadowedSequence` just once (e.g., as part of a `fup`/`fupdate` operation), then it is sufficient for the replacement to implement the `collections.abc.Iterator` API only (i.e. just `__iter__` and `__next__`). + +So, as of v0.15.0, this is supported: + +```python +from itertools import repeat, count +from unpythonic import fup + +lst = (1, 2, 3, 4, 5) +assert fup(lst)[::] << repeat(42) == (42, 42, 42, 42, 42) +assert fup(lst)[::] << count(start=10) == (10, 11, 12, 13, 14) +``` + +If you need to reverse-walk the start of an infinite replacement, then `imemoize(...)` to create a memoizing gfunc, and instantiate it: + +```python +from itertools import count +from unpythonic import fup, imemoize + +lst = (1, 2, 3, 4, 5) +assert fup(lst)[::-1] << imemoize(count(start=10))() == (14, 13, 12, 11, 10) +``` + +Note that as before, due to the `[::-1]`, the *fifth* item of the memoized iterable is used first. The `fup` succeeds, because all five items are stored in the memo (which is internally a sequence). + +Once enough items have been yielded to perform the replacement, this will internally use `__getitem__` to retrieve the actual items. This supports any generator instance created by `imemoize`, `fimemoize`, or `gmemoize`. + It is also possible to replace multiple individual items. These are treated as separate specifications, applied left to right (so later updates shadow earlier ones, if updating at the same index): ```python diff --git a/unpythonic/collections.py b/unpythonic/collections.py index 3eaf65b7..96c35dd3 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -27,8 +27,9 @@ from .env import env from .dynassign import _Dyn from .funutil import Values +from .it import drop from .llist import cons, Nil -from .misc import getattrrec +from .misc import getattrrec, CountingIterator def get_abcs(cls): """Return a set of the collections.abc superclasses of cls (virtuals too).""" @@ -743,16 +744,34 @@ class ShadowedSequence(Sequence, _StrReprEqMixin): Essentially, ``out[k] = v[index_in_slice(k, ix)] if in_slice(k, ix) else seq[k]``, but doesn't actually allocate ``out``. - ``ix`` may be integer (if ``v`` represents one item only) or slice (if ``v`` - is intended as a sequence). The default ``None`` means ``out[k] = seq[k]`` + ``ix`` may be integer (if ``v`` represents one item only) or ``slice`` (if ``v`` + is intended as a sequence). The default ``ix=None`` means ``out[k] = seq[k]`` with no shadower. + + If ``ix`` is a ``slice``, then: + + - If the replacement specification requires reading ``v`` backwards, + and/or if you plan to iterate over the ``ShadowedSequence`` more + than once, then ``v`` must implement ``collections.abc.Sequence``, + i.e. it must have ``__len__`` and ``__getitem__`` methods. + + - If the replacement specification only needs reading ``v`` forwards, + **AND** if you plan to read the ``ShadowedSequence`` only once (e.g. + as part of a `fupdate` or `fup` operation), then it is sufficient + for ``v`` to implement only ``collections.abc.Iterator``, i.e. the + ``__iter__`` and ``__next__`` methods only. """ def __init__(self, seq, ix=None, v=None): if ix is not None and not isinstance(ix, (slice, int)): raise TypeError(f"ix: expected slice or int, got {type(ix)} with value {ix}") + if not isinstance(seq, Sequence): + raise TypeError(f"seq: expected a sequence, got {type(seq)} with value {seq}") + if isinstance(ix, slice) and not isinstance(v, (Sequence, Iterable)): + raise TypeError(f"v: when ix is a slice, v must be a sequence or an iterable; got {type(v)} with value {v}") self.seq = seq self.ix = ix self.v = v + self._v_it = None # Provide __iter__ (even though implemented using len() and __getitem__()) # so that our __getitem__ can raise IndexError when needed, without it @@ -794,9 +813,29 @@ def _getone(self, k): return self.v # just one item # we already know k is in ix, so skip validation for speed. i = _index_in_slice(k, ix, n, _validate=False) - if i >= len(self.v): - raise IndexError(f"Replacement sequence too short; attempted to access index {i} with len {len(self.v)} (items: {self.v})") - return self.v[i] + if isinstance(self.v, Sequence): + if i >= len(self.v): + raise IndexError(f"Replacement sequence too short; attempted to access index {i} with len {len(self.v)} (items: {self.v})") + return self.v[i] + elif isinstance(self.v, Iterable): + if not self._v_it: + self._v_it = CountingIterator(self.v) + if i < self._v_it.count: + # Special case for `unpythonic.gmemo._MemoizedGenerator`, + # to support reverse-walking a replacement that was created + # using `imemoize`/`fimemoize`/`gmemoize`. + bare_it = self._v_it._it + if all(hasattr(bare_it, name) for name in ("__len__", "__getitem__")): + assert i < len(bare_it) # because we counted them! + return bare_it[i] + raise IndexError(f"Trying to read an already consumed item of a non-sequence iterable; attempted to access index {i} with {self._v_it.count} items already consumed.") + n_skip = i - self._v_it.count + assert n_skip >= 0 + if n_skip: + self._v_it = drop(n_skip, self._v_it) + return next(self._v_it) + else: + assert False return self.seq[k] # not in slice def in_slice(i, s, length=None): diff --git a/unpythonic/gmemo.py b/unpythonic/gmemo.py index 30607a5c..6e9194e2 100644 --- a/unpythonic/gmemo.py +++ b/unpythonic/gmemo.py @@ -112,6 +112,7 @@ def __init__(self, g, memo, lock): self.j = 0 # current position in memo def __repr__(self): return f"<_MemoizedGenerator object {self.g.__name__} at 0x{id(self):x}>" + # Support the `collections.abc.Iterable` API def __iter__(self): return self def __next__(self): @@ -131,6 +132,16 @@ def __next__(self): if kind is _fail: raise value return value + # Support the `collections.abc.Sequence` API for already-computed items + def __len__(self): + return len(self.memo) + def __getitem__(self, k): + if k >= len(self.memo): + raise IndexError(f"Attempted to access index {k} of memoized generator; only {len(self.memo)} items available (at least so far)") + kind, value = self.memo[k] + if kind is _fail: + raise value + return value def imemoize(iterable): """Memoize an iterable. diff --git a/unpythonic/tests/test_collections.py b/unpythonic/tests/test_collections.py index d318607d..3d6ce120 100644 --- a/unpythonic/tests/test_collections.py +++ b/unpythonic/tests/test_collections.py @@ -4,6 +4,7 @@ from ..test.fixtures import session, testset from collections.abc import Mapping, MutableMapping, Hashable, Container, Iterable, Sized +from itertools import count, repeat from pickle import dumps, loads import threading @@ -11,6 +12,7 @@ frozendict, view, roview, ShadowedSequence, mogrify, in_slice, index_in_slice) from ..fold import foldr +from ..gmemo import imemoize from ..llist import cons, ll def runtests(): @@ -469,6 +471,16 @@ class Zee: s6 = ShadowedSequence(tpl, slice(2, 4), (23,)) # replacement too short... test_raises[IndexError, s6[3]] # ...which is detected here + # infinite replacements + # Here we must `tuple()` the LHS so that the replacement *iterable*, + # which is not a sequence, is iterated over only once. + test[tuple(ShadowedSequence(tpl, slice(None, None, None), repeat(42))) == (42, 42, 42, 42, 42)] + test[tuple(ShadowedSequence(tpl, slice(None, None, None), count(start=10))) == (10, 11, 12, 13, 14)] + + # reading the start of a memoized infinite replacement backwards + test[tuple(ShadowedSequence(tpl, slice(None, None, -1), imemoize(repeat(42))())) == (42, 42, 42, 42, 42)] + test[tuple(ShadowedSequence(tpl, slice(None, None, -1), imemoize(count(start=10))())) == (14, 13, 12, 11, 10)] + # mogrify: in-place map for various data structures (see docstring for details) with testset("mogrify"): double = lambda x: 2 * x diff --git a/unpythonic/tests/test_fup.py b/unpythonic/tests/test_fup.py index 335ca0b0..c6522c31 100644 --- a/unpythonic/tests/test_fup.py +++ b/unpythonic/tests/test_fup.py @@ -3,11 +3,12 @@ from ..syntax import macros, test, test_raises, the # noqa: F401 from ..test.fixtures import session, testset -from itertools import repeat +from itertools import count, repeat from collections import namedtuple from ..fup import fupdate from ..collections import frozendict +from ..gmemo import imemoize def runtests(): with testset("mutable sequence"): @@ -90,6 +91,24 @@ def runtests(): test[tup == tuple(range(10))] test[out == (2, 3, 2, 3, 2, 3, 2, 3, 2, 3)] + with testset("infinite replacement"): + tup = (1, 2, 3, 4, 5) + out = fupdate(tup, slice(None, None, None), repeat(42)) + test[out == (42, 42, 42, 42, 42)] + + tup = (1, 2, 3, 4, 5) + out = fupdate(tup, slice(None, None, None), count(start=10)) + test[out == (10, 11, 12, 13, 14)] + + with testset("memoized infinite replacement, reading its start backwards"): + tup = (1, 2, 3, 4, 5) + out = fupdate(tup, slice(None, None, -1), imemoize(repeat(42))()) + test[out == (42, 42, 42, 42, 42)] + + tup = (1, 2, 3, 4, 5) + out = fupdate(tup, slice(None, None, -1), imemoize(count(start=10))()) + test[out == (14, 13, 12, 11, 10)] + with testset("mix and match"): tup = tuple(range(10)) out = fupdate(tup, (slice(0, 10, 2), slice(1, 10, 2), 6), diff --git a/unpythonic/tests/test_slicing.py b/unpythonic/tests/test_slicing.py index 4ae6bcd5..b810abf9 100644 --- a/unpythonic/tests/test_slicing.py +++ b/unpythonic/tests/test_slicing.py @@ -4,9 +4,10 @@ from ..syntax import macros, test, test_raises # noqa: F401 from ..test.fixtures import session, testset -from itertools import repeat +from itertools import count, repeat from ..slicing import fup, islice +from ..gmemo import imemoize from ..mathseq import primes, s def runtests(): @@ -20,6 +21,10 @@ def runtests(): test[fup(tup)[1::2] << tuple(repeat(10, 3)) == (1, 10, 3, 10, 5)] test[fup(tup)[::2] << tuple(repeat(10, 3)) == (10, 2, 10, 4, 10)] test[fup(tup)[::-1] << tuple(range(5)) == (4, 3, 2, 1, 0)] + test[fup(tup)[0::2] << repeat(10) == (10, 2, 10, 4, 10)] # infinite replacement + test[fup(tup)[0::2] << count(start=10) == (10, 2, 11, 4, 12)] + test[fup(tup)[::2] << imemoize(repeat(10))() == (10, 2, 10, 4, 10)] # memoized infinite replacement backwards + test[fup(tup)[::-2] << imemoize(count(start=10))() == (12, 2, 11, 4, 10)] test[tup == (1, 2, 3, 4, 5)] test_raises[TypeError, fup(tup)[2, 3]] # multidimensional indexing not supported From 7a22c41f64b92791d48d62b804c499ae4c55fc5a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 02:19:43 +0300 Subject: [PATCH 510/832] wording/styling --- doc/features.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/features.md b/doc/features.md index 20073ff8..b42703f6 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2121,7 +2121,7 @@ For a usage example of `unpythonic.net.PTYProxy`, see the source code of `unpyth **Changed in v0.14.2.** *Added support for negative `start` and `stop`.* -Slice an iterable, using the regular slicing syntax: +Slice any iterable, using the regular slicing syntax: ```python from unpythonic import islice, primes, s @@ -2139,7 +2139,7 @@ assert tuple(islice(odds)[:5]) == (1, 3, 5, 7, 9) assert tuple(islice(odds)[:5]) == (11, 13, 15, 17, 19) # five more ``` -As a convenience feature: a single index is interpreted as a length-1 islice starting at that index. The slice is then immediately evaluated and the item is returned. +As a convenience feature: a single index is interpreted as a length-1 `islice` starting at that index. The slice is then immediately evaluated and the item is returned. The slicing variant calls ``itertools.islice`` with the corresponding slicing parameters, after possibly converting negative `start` and `stop` to the appropriate positive values. @@ -2162,17 +2162,17 @@ Memoize iterables; like `itertools.tee`, but no need to know in advance how many - `gmemoize` is a decorator for a gfunc, which makes it memoize the instantiated generators. - If the gfunc takes arguments, they must be hashable. A separate memoized sequence is created for each unique set of argument values seen. - - For simplicity, the generator itself may use ``yield`` for output only; ``send`` is not supported. + - For simplicity, the generator itself may use ``yield`` for output only; ``send`` is **not** supported. - Any exceptions raised by the generator (except StopIteration) are also memoized, like in ``memoize``. - - Thread-safe. Calls to ``next`` on the memoized generator from different threads are serialized via a lock. Each memoized sequence has its own lock. This uses ``threading.RLock``, so re-entering from the same thread (e.g. in recursively defined sequences) is fine. + - Thread-safe. Calls to ``next`` on the memoized generator from different threads are serialized via a lock. Each memoized sequence has its own lock. This uses ``threading.RLock``, so re-entering from the same thread (e.g. in recursively defined mathematical sequences) is fine. - The whole history is kept indefinitely. For infinite iterables, use this only if you can guarantee that only a reasonable number of terms will ever be evaluated (w.r.t. available RAM). - - Typically, this should be the outermost decorator if several are used on the same gfunc. + - Typically, `gmemoize` should be the outermost decorator if several are used on the same gfunc. - `imemoize`: memoize an iterable. Like `itertools.tee`, but keeps the whole history, so more copies can be teed off later. - Same limitation: **do not** use the original iterator after it is memoized. The danger is that if anything other than the memoization mechanism advances the original iterator, some values will be lost before they can reach the memo. - Returns a gfunc with no parameters which, when called, returns a generator that yields items from the memoized iterable. The original iterable is used to retrieve more terms when needed. - Calling the gfunc essentially tees off a new instance, which begins from the first memoized item. - `fimemoize`: convert a factory function, that returns an iterable, into the corresponding gfunc, and `gmemoize` that. Return the memoized gfunc. - - Especially convenient with short lambdas, where `(yield from ...)` instead of `...` is just too much text. + - Especially convenient with short lambdas, where `(yield from ...)` instead of `...` is just too much text. See example below. ```python from itertools import count, takewhile @@ -2256,7 +2256,7 @@ We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, b The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy, of the same type as the given input sequence, when it finalizes its output. -**The preferred way** to use ``fupdate`` on sequences is through the ``fup`` utility function, which specializes ``fupdate`` to sequences, and adds support for Python's standard slicing syntax: +**The preferred way** to use ``fupdate`` on sequences is through the ``fup`` utility function, which specializes ``fupdate`` to sequences, and adds support for Python's standard **slicing syntax**: ```python from unpythonic import fup @@ -2284,7 +2284,7 @@ assert lst == [1, 2, 3] # the original remains untouched assert out == [1, 42, 3] lst = [1, 2, 3] -out = fupdate(lst, -1, 42) # negative indices also supported +out = fupdate(lst, -1, 42) # negative indices are also supported assert lst == [1, 2, 3] assert out == [1, 2, 42] ``` From 8f6d61185ecc9b8f5cb3e29e3505a28f9450ca35 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 02:19:53 +0300 Subject: [PATCH 511/832] add missing TOC link --- doc/features.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/features.md b/doc/features.md index b42703f6..79b51d6c 100644 --- a/doc/features.md +++ b/doc/features.md @@ -56,6 +56,7 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``fix``: break infinite recursion cycles](#fix-break-infinite-recursion-cycles) - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. - [**Batteries for network programming**](#batteries-for-network-programming): message protocol, PTY/socket proxy, etc. + - [`unpythonic.net.msg`](#unpythonic-net-msg): message protocol. - [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) - [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. - [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. From 1e97c5ff0037cd46df4d6fae4cb33213f385835b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 02:20:21 +0300 Subject: [PATCH 512/832] wording --- doc/features.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 79b51d6c..2ece4381 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2072,10 +2072,16 @@ assert tuple(flatten_in(data, is_nested)) == (((1, 2), (3, 4), (5, 6), 7), (8, While all other pure-Python features of `unpythonic` live in the main `unpythonic` package, the network-related features are placed in the subpackage `unpythonic.net`. This subpackage also contains the [REPL server and client](repl.md) for hot-patching live processes. - `unpythonic.net.msg`: A simplistic message protocol for sending message data over a stream-based transport, such as TCP. -- `unpythonic.net.ptyproxy`: Proxy between a Linux [PTY](https://en.wikipedia.org/wiki/Pseudoterminal) and a network socket. Useful for serving terminal utilities over the network. The selling point is this doesn't use `pty.spawn`, so it can be used for proxying also Python libraries that expect to run in a terminal. +- `unpythonic.net.ptyproxy`: Proxy between a Linux [PTY](https://en.wikipedia.org/wiki/Pseudoterminal) and a network socket. Useful for serving terminal utilities over the network. The selling point is this does **not** use `pty.spawn`, so it can be used for proxying also Python libraries that expect to run in a terminal. - `unpythonic.net.util`: Miscellaneous small utilities. -The thing about stream-based transports is that they have no concept of a message boundary [[1]](http://stupidpythonideas.blogspot.com/2013/05/sockets-are-byte-streams-not-message.html) [[2]](https://eli.thegreenplace.net/2011/08/02/length-prefix-framing-for-protocol-buffers) [[3]](https://docs.python.org/3/howto/sockets.html). This is where a message protocol comes in. We provide a [sans-io](https://sans-io.readthedocs.io/) implementation of a minimalistic custom protocol that adds rudimentary [message framing](https://blog.stephencleary.com/2009/04/message-framing.html) and [stream re-synchronization](https://en.wikipedia.org/wiki/Frame_synchronization). Example: +For a usage example of `unpythonic.net.ptyproxy`, see the source code of `unpythonic.net.server`. + +More details can be found in the docstrings. + +#### `unpythonic.net.msg` + +The problem with stream-based transports, such as network sockets, is that they have no concept of a message boundary [[1]](http://stupidpythonideas.blogspot.com/2013/05/sockets-are-byte-streams-not-message.html) [[2]](https://eli.thegreenplace.net/2011/08/02/length-prefix-framing-for-protocol-buffers) [[3]](https://docs.python.org/3/howto/sockets.html). This is where a message protocol comes in. We provide a [sans-io](https://sans-io.readthedocs.io/) implementation of a minimalistic message protocol that adds rudimentary [message framing](https://blog.stephencleary.com/2009/04/message-framing.html) and [stream re-synchronization](https://en.wikipedia.org/wiki/Frame_synchronization). Example: ```python from io import BytesIO, SEEK_SET @@ -2115,8 +2121,6 @@ assert decoder.decode() == b"mew" assert decoder.decode() is None ``` -For a usage example of `unpythonic.net.PTYProxy`, see the source code of `unpythonic.net.server`. - ### ``islice``: slice syntax support for ``itertools.islice` From 5acd50b6ef54d50fc1121bac0e806c1d0f0f0bc4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 02:20:35 +0300 Subject: [PATCH 513/832] islice doc: the desired elements are held in an internal buffer --- doc/features.md | 2 +- unpythonic/slicing.py | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 2ece4381..fbd4d91a 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2148,7 +2148,7 @@ As a convenience feature: a single index is interpreted as a length-1 `islice` s The slicing variant calls ``itertools.islice`` with the corresponding slicing parameters, after possibly converting negative `start` and `stop` to the appropriate positive values. -**CAUTION**: When using negative `start` and/or `stop`, we must consume the whole iterable to determine where it ends, if at all. Obviously, this will not terminate for infinite iterables. +**CAUTION**: When using negative `start` and/or `stop`, the whole iterable is consumed to determine where it ends, if at all. Obviously, this will not terminate for infinite iterables. The desired elements are then held in an internal buffer until they are yielded by iterating over the `islice`. **CAUTION**: Keep in mind that negative `step` is not supported, and that the slicing process consumes elements from the iterable. diff --git a/unpythonic/slicing.py b/unpythonic/slicing.py index 3aea8e8e..11884073 100644 --- a/unpythonic/slicing.py +++ b/unpythonic/slicing.py @@ -22,6 +22,9 @@ def islice(iterable): start or stop will force the iterable, because that is the only way to know its length. + The desired elements are held in an internal buffer until they are yielded + by iterating over the `islice`. + - A single index (negative also allowed) is interpreted as a length-1 islice starting at that index. The slice is then immediately evaluated and the item is returned. From 58f82836bbb13ed1b0451de2ad5fb595eec1418c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 02:26:45 +0300 Subject: [PATCH 514/832] improve comments --- unpythonic/collections.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/unpythonic/collections.py b/unpythonic/collections.py index 96c35dd3..748ab5cf 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -822,8 +822,11 @@ def _getone(self, k): self._v_it = CountingIterator(self.v) if i < self._v_it.count: # Special case for `unpythonic.gmemo._MemoizedGenerator`, - # to support reverse-walking a replacement that was created - # using `imemoize`/`fimemoize`/`gmemoize`. + # to support reverse-walking the start of a memoized infinite replacement + # that was created using `imemoize`/`fimemoize`/`gmemoize`. + # It has the `__len__` and `__getitem__` methods, but does + # **not** support the full `collections.abc.Sequence` API. + # At this point, the memo contains all the items accessed or dropped so far. bare_it = self._v_it._it if all(hasattr(bare_it, name) for name in ("__len__", "__getitem__")): assert i < len(bare_it) # because we counted them! @@ -832,6 +835,7 @@ def _getone(self, k): n_skip = i - self._v_it.count assert n_skip >= 0 if n_skip: + # NOTE: If the iterable is memoized, the items we drop here will enter the memo. self._v_it = drop(n_skip, self._v_it) return next(self._v_it) else: From e782d22ba1a5ff332feac627db62acf832cc3ba2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 03:17:54 +0300 Subject: [PATCH 515/832] improve fup/fupdate docs --- doc/features.md | 106 ++++++++++++++++++++++++++++++------------------ 1 file changed, 67 insertions(+), 39 deletions(-) diff --git a/doc/features.md b/doc/features.md index fbd4d91a..c98c615f 100644 --- a/doc/features.md +++ b/doc/features.md @@ -60,6 +60,8 @@ The exception are the features marked **[M]**, which are primarily intended as a - [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) - [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. - [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. + - [`fup`](#fup): the high-level syntactic sugar to update a sequence functionally. + - [`fupdate`](#fupdate): the low-level workhorse. - [``view``: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. - [``mogrify``: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) - [``s``, ``imathify``, ``gmathify``: lazy mathematical sequences with infix arithmetic](#s-imathify-gmathify-lazy-mathematical-sequences-with-infix-arithmetic) @@ -2257,28 +2259,39 @@ The only differences are the name of the decorator and ``return`` vs. ``yield fr **Changed in 0.15.0.** *Bug fixed: Now an infinite replacement sequence to pull items from is actually ok, as the documentation has always claimed.* -We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). It supports slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. See the docstring of ``ShadowedSequence`` for details. +We provide three layers, in increasing order of the level of abstraction: `ShadowedSequence`, `fupdate`, and `fup`. + +The class ``ShadowedSequence`` is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). It supports slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. We will not discuss ``ShadowedSequence`` in more detail here, as it is a low-level tool; see its docstring for details. The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy, of the same type as the given input sequence, when it finalizes its output. +Finally, the function ``fup`` provides a high-level API to functionally update a sequence, with nice syntax. + +#### `fup` + **The preferred way** to use ``fupdate`` on sequences is through the ``fup`` utility function, which specializes ``fupdate`` to sequences, and adds support for Python's standard **slicing syntax**: ```python from unpythonic import fup from itertools import repeat -lst = (1, 2, 3, 4, 5) -assert fup(lst)[3] << 42 == (1, 2, 3, 42, 5) -assert fup(lst)[0::2] << tuple(repeat(10, 3)) == (10, 2, 10, 4, 10) +tup = (1, 2, 3, 4, 5) +assert fup(tup)[3] << 42 == (1, 2, 3, 42, 5) +assert fup(tup)[0::2] << tuple(repeat(10, 3)) == (10, 2, 10, 4, 10) +assert fup(tup)[0::2] << repeat(10) == (10, 2, 10, 4, 10) # infinite replacement ``` -Currently only one update specification is supported in a single ``fup()``. (The ``fupdate`` function supports more; see below.) +Currently only one *update specification* is supported in a single ``fup()``. The low-level ``fupdate`` function supports more; see below. + +An *update specification* is a combination of **where** to update, and **what** to put there. The *where* part can be a single index or a slice. When it is a single index, the *what* is a single item; and when a slice, the *what* is a sequence or an iterable, which must contain at least as many items as are required to perform the update. (For details, see `fupdate` below.) + +The ``fup`` function is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. The notation follows the ``unpythonic`` convention that ``<<`` denotes an assignment of some sort. Here it denotes a functional update, which returns a modified copy, leaving the original untouched. -The ``fup`` call is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. +#### `fupdate` -The ``fupdate`` function itself works as follows: +The ``fupdate`` function itself, which is the next lower abstraction level, works as follows: ```python from unpythonic import fupdate @@ -2294,80 +2307,92 @@ assert lst == [1, 2, 3] assert out == [1, 2, 42] ``` -Immutable input sequences are allowed. Replacing a slice of a tuple by a sequence: +Because the update is functional - i.e. the result is a new object, without mutating the original - immutable update target sequences are allowed. For example, we can replace a slice of a tuple by a sequence: ```python from itertools import repeat -lst = (1, 2, 3, 4, 5) -assert fupdate(lst, slice(0, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) -assert fupdate(lst, slice(1, None, 2), tuple(repeat(10, 2))) == (1, 10, 3, 10, 5) -assert fupdate(lst, slice(None, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) -assert fupdate(lst, slice(None, None, -1), tuple(range(5))) == (4, 3, 2, 1, 0) +tup = (1, 2, 3, 4, 5) +assert fupdate(tup, slice(0, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) +assert fupdate(tup, slice(1, None, 2), tuple(repeat(10, 2))) == (1, 10, 3, 10, 5) +assert fupdate(tup, slice(None, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) +assert fupdate(tup, slice(None, None, -1), range(5)) == (4, 3, 2, 1, 0) ``` Slicing supports negative indices and steps, and default starts, stops and steps, as usual in Python. Just remember ``a[start:stop:step]`` actually means ``a[slice(start, stop, step)]`` (with ``None`` replacing omitted ``start``, ``stop`` and ``step``), and everything should follow. Multidimensional arrays are **not** supported. -When ``fupdate`` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. (In the last example above, ``tuple(range(5))`` was read in the order ``(4, 3, 2, 1, 0)``.) +When ``fupdate`` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. In the last example above, the ``range(5)`` was read in the order ``4, 3, 2, 1, 0``. This is because when `slice(None, None, -1)` is applied to the input sequence, the first item of the input sequence is index `4` in the slice. So when replacing the first item, ``fupdate`` looked up index `4` in the replacement sequence. Because the replacement was just `range(5)`, the value at index `4` was also `4`. + +The replacement sequence must have at least as many items as the slice requires, when the slice is applied to the original input sequence. Any extra items in the replacement sequence are simply ignored, but if the replacement is too short, ``IndexError`` is raised. -The replacement sequence must have at least as many items as the slice requires (when applied to the original input). Any extra items in the replacement sequence are simply ignored (so e.g. an infinite ``repeat`` is fine), but if the replacement is too short, ``IndexError`` is raised. +The replacement must have `__len__` and `__getitem__` methods if the slice (when treated as explained above) requires reading the replacement backwards, and/or if you plan to iterate over the `ShadowedSequence` multiple times. If the replacement only needs to be read forwards, **AND** you only plan to iterate over the `ShadowedSequence` just once (e.g., as part of a `fup`/`fupdate` operation), then it is sufficient for the replacement to implement the `collections.abc.Iterator` API only (i.e. just `__iter__` and `__next__`). -Note that the replacement must have `__len__` and `__getitem__` methods if the replacement specification requires reading it backwards, and/or if you plan to iterate over the `ShadowedSequence` multiple times. If the replacement only needs to be read forwards, **AND** you only plan to iterate over the `ShadowedSequence` just once (e.g., as part of a `fup`/`fupdate` operation), then it is sufficient for the replacement to implement the `collections.abc.Iterator` API only (i.e. just `__iter__` and `__next__`). +##### Infinite replacements -So, as of v0.15.0, this is supported: +An infinite replacement causes `fupdate` (and `fup`) to pull as many items as are needed: ```python from itertools import repeat, count from unpythonic import fup -lst = (1, 2, 3, 4, 5) -assert fup(lst)[::] << repeat(42) == (42, 42, 42, 42, 42) -assert fup(lst)[::] << count(start=10) == (10, 11, 12, 13, 14) +tup = (1, 2, 3, 4, 5) +assert fup(tup)[::] << repeat(42) == (42, 42, 42, 42, 42) +assert fup(tup)[::] << count(start=10) == (10, 11, 12, 13, 14) ``` -If you need to reverse-walk the start of an infinite replacement, then `imemoize(...)` to create a memoizing gfunc, and instantiate it: +The rest of the infinite replacement is considered as extra items, and is ignored. + +**CAUTION**: If converting existing code, **be careful** not to accidentally `tuple(...)` an infinite replacement. Python will happily fill all available RAM and essentially crash your machine trying to exhaust the infinite generator. + +If you need to reverse-walk the start of an infinite replacement: use `imemoize(...)` on the original iterable, instantiate the generator, and use that generator instance as the replacement: ```python from itertools import count from unpythonic import fup, imemoize -lst = (1, 2, 3, 4, 5) -assert fup(lst)[::-1] << imemoize(count(start=10))() == (14, 13, 12, 11, 10) +tup = (1, 2, 3, 4, 5) +assert fup(tup)[::-1] << imemoize(count(start=10))() == (14, 13, 12, 11, 10) ``` -Note that as before, due to the `[::-1]`, the *fifth* item of the memoized iterable is used first. The `fup` succeeds, because all five items are stored in the memo (which is internally a sequence). +Just like above, due to the slice `[::-1]`, `fup` calculates that - when walking *the input sequence* left-to-right - it first needs to take the item at index `4` of the replacement. The `fup` succeeds, because when it retrieves this fifth item, all of the first five items are stored in the memo (which is internally a sequence). So `fup` can retrieve the fifth item, then the fourth, and so on - even though from the viewpoint of the original underlying iterable, the earlier items have already been consumed when the fifth item is accessed. + +`ShadowedSequence` (and thus also `fupdate` and `fup`) internally uses `__getitem__` to retrieve the actual previous items from the memo, so even the memoized generator is only iterated over once. This functionality supports any generator instance created by the gfuncs returned by `imemoize`, `fimemoize`, or `gmemoize`. -Once enough items have been yielded to perform the replacement, this will internally use `__getitem__` to retrieve the actual items. This supports any generator instance created by `imemoize`, `fimemoize`, or `gmemoize`. +##### Multiple update specifications -It is also possible to replace multiple individual items. These are treated as separate specifications, applied left to right (so later updates shadow earlier ones, if updating at the same index): +In `fupdate`, it is also possible to replace multiple individual items: ```python -lst = (1, 2, 3, 4, 5) -out = fupdate(lst, (1, 2, 3), (17, 23, 42)) -assert lst == (1, 2, 3, 4, 5) +tup = (1, 2, 3, 4, 5) +out = fupdate(tup, (1, 2, 3), (17, 23, 42)) # target, (*where), (*what) +assert tup == (1, 2, 3, 4, 5) assert out == (1, 17, 23, 42, 5) ``` +These are treated as separate specifications, applied left to right. This means later updates shadow earlier ones, if updating at the same index: + Multiple specifications can be used with slices and sequences as well: ```python -lst = tuple(range(10)) -out = fupdate(lst, (slice(0, 10, 2), slice(1, 10, 2)), +tup = tuple(range(10)) +out = fupdate(tup, (slice(0, 10, 2), slice(1, 10, 2)), (tuple(repeat(2, 5)), tuple(repeat(3, 5)))) -assert lst == tuple(range(10)) +assert tup == tuple(range(10)) assert out == (2, 3, 2, 3, 2, 3, 2, 3, 2, 3) ``` Strictly speaking, each specification can be either a slice/sequence pair or an index/item pair: ```python -lst = tuple(range(10)) -out = fupdate(lst, (slice(0, 10, 2), slice(1, 10, 2), 6), +tup = tuple(range(10)) +out = fupdate(tup, (slice(0, 10, 2), slice(1, 10, 2), 6), (tuple(repeat(2, 5)), tuple(repeat(3, 5)), 42)) -assert lst == tuple(range(10)) +assert tup == tuple(range(10)) assert out == (2, 3, 2, 3, 2, 3, 42, 3, 2, 3) ``` -Also mappings can be functionally updated: +##### `fupdate` and mappings + +Mappings can be functionally updated, too: ```python d1 = {'foo': 'bar', 'fruit': 'apple'} @@ -2378,7 +2403,9 @@ assert sorted(d2.items()) == [('foo', 'tavern'), ('fruit', 'apple')] For immutable mappings, ``fupdate`` supports ``frozendict`` (see below). Any other mapping is assumed mutable, and ``fupdate`` essentially just performs ``copy.copy()`` and then ``.update()``. -We can also functionally update a namedtuple: +##### `fupdate` and named tuples + +Named tuples can be functionally updated, too: ```python from collections import namedtuple @@ -2389,9 +2416,10 @@ assert a == A(17, 23) assert out == A(42, 23) ``` -Namedtuples export only a sequence interface, so they cannot be treated as mappings. +Named tuples export only a sequence interface, so they **cannot** be treated as mappings, even though their elements have names. + +Support for ``namedtuple`` uses an extra feature of ``fupdate``, which is available for custom classes, too. When constructing the output sequence, ``fupdate`` first checks whether the type of the input sequence has a ``._make()`` method, and if so, hands the iterable containing the final data to that to construct the output. Otherwise the regular constructor is called (and it must accept a single iterable). -Support for ``namedtuple`` requires an extra feature, which is available for custom classes, too. When constructing the output sequence, ``fupdate`` first checks whether the input type has a ``._make()`` method, and if so, hands the iterable containing the final data to that to construct the output. Otherwise the regular constructor is called (and it must accept a single iterable). ### ``view``: writable, sliceable view into a sequence From 5dec0ca1c237183f97e8cd4658744251e4d90e7e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 03:18:07 +0300 Subject: [PATCH 516/832] test with bare range too --- unpythonic/tests/test_fup.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/tests/test_fup.py b/unpythonic/tests/test_fup.py index c6522c31..827bb9ce 100644 --- a/unpythonic/tests/test_fup.py +++ b/unpythonic/tests/test_fup.py @@ -78,6 +78,9 @@ def runtests(): test[tup == (1, 2, 3, 4, 5)] test[out == (4, 3, 2, 1, 0)] + out = fupdate(tup, slice(None, None, -1), range(5)) # no tuple() needed + test[out == (4, 3, 2, 1, 0)] + with testset("multiple individual items"): tup = (1, 2, 3, 4, 5) out = fupdate(tup, (1, 2, 3), (17, 23, 42)) From b257a1cc00ff8c6daba3200876faa6f1213d7f19 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 03:21:11 +0300 Subject: [PATCH 517/832] styling --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index c98c615f..71b910be 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2283,7 +2283,7 @@ assert fup(tup)[0::2] << repeat(10) == (10, 2, 10, 4, 10) # infinite replacemen Currently only one *update specification* is supported in a single ``fup()``. The low-level ``fupdate`` function supports more; see below. -An *update specification* is a combination of **where** to update, and **what** to put there. The *where* part can be a single index or a slice. When it is a single index, the *what* is a single item; and when a slice, the *what* is a sequence or an iterable, which must contain at least as many items as are required to perform the update. (For details, see `fupdate` below.) +An *update specification* is a combination of **where** to update, and **what** to put there. The *where* part can be a single index or a slice. When it is a single index, the *what* is a single item; and when a slice, the *what* is a sequence or an iterable, which must contain at least as many items as are required to perform the update. For details, see `fupdate` below. The ``fup`` function is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. From abae15671393f28215e303565baecab222835963 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 16:43:54 +0300 Subject: [PATCH 518/832] improve view docs --- doc/features.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index 71b910be..85c9f3d8 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2450,7 +2450,7 @@ While ``fupdate`` lets you be more functional than Python otherwise allows, ``vi We store slice specs, not actual indices, so this works also if the underlying sequence undergoes length changes. -Slicing a view returns a new view. Slicing anything else will usually copy, because the object being sliced does, before we get control. To slice lazily, first view the sequence itself and then slice that. The initial no-op view is optimized away, so it won't slow down accesses. Alternatively, pass a ``slice`` object into the ``view`` constructor. +Slicing a view returns a new view. Slicing anything else will usually shallow-copy, because the object being sliced does, before we get control. To slice lazily, first view the sequence itself and then slice that. The initial no-op view is optimized away, so it won't slow down accesses. Alternatively, pass a ``slice`` object into the ``view`` constructor. The view can be efficiently iterated over. As usual, iteration assumes that no inserts/deletes in the underlying sequence occur during the iteration. @@ -2458,7 +2458,9 @@ Getting/setting an item (subscripting) checks whether the index cache needs upda The ``unpythonic.collections`` module also provides the ``SequenceView`` and ``MutableSequenceView`` abstract base classes; ``view`` is a ``MutableSequenceView``. -There is the read-only cousin ``roview``, which behaves the same except it has no ``__setitem__`` or ``reverse``. This can be useful for giving read-only access to an internal sequence. The constructor of the writable ``view`` checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. +There is also the read-only cousin ``roview``, which is like ``view``, except it has no ``__setitem__`` or ``reverse``. This can be useful for providing explicit read-only access to a sequence, when it is undesirable to have clients write into it. + +The constructor of the writable ``view`` checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. ### ``mogrify``: update a mutable container in-place From 3f7433da1e87c6dbcda696174efe61b130a03a5c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 16:44:35 +0300 Subject: [PATCH 519/832] 0.15.0: improve mogrify docs --- doc/features.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/features.md b/doc/features.md index 85c9f3d8..92c36ff7 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2467,11 +2467,11 @@ The constructor of the writable ``view`` checks that the input is not read-only **Changed in v0.14.3.** *`mogrify` now skips `nil`, actually making it useful for processing `ll` linked lists.* -Recurse on given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like ``map`` does. +Recurse on a given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like ``map`` does. If the container is a mapping, the function is applied to the values; keys are left untouched. -Unlike ``map`` and its cousins, only a single input container is supported. (Supporting multiple containers as input would require enforcing some compatibility constraints on their type and shape, since ``mogrify`` is not limited to sequences.) +Unlike ``map`` and its cousins, **``mogrify`` only supports a single input container**. Supporting multiple containers as input would require enforcing some compatibility constraints on their type and shape, because ``mogrify`` is not limited to sequences. ```python from unpythonic import mogrify @@ -2484,17 +2484,17 @@ assert lst2 is lst1 Containers are detected by checking for instances of ``collections.abc`` superclasses (also virtuals are ok). Supported abcs are ``MutableMapping``, ``MutableSequence``, ``MutableSet``, ``Mapping``, ``Sequence`` and ``Set``. Any value that does not match any of these is treated as an atom. Containers can be nested, with an arbitrary combination of the types supported. -For convenience, we introduce some special cases: +For convenience, we support some special cases: - - Any classes created by ``collections.namedtuple``, because they do not conform to the standard constructor API for a ``Sequence``. + - Any classes created by ``collections.namedtuple``; they do not conform to the standard constructor API for a ``Sequence``. - Thus, for (an immutable) ``Sequence``, we first check for the presence of a ``._make()`` method, and if found, use it as the constructor. Otherwise we use the regular constructor. + Thus, to support also named tuples: for any immutable ``Sequence``, we first check for the presence of a ``._make()`` method, and if found, use it as the constructor. Otherwise we use the regular constructor. - ``str`` is treated as an atom, although technically a ``Sequence``. - It doesn't conform to the exact same API (its constructor does not take an iterable), and often we don't want to treat strings as containers anyway. + It does not conform to the exact same API (its constructor does not take an iterable), and often one does not want to treat strings as containers anyway. - If you want to process strings, implement it in your function that is called by ``mogrify``. + If you want to process strings, implement it in your function that is called by ``mogrify``. You can e.g. `tuple(thestring)` and then call ``mogrify`` on that. - The ``box``, `ThreadLocalBox` and `Some` containers from ``unpythonic.collections``. Although the first two are mutable, their update is not conveniently expressible by the ``collections.abc`` APIs. From 7ae14607cc3d88917dd45a28e2bb79b69dc70daf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 16:45:13 +0300 Subject: [PATCH 520/832] markdown: use single backticks In markdown, it renders the same as double backticks, looks cleaner in the source, and is easier to type on a Finnish keyboard (where the backtick is behind shift on a dead key, requiring "shift+tick, space" to get *one* backtick). It also plays more nicely with smartparens-mode in Emacs. --- doc/features.md | 752 ++++++++++++++++++++++++------------------------ 1 file changed, 376 insertions(+), 376 deletions(-) diff --git a/doc/features.md b/doc/features.md index 92c36ff7..49b95060 100644 --- a/doc/features.md +++ b/doc/features.md @@ -20,72 +20,72 @@ The exception are the features marked **[M]**, which are primarily intended as a ### Features [**Bindings**](#bindings) -- [``let``, ``letrec``: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) **[M]** - - [``let``](#let) - - [``dlet``, ``blet``](#dlet-blet): *let-over-def*, like the classic let-over-lambda. - - [``letrec``](#letrec) +- [`let`, `letrec`: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) **[M]** + - [`let`](#let) + - [`dlet`, `blet`](#dlet-blet): *let-over-def*, like the classic let-over-lambda. + - [`letrec`](#letrec) - [Lispylet: alternative syntax](#lispylet-alternative-syntax) **[M]** -- [``env``: the environment](#env-the-environment) -- [``assignonce``](#assignonce), a relative of ``env``. -- [``dyn``: dynamic assignment](#dyn-dynamic-assignment) a.k.a. parameterize, special variables, fluid variables, "dynamic scoping". +- [`env`: the environment](#env-the-environment) +- [`assignonce`](#assignonce), a relative of `env`. +- [`dyn`: dynamic assignment](#dyn-dynamic-assignment) a.k.a. parameterize, special variables, fluid variables, "dynamic scoping". [**Containers**](#containers) -- [``frozendict``: an immutable dictionary](#frozendict-an-immutable-dictionary) +- [`frozendict`: an immutable dictionary](#frozendict-an-immutable-dictionary) - [`cons` and friends: pythonic lispy linked lists](#cons-and-friends-pythonic-lispy-linked-lists) -- [``box``: a mutable single-item container](#box-a-mutable-single-item-container) - - [``box``](#box) - - [``Some``](#some): immutable box, to explicitly indicate the presence of a value. - - [``ThreadLocalBox``](#threadlocalbox) -- [``Shim``: redirect attribute accesses](#shim-redirect-attribute-accesses) -- [Container utilities](#container-utilities): ``get_abcs``, ``in_slice``, ``index_in_slice`` - -[**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a ``lambda``). -- [``begin``: sequence side effects](#begin-sequence-side-effects) -- [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) **[M]** - - [``do``](#do) - - [``do0``](#do0) -- [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) - - [``pipe``](#pipe) - - [``piped``](#piped) - - [``lazy_piped``](#lazy_piped) +- [`box`: a mutable single-item container](#box-a-mutable-single-item-container) + - [`box`](#box) + - [`Some`](#some): immutable box, to explicitly indicate the presence of a value. + - [`ThreadLocalBox`](#threadlocalbox) +- [`Shim`: redirect attribute accesses](#shim-redirect-attribute-accesses) +- [Container utilities](#container-utilities): `get_abcs`, `in_slice`, `index_in_slice` + +[**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a `lambda`). +- [`begin`: sequence side effects](#begin-sequence-side-effects) +- [`do`: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) **[M]** + - [`do`](#do) + - [`do0`](#do0) +- [`pipe`, `piped`, `lazy_piped`: sequence functions](#pipe-piped-lazy_piped-sequence-functions) + - [`pipe`](#pipe) + - [`piped`](#piped) + - [`lazy_piped`](#lazy_piped) [**Batteries**](#batteries) missing from the standard library. - [**Batteries for functools**](#batteries-for-functools): `curry`, `compose`, `withself`, and more. - - [``memoize``](#memoize): a detailed explanation of the memoizer. - - [``curry``](#curry): a detailed explanation of the curry utility and its haskelly extra features. - - [``fix``: break infinite recursion cycles](#fix-break-infinite-recursion-cycles) + - [`memoize`](#memoize): a detailed explanation of the memoizer. + - [`curry`](#curry): a detailed explanation of the curry utility and its haskelly extra features. + - [`fix`: break infinite recursion cycles](#fix-break-infinite-recursion-cycles) - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. - [**Batteries for network programming**](#batteries-for-network-programming): message protocol, PTY/socket proxy, etc. - [`unpythonic.net.msg`](#unpythonic-net-msg): message protocol. -- [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) +- [`islice`: slice syntax support for `itertools.islice`](#islice-slice-syntax-support-for-itertoolsislice) - [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. -- [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. +- [`fup`: functional update; `ShadowedSequence`](#fup-functional-update-shadowedsequence): like `collections.ChainMap`, but for sequences. - [`fup`](#fup): the high-level syntactic sugar to update a sequence functionally. - [`fupdate`](#fupdate): the low-level workhorse. -- [``view``: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. -- [``mogrify``: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) -- [``s``, ``imathify``, ``gmathify``: lazy mathematical sequences with infix arithmetic](#s-imathify-gmathify-lazy-mathematical-sequences-with-infix-arithmetic) -- [``sym``, ``gensym``, ``Singleton``: symbols and singletons](#sym-gensym-Singleton-symbols-and-singletons) +- [`view`: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. +- [`mogrify`: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) +- [`s`, `imathify`, `gmathify`: lazy mathematical sequences with infix arithmetic](#s-imathify-gmathify-lazy-mathematical-sequences-with-infix-arithmetic) +- [`sym`, `gensym`, `Singleton`: symbols and singletons](#sym-gensym-Singleton-symbols-and-singletons) [**Control flow tools**](#control-flow-tools) -- [``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) -- [``looped``, ``looped_over``: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) -- [``gtrampolined``: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like ``itertools.chain``, but from inside a generator. -- [``catch``, ``throw``: escape continuations (ec)](#catch-throw-escape-continuations-ec) (as in [Lisp's `catch`/`throw`](http://www.gigamonkeys.com/book/the-special-operators.html), unlike C++ or Java) - - [``call_ec``: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's ``call/ec``. -- [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. -- [``handlers``, ``restarts``: conditions and restarts](#handlers-restarts-conditions-and-restarts), a.k.a. **resumable exceptions**. -- [``generic``, ``typed``, ``isoftype``: multiple dispatch](#generic-typed-isoftype-multiple-dispatch): create generic functions with type annotation syntax; also some friendly utilities. +- [`trampolined`, `jump`: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) +- [`looped`, `looped_over`: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) +- [`gtrampolined`: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like `itertools.chain`, but from inside a generator. +- [`catch`, `throw`: escape continuations (ec)](#catch-throw-escape-continuations-ec) (as in [Lisp's `catch`/`throw`](http://www.gigamonkeys.com/book/the-special-operators.html), unlike C++ or Java) + - [`call_ec`: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's `call/ec`. +- [`forall`: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. +- [`handlers`, `restarts`: conditions and restarts](#handlers-restarts-conditions-and-restarts), a.k.a. **resumable exceptions**. +- [`generic`, `typed`, `isoftype`: multiple dispatch](#generic-typed-isoftype-multiple-dispatch): create generic functions with type annotation syntax; also some friendly utilities. [**Exception tools**](#exception-tools) -- [``raisef``, ``tryf``: ``raise`` and ``try`` as functions](#raisef-tryf-raise-and-try-as-functions), useful inside a lambda. -- [``equip_with_traceback``](#equip-with-traceback), equip a manually created exception instance with a traceback. -- [``async_raise``: inject an exception to another thread](#async_raise-inject-an-exception-to-another-thread) *(CPython only)* +- [`raisef`, `tryf`: `raise` and `try` as functions](#raisef-tryf-raise-and-try-as-functions), useful inside a lambda. +- [`equip_with_traceback`](#equip-with-traceback), equip a manually created exception instance with a traceback. +- [`async_raise`: inject an exception to another thread](#async_raise-inject-an-exception-to-another-thread) *(CPython only)* - [`reraise_in`, `reraise`: automatically convert exception types](#reraise_in-reraise-automatically-convert-exception-types) [**Function call and return value tools**](#function-call-and-return-value-tools) -- [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. -- [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) +- [`def` as a code block: `@call`](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. +- [`@callwith`: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) - [`Values`: multiple and named return values](#values-multiple-and-named-return-values) - [`valuify`](#valuify): convert pythonic multiple-return-values idiom of `tuple` into `Values`. @@ -93,17 +93,17 @@ The exception are the features marked **[M]**, which are primarily intended as a - [`almosteq`: floating-point almost-equality](#almosteq-floating-point-almost-equality) - [`fixpoint`: arithmetic fixed-point finder](#fixpoint-arithmetic-fixed-point-finder) - [`partition_int`, `partition_int_triangular`: partition integers](#partition_int-partition_int_triangular-partition-integers) - - [``ulp``: unit in last place](#ulp-unit-in-last-place) + - [`ulp`: unit in last place](#ulp-unit-in-last-place) [**Other**](#other) -- [``callsite_filename``](#callsite-filename) -- [``safeissubclass``](#safeissubclass), convenience function. -- [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) -- [``namelambda``: rename a function](#namelambda-rename-a-function) -- [``timer``: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) -- [``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) -- [``arities``, ``kwargs``, ``resolve_bindings``: Function signature inspection utilities](#arities-kwargs-resolve_bindings-function-signature-inspection-utilities) -- [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) +- [`callsite_filename`](#callsite-filename) +- [`safeissubclass`](#safeissubclass), convenience function. +- [`pack`: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) +- [`namelambda`: rename a function](#namelambda-rename-a-function) +- [`timer`: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) +- [`getattrrec`, `setattrrec`: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) +- [`arities`, `kwargs`, `resolve_bindings`: Function signature inspection utilities](#arities-kwargs-resolve_bindings-function-signature-inspection-utilities) +- [`Popper`: a pop-while iterator](#popper-a-pop-while-iterator) For many examples, see [the unit tests](unpythonic/tests/), the docstrings of the individual features, and this guide. @@ -115,15 +115,15 @@ For many examples, see [the unit tests](unpythonic/tests/), the docstrings of th Tools to bind identifiers in ways not ordinarily supported by Python. -### ``let``, ``letrec``: local bindings in an expression +### `let`, `letrec`: local bindings in an expression -**NOTE**: *This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API.* +**NOTE**: *This is primarily a code generation target API for the `let[]` family of [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API.* -The `let` constructs introduce bindings local to an expression, like Scheme's ``let`` and ``letrec``. +The `let` constructs introduce bindings local to an expression, like Scheme's `let` and `letrec`. -#### ``let`` +#### `let` -In ``let``, the bindings are independent (do not see each other). A binding is of the form ``name=value``, where ``name`` is a Python identifier, and ``value`` is any expression. +In `let`, the bindings are independent (do not see each other). A binding is of the form `name=value`, where `name` is a Python identifier, and `value` is any expression. Use a `lambda e: ...` to supply the environment to the body: @@ -140,7 +140,7 @@ u(L) # --> [1, 3, 2, 4] Generally speaking, `body` is a one-argument function, which takes in the environment instance as the first positional parameter (by convention, named `e` or `env`). In typical inline usage, `body` is `lambda e: expr`. -*Let over lambda*. Here the inner ``lambda`` is the definition of the function ``counter``: +*Let over lambda*. Here the inner `lambda` is the definition of the function `counter`: ```python from unpythonic import let, begin @@ -181,9 +181,9 @@ counter() ; --> 1 counter() ; --> 2 ``` -#### ``dlet``, ``blet`` +#### `dlet`, `blet` -*Let over def* decorator ``@dlet``, to *let over lambda* more pythonically: +*Let over def* decorator `@dlet`, to *let over lambda* more pythonically: ```python from unpythonic import dlet @@ -209,9 +209,9 @@ counter() # --> 1 counter() # --> 2 ``` -The ``@blet`` decorator is otherwise the same as ``@dlet``, but instead of decorating a function definition in the usual manner, it runs the `def` block immediately, and upon exit, replaces the function definition with the return value. The name ``blet`` is an abbreviation of *block let*, since the role of the `def` is just a code block to be run immediately. +The `@blet` decorator is otherwise the same as `@dlet`, but instead of decorating a function definition in the usual manner, it runs the `def` block immediately, and upon exit, replaces the function definition with the return value. The name `blet` is an abbreviation of *block let*, since the role of the `def` is just a code block to be run immediately. -#### ``letrec`` +#### `letrec` The name of this construct comes from the Scheme family of Lisps, and stands for *let (mutually) recursive*. The "[mutually recursive](https://en.wikipedia.org/wiki/Mutual_recursion)" refers to the kind of scoping between the bindings in the same `letrec`. @@ -239,11 +239,11 @@ x = letrec[[a << 1, b] ``` -In the non-macro `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent misinterpretation by the machinery when the environment initialization procedure runs. +In the non-macro `letrec`, the `value` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form `lambda e: valexpr`, providing access to the environment as `e`. If `valexpr` itself is callable, the binding **must** have the `lambda e: ...` wrapper to prevent misinterpretation by the machinery when the environment initialization procedure runs. -In a non-callable ``valexpr``, trying to depend on a binding below it raises ``AttributeError``. +In a non-callable `valexpr`, trying to depend on a binding below it raises `AttributeError`. -A callable ``valexpr`` may depend on any bindings (**also later ones**) in the same `letrec`. For example, here is a pair of [mutually recursive](https://en.wikipedia.org/wiki/Mutual_recursion) functions: +A callable `valexpr` may depend on any bindings (**also later ones**) in the same `letrec`. For example, here is a pair of [mutually recursive](https://en.wikipedia.org/wiki/Mutual_recursion) functions: ```python from unpythonic import letrec @@ -299,16 +299,16 @@ u = lambda lst: letrec[[seen << set(), (*The double brackets around the `letrec` body are needed because brackets denote a multiple-expression `letrec` body. So it is a multiple-expression body that contains just one expression, which is a list comprehension.*) -The decorators ``@dletrec`` and ``@bletrec`` work otherwise exactly like ``@dlet`` and ``@blet``, respectively, but the bindings are scoped like in ``letrec`` (mutually recursive scope). +The decorators `@dletrec` and `@bletrec` work otherwise exactly like `@dlet` and `@blet`, respectively, but the bindings are scoped like in `letrec` (mutually recursive scope). #### Lispylet: alternative syntax -**NOTE**: *This is primarily a code generation target API for the ``let[]`` family of [macros](macros.md), which make the constructs easier to use. Below is the documentation for the raw API.* +**NOTE**: *This is primarily a code generation target API for the `let[]` family of [macros](macros.md), which make the constructs easier to use. Below is the documentation for the raw API.* The `lispylet` module was originally created to allow guaranteed left-to-right initialization of `letrec` bindings in Pythons older than 3.6, hence the positional syntax and more parentheses. The only difference is the syntax; the behavior is identical with the other implementation. As of 0.15, the main role of `lispylet` is to act as the run-time backend for the `let` family of macros. -These constructs are available in the top-level `unpythonic` namespace, with the ``ordered_`` prefix: ``ordered_let``, ``ordered_letrec``, ``ordered_dlet``, ``ordered_dletrec``, ``ordered_blet``, ``ordered_bletrec``. +These constructs are available in the top-level `unpythonic` namespace, with the `ordered_` prefix: `ordered_let`, `ordered_letrec`, `ordered_dlet`, `ordered_dletrec`, `ordered_blet`, `ordered_bletrec`. It is also possible to override the default `let` constructs by the `ordered_` variants, like this: @@ -352,11 +352,11 @@ letrec[[evenp << (lambda x: (*The transformations made by the macros may be the most apparent when comparing these examples. Note that the macros scope the `let` bindings lexically, automatically figuring out which `let` environment, if any, to refer to.*) -### ``env``: the environment +### `env`: the environment -The environment used by all the ``let`` constructs and ``assignonce`` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. It is somewhat similar to [`types.SimpleNamespace`](https://docs.python.org/3/library/types.html#types.SimpleNamespace), but with many extra features. For details, see `unpythonic.env`. +The environment used by all the `let` constructs and `assignonce` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. It is somewhat similar to [`types.SimpleNamespace`](https://docs.python.org/3/library/types.html#types.SimpleNamespace), but with many extra features. For details, see `unpythonic.env`. -Our ``env`` allows things like: +Our `env` allows things like: ```python let(x=1, y=2, z=3, @@ -392,10 +392,10 @@ When the `with` block exits, the environment clears itself. The environment inst (This allows using `with env(...) as e:` as a poor man's `let`, if you have a block of statements you want to locally scope some names to, but don't want to introduce a `def`.) -``env`` provides the ``collections.abc.Mapping`` and ``collections.abc.MutableMapping`` APIs. +`env` provides the `collections.abc.Mapping` and `collections.abc.MutableMapping` APIs. -### ``assignonce`` +### `assignonce` *As of v0.15.0, `assignonce` is mostly a standalone curiosity that has never been integrated with the rest of `unpythonic`. But anything that works with arbitrary subclasses of `env`, for example `mogrify`, works with it, too.* @@ -411,14 +411,14 @@ with assignonce() as e: e.foo = "quux" # AttributeError, e.foo already defined. ``` -The `assignonce` construct is a subclass of ``env``, so it shares most of the same [features](#env-the-environment) and allows similar usage. +The `assignonce` construct is a subclass of `env`, so it shares most of the same [features](#env-the-environment) and allows similar usage. #### Historical note The fact that in Python creating bindings and updating (rebinding) them look the same was already noted in 2000, in [PEP 227](https://www.python.org/dev/peps/pep-0227/#discussion), which introduced true closures to Python 2.1. For related history concerning the `nonlocal` keyword, see [PEP 3104](https://www.python.org/dev/peps/pep-3104/). -### ``dyn``: dynamic assignment +### `dyn`: dynamic assignment **Changed in v0.14.2.** *To bring this in line with [SRFI-39](https://srfi.schemers.org/srfi-39/srfi-39.html), `dyn` now supports rebinding, using assignment syntax such as `dyn.x = 42`, and the function `dyn.update(x=42, y=17, ...)`.* @@ -460,9 +460,9 @@ Dynvars are created using `with dyn.let(k0=v0, ...)`. The syntax is in line with The point of dynamic assignment is that dynvars are seen also by code that is *outside the lexical scope* where the `with dyn.let` resides. The use case is to avoid a function parameter definition cascade, when you need to pass some information through several layers that do not care about it. This is especially useful for passing "background" information, such as plotter settings in scientific visualization, or the macro expander instance in metaprogramming. -To give a dynvar a top-level default value, use ``make_dynvar(k0=v0, ...)``. Usually this is done at the top-level scope of the module for which that dynvar is meaningful. Each dynvar, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). +To give a dynvar a top-level default value, use `make_dynvar(k0=v0, ...)`. Usually this is done at the top-level scope of the module for which that dynvar is meaningful. Each dynvar, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a `main()` for tests, and the file gets loaded both as a module and as the main program). -To rebind existing dynvars, use `dyn.k = v`, or `dyn.update(k0=v0, ...)`. Rebinding occurs in the closest enclosing dynamic environment that has the target name bound. If the name is not bound in any dynamic environment (including the top-level one), ``AttributeError`` is raised. +To rebind existing dynvars, use `dyn.k = v`, or `dyn.update(k0=v0, ...)`. Rebinding occurs in the closest enclosing dynamic environment that has the target name bound. If the name is not bound in any dynamic environment (including the top-level one), `AttributeError` is raised. **CAUTION**: Use rebinding of dynvars carefully, if at all. Stealth updates of dynvars defined in an enclosing dynamic extent can destroy any chance of statically reasoning about your code. @@ -474,18 +474,18 @@ A newly spawned thread automatically copies the then-current state of the dynami The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, a PR is welcome!) -Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (at any time after the global definition has been executed). +Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when `dyn` is queried for the value outside the dynamic extent of any `with dyn.let()` blocks. Having a default value is convenient for eliminating the need for `if "x" in dyn` checks, since the variable will always exist (at any time after the global definition has been executed).
-For more details, see the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a *live view* to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. ``dyn`` also provides the ``collections.abc.Mapping`` API. +For more details, see the methods of `dyn`; particularly noteworthy are `asdict` and `items`, which give access to a *live view* to dyn's contents in a dictionary format (intended for reading only!). The `asdict` method essentially creates a `collections.ChainMap` instance, while `items` is an abbreviation for `asdict().items()`. The `dyn` object itself can also be iterated over; this creates a `ChainMap` instance and redirects to iterate over it. `dyn` also provides the `collections.abc.Mapping` API. -To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, to allow things like: +To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; `dyn["x"]` has the same meaning as `dyn.x`, to allow things like: ```python print(tuple((k, dyn[k]) for k in dyn)) ``` -Finally, ``dyn`` supports membership testing as ``"x" in dyn``, ``"y" not in dyn``, where the string is the name of the dynvar whose presence is being tested. +Finally, `dyn` supports membership testing as `"x" in dyn`, `"y" not in dyn`, where the string is the name of the dynvar whose presence is being tested. For some more details, see [the unit tests](../unpythonic/tests/test_dynassign.py). @@ -506,11 +506,11 @@ We provide some additional low-level containers beyond those provided by Python The class names are lowercase, because these are intended as low-level utility classes in principle on par with the builtins. The immutable containers are hashable. All containers are pickleable (if their contents are). -### ``frozendict``: an immutable dictionary +### `frozendict`: an immutable dictionary **Changed in 0.14.2**. *[A bug in `frozendict` pickling](https://github.com/Technologicat/unpythonic/issues/55) has been fixed. Now also the empty `frozendict` pickles and unpickles correctly.* -Given the existence of ``dict`` and ``frozenset``, this one is oddly missing from the language. +Given the existence of `dict` and `frozenset`, this one is oddly missing from the language. ```python from unpythonic import frozendict @@ -536,7 +536,7 @@ assert d4['a'] == 23 and d4['b'] == 2 assert d3['a'] == 42 and d3['b'] == 2 # ...of course without touching the original ``` -Any mappings used when creating an instance are shallow-copied, so that the bindings of the ``frozendict`` do not change even if the original input is later mutated: +Any mappings used when creating an instance are shallow-copied, so that the bindings of the `frozendict` do not change even if the original input is later mutated: ```python d = {1:2, 3:4} @@ -567,7 +567,7 @@ assert d7.get(5, 0) == 0 assert d7.get(5) is None ``` -In terms of ``collections.abc``, a ``frozendict`` is a hashable immutable mapping: +In terms of `collections.abc`, a `frozendict` is a hashable immutable mapping: ```python assert issubclass(frozendict, Mapping) @@ -578,9 +578,9 @@ assert hash(d7) == hash(frozendict({1:2, 3:4})) assert hash(d7) != hash(frozendict({1:2})) ``` -The abstract superclasses are virtual, just like for ``dict``. We mean *virtual* in the sense of [`abc.ABCMeta`](https://docs.python.org/3/library/abc.html#abc.ABCMeta), i.e. a virtual superclass does not appear in the MRO. +The abstract superclasses are virtual, just like for `dict`. We mean *virtual* in the sense of [`abc.ABCMeta`](https://docs.python.org/3/library/abc.html#abc.ABCMeta), i.e. a virtual superclass does not appear in the MRO. -Finally, ``frozendict`` obeys the empty-immutable-container singleton invariant: +Finally, `frozendict` obeys the empty-immutable-container singleton invariant: ```python assert frozendict() is frozendict() @@ -628,13 +628,13 @@ assert lzip(ll(1, 2, 3), ll(4, 5, 6)) == ll(ll(1, 4), ll(2, 5), ll(3, 6)) Cons cells are immutable à la Racket (no `set-car!`/`rplaca`, `set-cdr!`/`rplacd`). Accessors are provided up to `caaaar`, ..., `cddddr`. -Although linked lists are created with the functions ``ll`` or ``llist``, the data type (for e.g. ``isinstance``) is ``cons``. +Although linked lists are created with the functions `ll` or `llist`, the data type (for e.g. `isinstance`) is `cons`. -Iterators are supported, to walk over linked lists. This also gives sequence unpacking support. When ``next()`` is called, we return the `car` of the current cell the iterator points to, and the iterator moves to point to the cons cell in the `cdr`, if any. When the `cdr` is not a cons cell, it is the next (and last) item returned; except if it `is nil`, then iteration ends without returning the `nil`. +Iterators are supported, to walk over linked lists. This also gives sequence unpacking support. When `next()` is called, we return the `car` of the current cell the iterator points to, and the iterator moves to point to the cons cell in the `cdr`, if any. When the `cdr` is not a cons cell, it is the next (and last) item returned; except if it `is nil`, then iteration ends without returning the `nil`. -Python's builtin ``reversed`` can be applied to linked lists; it will internally ``lreverse`` the list (which is O(n)), then return an iterator to that. The ``llist`` constructor is special-cased so that if the input is ``reversed(some_ll)``, it just returns the internal already reversed list. (This is safe because cons cells are immutable.) +Python's builtin `reversed` can be applied to linked lists; it will internally `lreverse` the list (which is O(n)), then return an iterator to that. The `llist` constructor is special-cased so that if the input is `reversed(some_ll)`, it just returns the internal already reversed list. (This is safe because cons cells are immutable.) -Cons structures, by default, print in a pythonic format suitable for ``eval`` (if all elements are): +Cons structures, by default, print in a pythonic format suitable for `eval` (if all elements are): ```python print(cons(1, 2)) # --> cons(1, 2) @@ -650,24 +650,24 @@ print(ll(1, 2, 3).lispyrepr()) # --> (1 2 3) print(cons(cons(1, 2), cons(3, 4)).lispyrepr()) # --> ((1 . 2) . (3 . 4)) ``` -For more, see the ``llist`` submodule. +For more, see the `llist` submodule. #### Notes -There is no ``copy`` method or ``lcopy`` function, because cons cells are immutable; which makes cons structures immutable. +There is no `copy` method or `lcopy` function, because cons cells are immutable; which makes cons structures immutable. -However, for example, it is possible to ``cons`` a new item onto an existing linked list; that is fine, because it produces a new cons structure - which shares data with the original, just like in Racket. +However, for example, it is possible to `cons` a new item onto an existing linked list; that is fine, because it produces a new cons structure - which shares data with the original, just like in Racket. In general, copying cons structures can be error-prone. Given just a starting cell it is impossible to tell if a given instance of a cons structure represents a linked list, or something more general (such as a binary tree) that just happens to locally look like one, along the path that would be traversed if it was indeed a linked list. -The linked list iteration strategy does not recurse in the ``car`` half, which could lead to incomplete copying. The tree strategy that recurses on both halves, on the other hand, will flatten nested linked lists and produce also the final ``nil``. +The linked list iteration strategy does not recurse in the `car` half, which could lead to incomplete copying. The tree strategy that recurses on both halves, on the other hand, will flatten nested linked lists and produce also the final `nil`. -We provide a ``JackOfAllTradesIterator`` as a compromise that understands both trees and linked lists. Nested lists will be flattened, and in a tree any ``nil`` in a ``cdr`` position will be omitted from the output. ``BinaryTreeIterator`` and ``JackOfAllTradesIterator`` use an explicit data stack instead of implicitly using the call stack for keeping track of the recursion. All ``cons`` iterators work for arbitrarily deep cons structures without causing Python's call stack to overflow, and without the need for TCO. +We provide a `JackOfAllTradesIterator` as a compromise that understands both trees and linked lists. Nested lists will be flattened, and in a tree any `nil` in a `cdr` position will be omitted from the output. `BinaryTreeIterator` and `JackOfAllTradesIterator` use an explicit data stack instead of implicitly using the call stack for keeping track of the recursion. All `cons` iterators work for arbitrarily deep cons structures without causing Python's call stack to overflow, and without the need for TCO. -``cons`` has no ``collections.abc`` virtual superclasses (except the implicit ``Hashable`` since ``cons`` provides ``__hash__`` and ``__eq__``), because general cons structures do not fit into the contracts represented by membership in those classes. For example, size cannot be known without iterating, and depends on which iteration scheme is used (e.g. ``nil`` dropping, flattening); which scheme is appropriate depends on the content. +`cons` has no `collections.abc` virtual superclasses (except the implicit `Hashable` since `cons` provides `__hash__` and `__eq__`), because general cons structures do not fit into the contracts represented by membership in those classes. For example, size cannot be known without iterating, and depends on which iteration scheme is used (e.g. `nil` dropping, flattening); which scheme is appropriate depends on the content. -### ``box``: a mutable single-item container +### `box`: a mutable single-item container **Changed in v0.14.2**. *The `box` container API is now `b.set(newvalue)` to rebind, returning the new value as a convenience. The equivalent syntactic sugar is `b << newvalue`. The item inside the box can be extracted with `b.get()`. The equivalent syntactic sugar is `unbox(b)`.* @@ -677,7 +677,7 @@ We provide a ``JackOfAllTradesIterator`` as a compromise that understands both t **Changed in v0.14.2**. *Accessing the `.x` attribute of a `box` directly is now deprecated. It will continue to work with `box` at least until 0.15, but it does not and cannot work with `ThreadLocalBox`, which must handle things differently due to implementation reasons. Use the API mentioned above; it supports both kinds of boxes with the same syntax.* -#### ``box`` +#### `box` Consider this highly artificial example: @@ -691,9 +691,9 @@ f(animal) assert animal == "dog" ``` -Many solutions exist. Common pythonic ones are abusing a ``list`` to represent a box (and then trying to remember that it is supposed to hold only a single item), or (if the lexical structure of the particular piece of code allows it) using the ``global`` or ``nonlocal`` keywords to tell Python, on assignment, to overwrite a name that already exists in a surrounding scope. +Many solutions exist. Common pythonic ones are abusing a `list` to represent a box (and then trying to remember that it is supposed to hold only a single item), or (if the lexical structure of the particular piece of code allows it) using the `global` or `nonlocal` keywords to tell Python, on assignment, to overwrite a name that already exists in a surrounding scope. -As an alternative to the rampant abuse of lists, we provide a rackety ``box``, which is a minimalistic mutable container that holds exactly one item. Any code that has a reference to the box can update the data in it: +As an alternative to the rampant abuse of lists, we provide a rackety `box`, which is a minimalistic mutable container that holds exactly one item. Any code that has a reference to the box can update the data in it: ```python from unpythonic import box, unbox @@ -725,7 +725,7 @@ f("dog") Here `g` *effectively rebinds a local variable of `f`* - whether that is a good idea is a separate question, but technically speaking, this would not be possible without a container. As mentioned, abusing a `list` is the standard Python (but not very pythonic!) solution. Using specifically a `box` makes the intent explicit. -The ``box`` API is summarized by: +The `box` API is summarized by: ```python from unpythonic import box, unbox @@ -758,13 +758,13 @@ box3.set("fox") # same without syntactic sugar assert "fox" in box3 ``` -The expression ``item in b`` has the same meaning as ``unbox(b) == item``. Note ``box`` is a **mutable container**, so it is **not hashable**. +The expression `item in b` has the same meaning as `unbox(b) == item`. Note `box` is a **mutable container**, so it is **not hashable**. The expression `unbox(b)` has the same meaning as `b.get()`, but because it is a function (instead of a method), it additionally sanity-checks that `b` is a box, and if not, raises `TypeError`. The expression `b << newitem` has the same meaning as `b.set(newitem)`. In both cases, the new value is returned as a convenience. -#### ``Some`` +#### `Some` We also provide an **immutable** box, `Some`. This can be useful to represent optional data. @@ -772,7 +772,7 @@ The idea is that the value, when present, is placed into a `Some`, such as `Some (It is like the `Some` constructor of a `Maybe` monad, but with no monadic magic. In this interpretation, the bare constant `None` plays the role of `Nothing`.) -#### ``ThreadLocalBox`` +#### `ThreadLocalBox` `ThreadLocalBox` is otherwise exactly like `box`, but magical: its contents are thread-local. It also holds a default object, which is set initially when the `ThreadLocalBox` is instantiated. The default object is seen by threads that have not placed any object into the box. @@ -832,7 +832,7 @@ assert unbox(tlb) == "cat" # ...this thread sees the current default object aga ``` -### ``Shim``: redirect attribute accesses +### `Shim`: redirect attribute accesses **Added in v0.14.2**. @@ -951,7 +951,7 @@ from unpythonic import get_abcs print(get_abcs(list)) ``` -This includes virtual superclasses, i.e. those that are not part of the MRO. This works by ``issubclass(cls, v)`` on all classes defined in ``collections.abc``. +This includes virtual superclasses, i.e. those that are not part of the MRO. This works by `issubclass(cls, v)` on all classes defined in `collections.abc`. **Reflection on slices**: @@ -974,10 +974,10 @@ Sequencing refers to running multiple expressions, in sequence, in place of one Keep in mind the only reason to ever need multiple expressions: *side effects.* Assignment is a side effect, too; it modifies the environment. In functional style, intermediate named definitions to increase readability are perhaps the most useful kind of side effect. -See also ``multilambda`` in [macros](macros.md). +See also `multilambda` in [macros](macros.md). -### ``begin``: sequence side effects +### `begin`: sequence side effects **CAUTION**: the `begin` family of forms are provided **for use in pure-Python projects only**, and are a permanent part of the `unpythonic` API for that purpose. They are somewhat simpler and less flexible than the `do` family, described further below. @@ -1000,25 +1000,25 @@ The `begin` and `begin0` forms are actually tuples in disguise; evaluation of al We provide also `lazy_begin` and `lazy_begin0`, which use loops. The price is the need for a lambda wrapper for each expression to delay evaluation, see [`unpythonic.seq`](../unpythonic/seq.py) for details. -### ``do``: stuff imperative code into an expression +### `do`: stuff imperative code into an expression -**NOTE**: *This is primarily a code generation target API for the ``do[]`` and ``do0[]`` [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API.* +**NOTE**: *This is primarily a code generation target API for the `do[]` and `do0[]` [macros](macros.md), which make the constructs easier to use, and make the code look almost like normal Python. Below is the documentation for the raw API.* -Basically, the ``do`` family is a more advanced and flexible variant of the ``begin`` family. +Basically, the `do` family is a more advanced and flexible variant of the `begin` family. - - ``do`` can bind names to intermediate results and then use them in later items. + - `do` can bind names to intermediate results and then use them in later items. - - ``do`` is effectively a ``let*`` (technically, ``letrec``) where making a binding is optional, so that some items can have only side effects if so desired. There is no semantically distinct ``body``; all items play the same role. + - `do` is effectively a `let*` (technically, `letrec`) where making a binding is optional, so that some items can have only side effects if so desired. There is no semantically distinct `body`; all items play the same role. - Despite the name, there is no monadic magic. -Like in ``letrec``, use ``lambda e: ...`` to access the environment, and to wrap callable values (to prevent misinterpretation by the machinery). +Like in `letrec`, use `lambda e: ...` to access the environment, and to wrap callable values (to prevent misinterpretation by the machinery). -Unlike ``begin`` (and ``begin0``), there is no separate ``lazy_do`` (``lazy_do0``), because using a ``lambda e: ...`` wrapper for an item will already delay its evaluation; and the main point of ``do``/``do0`` is that there is an environment that holds local definitions. If you want a lazy variant, just wrap each item with a ``lambda e: ...``, also those that don't otherwise need it. +Unlike `begin` (and `begin0`), there is no separate `lazy_do` (`lazy_do0`), because using a `lambda e: ...` wrapper for an item will already delay its evaluation; and the main point of `do`/`do0` is that there is an environment that holds local definitions. If you want a lazy variant, just wrap each item with a `lambda e: ...`, also those that don't otherwise need it. -#### ``do`` +#### `do` -Like ``begin`` and ``lazy_begin``, the ``do`` form evaluates all items in order, and then returns the value of the **last** item. +Like `begin` and `lazy_begin`, the `do` form evaluates all items in order, and then returns the value of the **last** item. ```python from unpythonic import do, assign @@ -1066,7 +1066,7 @@ y = do[local[x << 5], assert y == 25 ``` -*In the macro version, all items are delayed automatically; that is, **every** item has an implicit ``lambda e: ...``. Note that instead of the `assign` function, the macro version uses the syntax ``local[name << value]`` to **create** an expression-local variable. Updating an existing variable in the `do` environment is just ``name << value``. Finally, there is also ``delete[name]``.* +*In the macro version, all items are delayed automatically; that is, **every** item has an implicit `lambda e: ...`. Note that instead of the `assign` function, the macro version uses the syntax `local[name << value]` to **create** an expression-local variable. Updating an existing variable in the `do` environment is just `name << value`. Finally, there is also `delete[name]`.* When using the raw API, beware of this pitfall: @@ -1080,7 +1080,7 @@ do(lambda e: print("hello 2 from 'do'"), # delayed because lambda e: ... # for do(). ``` -The above pitfall also applies to using escape continuations inside a ``do``. To do that, wrap the ec call into a ``lambda e: ...`` to delay its evaluation until the ``do`` actually runs: +The above pitfall also applies to using escape continuations inside a `do`. To do that, wrap the ec call into a `lambda e: ...` to delay its evaluation until the `do` actually runs: ```python from unpythonic import call_ec, do, assign @@ -1092,7 +1092,7 @@ call_ec( lambda e: print("never reached"))) # and this (as above) ``` -This way, any assignments made in the ``do`` (which occur only after ``do`` gets control), performed above the line with the ``ec`` call, will have been performed when the ``ec`` is called. +This way, any assignments made in the `do` (which occur only after `do` gets control), performed above the line with the `ec` call, will have been performed when the `ec` is called. For comparison, with the macro API, the last example becomes: @@ -1107,11 +1107,11 @@ call_ec( print("never reached")]) ``` -*In the macro version, all items are delayed automatically, so there ``do``/``do0`` gets control before any items are evaluated. The `ec` fires when the `do` evaluates that item, and the `print` is indeed never reached.* +*In the macro version, all items are delayed automatically, so there `do`/`do0` gets control before any items are evaluated. The `ec` fires when the `do` evaluates that item, and the `print` is indeed never reached.* -#### ``do0`` +#### `do0` -Like ``begin0`` and ``lazy_begin0``, the ``do0`` form evaluates all items in order, and then returns the value of the **first** item. +Like `begin0` and `lazy_begin0`, the `do0` form evaluates all items in order, and then returns the value of the **first** item. It effectively does this internally: @@ -1162,7 +1162,7 @@ assert y == 17 ``` -### ``pipe``, ``piped``, ``lazy_piped``: sequence functions +### `pipe`, `piped`, `lazy_piped`: sequence functions **Changed in v0.15.0.** *Multiple return values and named return values, for unpacking to the args and kwargs of the next function in the pipe, as well as in the final return value from the pipe, are now represented as a `Values`.* @@ -1174,13 +1174,13 @@ assert y == 17 Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/), but no macros. A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It is just function composition, but with an emphasis on data flow, which helps improve readability. -Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with ``1``, and they are slightly faster than the general versions. The use case is one-argument functions that return one value. +Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with `1`, and they are slightly faster than the general versions. The use case is one-argument functions that return one value. In the n-to-m versions, when a function returns a `Values`, it is unpacked to the args and kwargs of the next function in the pipeline. When a pipe exits, the `Values` wrapper (if any) around the final result is discarded if it contains only one positional value. The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as the args/kwargs of each output `Values` can be accepted as input by the next function in the pipe). Additional examples can be found in [the unit tests](../unpythonic/tests/test_seq.py). -#### ``pipe`` +#### `pipe` The function `pipe` represents a self-contained pipeline that starts from a given value (or values), applies some operations in sequence, and then exits: @@ -1207,11 +1207,11 @@ assert (a, b) == (6, 7) In this example, we pass the initial values positionally into the first function in the pipeline; that function passes its return values by name; and the second function in the pipeline passes the final results positionally. Because there are only positional values in the final `Values` object, it can be unpacked like a tuple. -#### ``pipec`` +#### `pipec` -The function ``pipec`` is otherwise exactly like ``pipe``, but it curries the functions before applying them. This is useful with the passthrough feature of ``curry``. +The function `pipec` is otherwise exactly like `pipe`, but it curries the functions before applying them. This is useful with the passthrough feature of `curry`. -With ``pipec`` you can do things like: +With `pipec` you can do things like: ```python from unpythonic import pipec, Values @@ -1222,15 +1222,15 @@ a, b = pipec(Values(1, 2), assert (a, b) == (4, 3) ``` -For more on passthrough, see the section on ``curry``. +For more on passthrough, see the section on `curry`. -#### ``piped`` +#### `piped` We also provide a **shell-like syntax**, with purely functional updates. -To set up a pipeline for use with the shell-like syntax, call ``piped`` to load the initial value(s). It is possible to provide both positional and named values. Each use of the pipe operator applies the given function, but keeps the result inside the pipeline, ready to accept another function. +To set up a pipeline for use with the shell-like syntax, call `piped` to load the initial value(s). It is possible to provide both positional and named values. Each use of the pipe operator applies the given function, but keeps the result inside the pipeline, ready to accept another function. -When done, pipe into the sentinel ``exitpipe`` to exit the pipeline and return the current value(s): +When done, pipe into the sentinel `exitpipe` to exit the pipeline and return the current value(s): ```python from unpythonic import piped, exitpipe @@ -1243,11 +1243,11 @@ assert p | inc | exitpipe == 85 assert p | exitpipe == 84 # p itself is never modified by the pipe system ``` -Multiple values work like in `pipe`, except the initial value(s) passed to ``piped`` are automatically packed into a `Values`. The pipe system then automatically unpacks a `Values` object into the args/kwargs of the next function in the pipeline. +Multiple values work like in `pipe`, except the initial value(s) passed to `piped` are automatically packed into a `Values`. The pipe system then automatically unpacks a `Values` object into the args/kwargs of the next function in the pipeline. To return multiple positional values and/or named values, return a `Values` object from your function. -When ``exitpipe`` is applied, if the last function returned anything other than one positional value, you will get a ``Values`` object. +When `exitpipe` is applied, if the last function returned anything other than one positional value, you will get a `Values` object. ```python from unpythonic import piped, exitpipe, Values @@ -1267,9 +1267,9 @@ a, b = piped(2, 3) | f | g | exitpipe # --> (5, 8) assert (a, b) == (5, 8) ``` -#### ``lazy_piped`` +#### `lazy_piped` -Lazy pipes are useful when you have mutable initial values. To perform the planned computation, pipe into the sentinel ``exitpipe``: +Lazy pipes are useful when you have mutable initial values. To perform the planned computation, pipe into the sentinel `exitpipe`: ```python from unpythonic import lazy_piped1, exitpipe @@ -1322,8 +1322,8 @@ Things missing from the standard library. - `composel1`, `composer1`: 1-in-1-out chains (faster). - suffix `i` to use with an iterable that contains the functions (`composeli`, `composeri`, `composelci`, `composerci`, `composel1i`, `composer1i`) - `withself`: essentially, the Y combinator trick as a decorator. Allows a lambda to refer to itself. - - The ``self`` argument is declared explicitly, but passed implicitly (as the first positional argument), just like the ``self`` argument of a method. - - `apply`: the lispy approach to starargs. Mainly useful with the ``prefix`` [macro](macros.md). + - The `self` argument is declared explicitly, but passed implicitly (as the first positional argument), just like the `self` argument of a method. + - `apply`: the lispy approach to starargs. Mainly useful with the `prefix` [macro](macros.md). - `andf`, `orf`, `notf`: compose predicates (like Racket's `conjoin`, `disjoin`, `negate`). - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, `andf` and `orf` are now marked lazy. Arguments will be forced only when a lazy predicate in the chain actually uses them, or when an eager (not lazy) predicate is encountered in the chain.* - `flip`: reverse the order of positional arguments. @@ -1379,20 +1379,20 @@ clip = lambda n1, n2: composel(*with_n((n1, drop), (n2, take))) assert tuple(clip(5, 10)(range(20))) == tuple(range(5, 15)) ``` -In the last example, essentially we just want to `clip 5 10 (range 20)`, the grouping of the parentheses being pretty much an implementation detail. Using the passthrough in ``curry`` (more on which in the section on ``curry``, below), we can rewrite the last line as: +In the last example, essentially we just want to `clip 5 10 (range 20)`, the grouping of the parentheses being pretty much an implementation detail. Using the passthrough in `curry` (more on which in the section on `curry`, below), we can rewrite the last line as: ```python assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) ``` -#### ``memoize`` +#### `memoize` [*Memoization*](https://en.wikipedia.org/wiki/Memoization) is a functional programming technique, meant to be used with [pure functions](https://en.wikipedia.org/wiki/Pure_function). It caches the return value, so that *for each unique set of arguments*, the original function will be evaluated only once. All arguments must be hashable. -Our ``memoize`` caches also exceptions, à la the [Mischief package in Racket](https://docs.racket-lang.org/mischief/memoize.html). If the memoized function is called again with arguments with which it raised an exception the first time, **that same exception instance** is raised again. +Our `memoize` caches also exceptions, à la the [Mischief package in Racket](https://docs.racket-lang.org/mischief/memoize.html). If the memoized function is called again with arguments with which it raised an exception the first time, **that same exception instance** is raised again. -The decorator **works also on instance methods**, with results cached separately for each instance. This is essentially because ``self`` is an argument, and custom classes have a default ``__hash__``. Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. (This approach does have the expected problem: if lots of instances are created and destroyed, and a memoized method is called for each, the memo will grow without bound.) +The decorator **works also on instance methods**, with results cached separately for each instance. This is essentially because `self` is an argument, and custom classes have a default `__hash__`. Hence it doesn't matter that the memo lives in the `memoized` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of `self` will create unique entries in it. (This approach does have the expected problem: if lots of instances are created and destroyed, and a memoized method is called for each, the memo will grow without bound.) *For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)).* @@ -1434,7 +1434,7 @@ There are some **important differences** to the nearest equivalents in the stand return 3.0 * x ``` - Without using ``@generic``, the essential idea is: + Without using `@generic`, the essential idea is: ```python from unpythonic import memoize @@ -1499,9 +1499,9 @@ thunk() Some languages, such as Haskell, curry all functions natively. In languages that do not, like Python or [Racket](https://docs.racket-lang.org/reference/procedures.html#%28def._%28%28lib._racket%2Ffunction..rkt%29._curry%29%29), when currying is implemented as a library function, this is often done as a form of [partial application](https://en.wikipedia.org/wiki/Partial_application), which is a subtly different concept, but encompasses the curried behavior as a special case. In practice this means that you can pass several arguments in a single step, and the original function will be called when all parameters have been bound. -Our ``curry`` can be used both as a decorator and as a regular function. As a decorator, `curry` takes no decorator arguments. As a regular function, `curry` itself is curried à la Racket. If any args or kwargs are given (beside the function to be curried), they are the first step. This helps eliminate many parentheses. +Our `curry` can be used both as a decorator and as a regular function. As a decorator, `curry` takes no decorator arguments. As a regular function, `curry` itself is curried à la Racket. If any args or kwargs are given (beside the function to be curried), they are the first step. This helps eliminate many parentheses. -**CAUTION**: If the signature of ``f`` cannot be inspected, currying fails, raising ``ValueError``, like ``inspect.signature`` does. This may happen with builtins such as ``list.append``, ``operator.add``, ``print``, or ``range``, depending on which version of Python is used (and whether it is CPython or PyPy3). +**CAUTION**: If the signature of `f` cannot be inspected, currying fails, raising `ValueError`, like `inspect.signature` does. This may happen with builtins such as `list.append`, `operator.add`, `print`, or `range`, depending on which version of Python is used (and whether it is CPython or PyPy3). Like Haskell, and [`spicy` for Racket](https://github.com/Technologicat/spicy), our `curry` supports *passthrough*; but we pass through **both positional and named arguments**. @@ -1519,8 +1519,8 @@ Some finer points concerning the passthrough feature: - Extra named args are passed through by name. They may be overridden by named return values (with the same name) from the curried function. - - If more args/kwargs are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. - - To override this behavior, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. + - If more args/kwargs are still remaining when the top-level curry context exits, by default `TypeError` is raised. + - To override this behavior, set the dynvar `curry_context`. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. - To set the dynvar, `from unpythonic import dyn`, and then `with dyn.let(curry_context=["whatever"]):`. Examples: @@ -1566,13 +1566,13 @@ map_one = lambda f: composer(rmap_one(f), lreverse) assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) ``` -which may be a useful pattern for lengthy iterables that could overflow the call stack (although not in ``foldr``, since our implementation uses a linear process). +which may be a useful pattern for lengthy iterables that could overflow the call stack (although not in `foldr`, since our implementation uses a linear process). -In the example, in ``rmap_one``, we can use either ``curry`` or ``partial``. In this case it does not matter which, since we want just one partial application anyway. We provide two arguments, and the minimum arity of ``foldl`` is 3, so ``curry`` will trigger the call as soon as (and only as soon as) it gets at least one more argument. +In the example, in `rmap_one`, we can use either `curry` or `partial`. In this case it does not matter which, since we want just one partial application anyway. We provide two arguments, and the minimum arity of `foldl` is 3, so `curry` will trigger the call as soon as (and only as soon as) it gets at least one more argument. -The final ``curry`` in the example uses the passthrough features. The function ``map_one`` has arity 1, but two positional arguments are given. It also invokes a call to the callable returned by ``map_one``, with the remaining arguments (in this case just one, the ``ll(1, 2, 3)``). +The final `curry` in the example uses the passthrough features. The function `map_one` has arity 1, but two positional arguments are given. It also invokes a call to the callable returned by `map_one`, with the remaining arguments (in this case just one, the `ll(1, 2, 3)`). -Yet another way to write ``map_one`` is: +Yet another way to write `map_one` is: ```python from unpythonic import curry, foldr, composer, cons, nil @@ -1580,9 +1580,9 @@ from unpythonic import curry, foldr, composer, cons, nil mymap = lambda f: curry(foldr, composer(cons, curry(f)), nil) ``` -The curried ``f`` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; these two values then end up as the arguments to ``cons``. +The curried `f` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; these two values then end up as the arguments to `cons`. -Using a **currying compose function** (name suffixed with ``c``), we can drop the inner curry: +Using a **currying compose function** (name suffixed with `c`), we can drop the inner curry: ```python from unpythonic import curry, foldr, composerc, cons, nil @@ -1592,33 +1592,33 @@ myadd = lambda a, b: a + b assert curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9) ``` -This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy)) as we're gonna get in pure Python. +This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in `#lang` [`spicy`](https://github.com/Technologicat/spicy)) as we're gonna get in pure Python. -Notice how the last two versions accept multiple input iterables; this is thanks to currying ``f`` inside the composition. An element from each of the iterables is taken by the processing function ``f``. Being the last argument, ``acc`` is passed through on the right. The output from the processing function - one new item - and ``acc`` then become two arguments, passed into cons. +Notice how the last two versions accept multiple input iterables; this is thanks to currying `f` inside the composition. An element from each of the iterables is taken by the processing function `f`. Being the last argument, `acc` is passed through on the right. The output from the processing function - one new item - and `acc` then become two arguments, passed into cons. -Finally, keep in mind the `mymap` example is intended as a feature demonstration. In production code, the builtin ``map`` is much better. It produces a lazy iterable, so it does not care which kind of actual data structure the items will be stored in (once they are computed). In other words, a lazy iterable is a much better model for a process that produces a sequence of values; how, and whether, to store that sequence is an orthogonal concern. +Finally, keep in mind the `mymap` example is intended as a feature demonstration. In production code, the builtin `map` is much better. It produces a lazy iterable, so it does not care which kind of actual data structure the items will be stored in (once they are computed). In other words, a lazy iterable is a much better model for a process that produces a sequence of values; how, and whether, to store that sequence is an orthogonal concern. The example we have here evaluates all items immediately, and specifically produces a linked list. It is just a nice example of function composition involving incompatible positional arities, thus demonstrating the kind of situation where the passthrough feature of `curry` is useful. It is taken from a paper by [John Hughes (1984)](https://www.cse.chalmers.se/~rjmh/Papers/whyfp.html). -##### ``curry`` and reduction rules +##### `curry` and reduction rules -Our ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. Let's consider a simple example with positional arguments only. When we say: +Our `curry`, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. Let's consider a simple example with positional arguments only. When we say: ```python curry(f, a0, a1, ..., a[n-1]) ``` -it means the following. Let ``m1`` and ``m2`` be the minimum and maximum positional arity of the callable ``f``, respectively. +it means the following. Let `m1` and `m2` be the minimum and maximum positional arity of the callable `f`, respectively. - - If ``n > m2``, call ``f`` with the first ``m2`` arguments. + - If `n > m2`, call `f` with the first `m2` arguments. - If the result is a callable, curry it, and recurse. - - Else form a tuple, where first item is the result, and the rest are the remaining arguments ``a[m2]``, ``a[m2+1]``, ..., ``a[n-1]``. Return it. - - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. Use the dynvar ``curry_context`` to override; see above for an example. - - If ``m1 <= n <= m2``, call ``f`` and return its result (like a normal function call). - - **Any** positional arity accepted by ``f`` triggers the call; beware when working with [variadic](https://en.wikipedia.org/wiki/Variadic_function) functions. - - If ``n < m1``, partially apply ``f`` to the given arguments, yielding a new function with smaller ``m1``, ``m2``. Then curry the result and return it. - - Internally we stack ``functools.partial`` applications, but there will be only one ``curried`` wrapper no matter how many invocations are used to build up arguments before ``f`` eventually gets called. + - Else form a tuple, where first item is the result, and the rest are the remaining arguments `a[m2]`, `a[m2+1]`, ..., `a[n-1]`. Return it. + - If more positional args are still remaining when the top-level curry context exits, by default `TypeError` is raised. Use the dynvar `curry_context` to override; see above for an example. + - If `m1 <= n <= m2`, call `f` and return its result (like a normal function call). + - **Any** positional arity accepted by `f` triggers the call; beware when working with [variadic](https://en.wikipedia.org/wiki/Variadic_function) functions. + - If `n < m1`, partially apply `f` to the given arguments, yielding a new function with smaller `m1`, `m2`. Then curry the result and return it. + - Internally we stack `functools.partial` applications, but there will be only one `curried` wrapper no matter how many invocations are used to build up arguments before `f` eventually gets called. As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the presence of kwargs, `@generic` functions, and `Values` multiple-return-values (and named return values), is: @@ -1650,13 +1650,13 @@ Getting back to the simple case, in the above example: curry(mapl_one, double, ll(1, 2, 3)) ``` -the callable ``mapl_one`` takes one argument, which is a function. It returns another function, let us call it ``g``. We are left with: +the callable `mapl_one` takes one argument, which is a function. It returns another function, let us call it `g`. We are left with: ```python curry(g, ll(1, 2, 3)) ``` -The remaining argument is then passed into ``g``; we obtain a result, and reduction is complete. +The remaining argument is then passed into `g`; we obtain a result, and reduction is complete. A curried function is also a curry context: @@ -1672,13 +1672,13 @@ so on the last line, we do not need to say curry(a2, a, b, c) ``` -because ``a2`` is already curried. Doing so does no harm, though; ``curry`` automatically prevents stacking ``curried`` wrappers: +because `a2` is already curried. Doing so does no harm, though; `curry` automatically prevents stacking `curried` wrappers: ```python curry(a2) is a2 # --> True ``` -If we wish to modify precedence, parentheses are needed, which takes us out of the curry context, unless we explicitly ``curry`` the subexpression. This works: +If we wish to modify precedence, parentheses are needed, which takes us out of the curry context, unless we explicitly `curry` the subexpression. This works: ```python curry(f, a, curry(g, x, y), b, c) @@ -1690,7 +1690,7 @@ but this **does not**: curry(f, a, (g, x, y), b, c) ``` -because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by design; as with all things Python, *explicit is better than implicit*. +because `(g, x, y)` is just a tuple of `g`, `x` and `y`. This is by design; as with all things Python, *explicit is better than implicit*. **Note**: to code in curried style, a [contract system](https://en.wikipedia.org/wiki/Design_by_contract) or a type checker can be useful. Also, be careful with variadic functions, because any allowable arity will trigger the call. @@ -1705,7 +1705,7 @@ because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by desi - You can also just use Python's type annotations; `unpythonic`'s `curry` type-checks the arguments before accepting the curried function. The annotations work if the stdlib function [`typing.get_type_hints`](https://docs.python.org/3/library/typing.html#typing.get_type_hints) can find them. -#### ``fix``: break infinite recursion cycles +#### `fix`: break infinite recursion cycles The name `fix` comes from the *least fixed point* with respect to the definedness relation, which is related to Haskell's `fix` function. However, this `fix` is **not** that function. Our `fix` breaks recursion cycles in strict functions - thus causing some non-terminating strict functions to return. (Here [*strict*](https://en.wikipedia.org/wiki/Evaluation_strategy#Strict_evaluation) means that the arguments are evaluated eagerly.) @@ -1838,15 +1838,15 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ ### Batteries for itertools - `unpack`: lazily unpack an iterable. Suitable for infinite inputs. - - Return the first ``n`` items and the ``k``th tail, in a tuple. Default is ``k = n``. - - Use ``k > n`` to fast-forward, consuming the skipped items. Works by `drop`. - - Use ``k < n`` to peek without permanently extracting an item. Works by [tee](https://docs.python.org/3/library/itertools.html#itertools.tee)ing; plan accordingly. + - Return the first `n` items and the `k`th tail, in a tuple. Default is `k = n`. + - Use `k > n` to fast-forward, consuming the skipped items. Works by `drop`. + - Use `k < n` to peek without permanently extracting an item. Works by [tee](https://docs.python.org/3/library/itertools.html#itertools.tee)ing; plan accordingly. - *fold, scan, unfold*: - `foldl`, `foldr` with support for multiple input iterables, like in Racket. - Like in Racket, `op(elt, acc)`; general case `op(e1, e2, ..., en, acc)`. Note Python's own `functools.reduce` uses the ordering `op(acc, elt)` instead. - No sane default for multi-input case, so the initial value for `acc` must be given. - One-input versions with optional init are provided as `reducel`, `reducer`, with semantics similar to Python's `functools.reduce`, but with the rackety ordering `op(elt, acc)`. - - By default, multi-input folds terminate on the shortest input. To instead terminate on the longest input, use the ``longest`` and ``fillvalue`` kwargs. + - By default, multi-input folds terminate on the shortest input. To instead terminate on the longest input, use the `longest` and `fillvalue` kwargs. - For multiple inputs with different lengths, `foldr` syncs the **left** ends. - `rfoldl`, `rreducel` reverse each input and then left-fold. This syncs the **right** ends. - `scanl`, `scanr`: scan (a.k.a. accumulate, partial fold); a lazy fold that returns a generator yielding intermediate results. @@ -1864,7 +1864,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - Unfold returns a generator yielding the collected values. The output can be finite or infinite; to signify that a finite sequence ends, the user function must return `None`. (Beside a `Values` object, a bare `None` is the only other allowed return value from the user function.) - *mapping and zipping*: - `map_longest`: the final missing battery for `map`. - - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by ``itertools``, but it's convenient to have a named shorthand to do that. + - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by `itertools`, but it's convenient to have a named shorthand to do that. - `rmap`, `rzip`, `rmap_longest`, `rzip_longest`: reverse each input, then map/zip. For multiple inputs, syncs the **right** ends. - `mapr`, `zipr`, `mapr_longest`, `zipr_longest`: map/zip, then reverse the result. For multiple inputs, syncs the **left** ends. - `map`: curry-friendly wrapper for the builtin, making it mandatory to specify at least one iterable. **Added in v0.14.2.** @@ -1876,7 +1876,7 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - This differs from `zip` in that the output is flattened, and the termination condition is checked after each item. So e.g. `interleave(['a', 'b', 'c'], ['+', '*'])` → `['a', '+', 'b', '*', 'c']` (the actual return value is a generator, not a list). - *flattening*: - `flatmap`: map a function, that returns a list or tuple, over an iterable and then flatten by one level, concatenating the results into a single tuple. - - Essentially, ``composel(map(...), flatten1)``; the same thing the bind operator of the List monad does. + - Essentially, `composel(map(...), flatten1)`; the same thing the bind operator of the List monad does. - `flatten1`, `flatten`, `flatten_in`: remove nested list structure. - `flatten1`: outermost level only. - `flatten`: recursive, with an optional predicate that controls whether to flatten a given sublist. @@ -1904,10 +1904,10 @@ For more, see [[1]](https://www.parsonsmatt.org/2016/10/26/grokking_fix.html) [[ - *miscellaneous*: - `uniqify`, `uniq`: remove duplicates (either all or consecutive only, respectively), preserving the original ordering of the items. - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. - - `scons`: prepend one element to the start of an iterable, return new iterable. ``scons(x, iterable)`` is lispy shorthand for ``itertools.chain((x,), iterable)``, allowing to omit the one-item tuple wrapper. The name is an abbreviation of [`stream-cons`](https://docs.racket-lang.org/reference/streams.html). - - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. - - Only applicable to monotonic divergent inputs (such as ``primes``). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. - - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. + - `scons`: prepend one element to the start of an iterable, return new iterable. `scons(x, iterable)` is lispy shorthand for `itertools.chain((x,), iterable)`, allowing to omit the one-item tuple wrapper. The name is an abbreviation of [`stream-cons`](https://docs.racket-lang.org/reference/streams.html). + - `inn`: contains-check (`x in iterable`) with automatic termination for monotonic divergent infinite iterables. + - Only applicable to monotonic divergent inputs (such as `primes`). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. + - `iindex`: like `list.index`, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. - `CountingIterator`: use `CountingIterator(iterable)` instead of `iter(iterable)` to produce an iterator that, as a side effect, counts how many items have been yielded. The count is stored in the `.count` attribute. **Added in v0.14.2.** - `slurp`: extract all items from a `queue.Queue` (until it is empty) to a list, returning that list. **Added in v0.14.2.** - `subset`: test whether an iterable is a subset of another. **Added in v0.14.3.** @@ -2124,7 +2124,7 @@ assert decoder.decode() is None ``` -### ``islice``: slice syntax support for ``itertools.islice` +### `islice`: slice syntax support for `itertools.islice` **Changed in v0.14.2.** *Added support for negative `start` and `stop`.* @@ -2148,13 +2148,13 @@ assert tuple(islice(odds)[:5]) == (11, 13, 15, 17, 19) # five more As a convenience feature: a single index is interpreted as a length-1 `islice` starting at that index. The slice is then immediately evaluated and the item is returned. -The slicing variant calls ``itertools.islice`` with the corresponding slicing parameters, after possibly converting negative `start` and `stop` to the appropriate positive values. +The slicing variant calls `itertools.islice` with the corresponding slicing parameters, after possibly converting negative `start` and `stop` to the appropriate positive values. **CAUTION**: When using negative `start` and/or `stop`, the whole iterable is consumed to determine where it ends, if at all. Obviously, this will not terminate for infinite iterables. The desired elements are then held in an internal buffer until they are yielded by iterating over the `islice`. **CAUTION**: Keep in mind that negative `step` is not supported, and that the slicing process consumes elements from the iterable. -Like ``fup``, our ``islice`` is essentially a manually curried function with unusual syntax; the initial call to ``islice`` passes in the iterable to be sliced. The object returned by the call accepts a subscript to specify the slice or index. Once the slice or index is provided, the call to ``itertools.islice`` triggers. +Like `fup`, our `islice` is essentially a manually curried function with unusual syntax; the initial call to `islice` passes in the iterable to be sliced. The object returned by the call accepts a subscript to specify the slice or index. Once the slice or index is provided, the call to `itertools.islice` triggers. Inspired by Python itself. @@ -2169,9 +2169,9 @@ Memoize iterables; like `itertools.tee`, but no need to know in advance how many - `gmemoize` is a decorator for a gfunc, which makes it memoize the instantiated generators. - If the gfunc takes arguments, they must be hashable. A separate memoized sequence is created for each unique set of argument values seen. - - For simplicity, the generator itself may use ``yield`` for output only; ``send`` is **not** supported. - - Any exceptions raised by the generator (except StopIteration) are also memoized, like in ``memoize``. - - Thread-safe. Calls to ``next`` on the memoized generator from different threads are serialized via a lock. Each memoized sequence has its own lock. This uses ``threading.RLock``, so re-entering from the same thread (e.g. in recursively defined mathematical sequences) is fine. + - For simplicity, the generator itself may use `yield` for output only; `send` is **not** supported. + - Any exceptions raised by the generator (except StopIteration) are also memoized, like in `memoize`. + - Thread-safe. Calls to `next` on the memoized generator from different threads are serialized via a lock. Each memoized sequence has its own lock. This uses `threading.RLock`, so re-entering from the same thread (e.g. in recursively defined mathematical sequences) is fine. - The whole history is kept indefinitely. For infinite iterables, use this only if you can guarantee that only a reasonable number of terms will ever be evaluated (w.r.t. available RAM). - Typically, `gmemoize` should be the outermost decorator if several are used on the same gfunc. - `imemoize`: memoize an iterable. Like `itertools.tee`, but keeps the whole history, so more copies can be teed off later. @@ -2219,21 +2219,21 @@ def some_evens(n): # we want to memoize the result without the n first terms assert last(some_evens(25)) == last(some_evens(25)) # iterating twice! ``` -Using a lambda, we can also write ``some_evens`` as: +Using a lambda, we can also write `some_evens` as: ```python se = gmemoize(lambda n: (yield from drop(n, evens()))) assert last(se(25)) == last(se(25)) ``` -Using `fimemoize`, we can omit the ``yield from``, shortening this to: +Using `fimemoize`, we can omit the `yield from`, shortening this to: ```python se = fimemoize(lambda n: drop(n, evens())) assert last(se(25)) == last(se(25)) ``` -If we don't need to take an argument, we can memoize the iterable directly, using ``imemoize``: +If we don't need to take an argument, we can memoize the iterable directly, using `imemoize`: ```python se = imemoize(drop(25, evens())) @@ -2252,24 +2252,24 @@ def some_evens(n): yield from drop(n, evens()) ``` -The only differences are the name of the decorator and ``return`` vs. ``yield from``. The point of `fimemoize` is that in simple cases like this, it allows us to use a regular factory function that makes an iterable, instead of a gfunc. Of course, the gfunc could have several `yield` expressions before it finishes, whereas the factory function terminates at the `return`. +The only differences are the name of the decorator and `return` vs. `yield from`. The point of `fimemoize` is that in simple cases like this, it allows us to use a regular factory function that makes an iterable, instead of a gfunc. Of course, the gfunc could have several `yield` expressions before it finishes, whereas the factory function terminates at the `return`. -### ``fup``: Functional update; ``ShadowedSequence`` +### `fup`: Functional update; `ShadowedSequence` **Changed in 0.15.0.** *Bug fixed: Now an infinite replacement sequence to pull items from is actually ok, as the documentation has always claimed.* We provide three layers, in increasing order of the level of abstraction: `ShadowedSequence`, `fupdate`, and `fup`. -The class ``ShadowedSequence`` is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). It supports slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. We will not discuss ``ShadowedSequence`` in more detail here, as it is a low-level tool; see its docstring for details. +The class `ShadowedSequence` is a bit like `collections.ChainMap`, but for sequences, and only two levels (but it's a sequence; instances can be chained). It supports slicing (read-only), equality comparison, `str` and `repr`. Out-of-range read access to a single item emits a meaningful error, like in `list`. We will not discuss `ShadowedSequence` in more detail here, as it is a low-level tool; see its docstring for details. -The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy, of the same type as the given input sequence, when it finalizes its output. +The function `fupdate` functionally updates sequences and mappings. Whereas `ShadowedSequence` reads directly from the original sequences at access time, `fupdate` makes a shallow copy, of the same type as the given input sequence, when it finalizes its output. -Finally, the function ``fup`` provides a high-level API to functionally update a sequence, with nice syntax. +Finally, the function `fup` provides a high-level API to functionally update a sequence, with nice syntax. #### `fup` -**The preferred way** to use ``fupdate`` on sequences is through the ``fup`` utility function, which specializes ``fupdate`` to sequences, and adds support for Python's standard **slicing syntax**: +**The preferred way** to use `fupdate` on sequences is through the `fup` utility function, which specializes `fupdate` to sequences, and adds support for Python's standard **slicing syntax**: ```python from unpythonic import fup @@ -2281,17 +2281,17 @@ assert fup(tup)[0::2] << tuple(repeat(10, 3)) == (10, 2, 10, 4, 10) assert fup(tup)[0::2] << repeat(10) == (10, 2, 10, 4, 10) # infinite replacement ``` -Currently only one *update specification* is supported in a single ``fup()``. The low-level ``fupdate`` function supports more; see below. +Currently only one *update specification* is supported in a single `fup()`. The low-level `fupdate` function supports more; see below. An *update specification* is a combination of **where** to update, and **what** to put there. The *where* part can be a single index or a slice. When it is a single index, the *what* is a single item; and when a slice, the *what* is a sequence or an iterable, which must contain at least as many items as are required to perform the update. For details, see `fupdate` below. -The ``fup`` function is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. +The `fup` function is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to `fupdate` triggers, and the result is returned. -The notation follows the ``unpythonic`` convention that ``<<`` denotes an assignment of some sort. Here it denotes a functional update, which returns a modified copy, leaving the original untouched. +The notation follows the `unpythonic` convention that `<<` denotes an assignment of some sort. Here it denotes a functional update, which returns a modified copy, leaving the original untouched. #### `fupdate` -The ``fupdate`` function itself, which is the next lower abstraction level, works as follows: +The `fupdate` function itself, which is the next lower abstraction level, works as follows: ```python from unpythonic import fupdate @@ -2318,11 +2318,11 @@ assert fupdate(tup, slice(None, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4 assert fupdate(tup, slice(None, None, -1), range(5)) == (4, 3, 2, 1, 0) ``` -Slicing supports negative indices and steps, and default starts, stops and steps, as usual in Python. Just remember ``a[start:stop:step]`` actually means ``a[slice(start, stop, step)]`` (with ``None`` replacing omitted ``start``, ``stop`` and ``step``), and everything should follow. Multidimensional arrays are **not** supported. +Slicing supports negative indices and steps, and default starts, stops and steps, as usual in Python. Just remember `a[start:stop:step]` actually means `a[slice(start, stop, step)]` (with `None` replacing omitted `start`, `stop` and `step`), and everything should follow. Multidimensional arrays are **not** supported. -When ``fupdate`` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. In the last example above, the ``range(5)`` was read in the order ``4, 3, 2, 1, 0``. This is because when `slice(None, None, -1)` is applied to the input sequence, the first item of the input sequence is index `4` in the slice. So when replacing the first item, ``fupdate`` looked up index `4` in the replacement sequence. Because the replacement was just `range(5)`, the value at index `4` was also `4`. +When `fupdate` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. In the last example above, the `range(5)` was read in the order `4, 3, 2, 1, 0`. This is because when `slice(None, None, -1)` is applied to the input sequence, the first item of the input sequence is index `4` in the slice. So when replacing the first item, `fupdate` looked up index `4` in the replacement sequence. Because the replacement was just `range(5)`, the value at index `4` was also `4`. -The replacement sequence must have at least as many items as the slice requires, when the slice is applied to the original input sequence. Any extra items in the replacement sequence are simply ignored, but if the replacement is too short, ``IndexError`` is raised. +The replacement sequence must have at least as many items as the slice requires, when the slice is applied to the original input sequence. Any extra items in the replacement sequence are simply ignored, but if the replacement is too short, `IndexError` is raised. The replacement must have `__len__` and `__getitem__` methods if the slice (when treated as explained above) requires reading the replacement backwards, and/or if you plan to iterate over the `ShadowedSequence` multiple times. If the replacement only needs to be read forwards, **AND** you only plan to iterate over the `ShadowedSequence` just once (e.g., as part of a `fup`/`fupdate` operation), then it is sufficient for the replacement to implement the `collections.abc.Iterator` API only (i.e. just `__iter__` and `__next__`). @@ -2401,7 +2401,7 @@ assert sorted(d1.items()) == [('foo', 'bar'), ('fruit', 'apple')] assert sorted(d2.items()) == [('foo', 'tavern'), ('fruit', 'apple')] ``` -For immutable mappings, ``fupdate`` supports ``frozendict`` (see below). Any other mapping is assumed mutable, and ``fupdate`` essentially just performs ``copy.copy()`` and then ``.update()``. +For immutable mappings, `fupdate` supports `frozendict` (see below). Any other mapping is assumed mutable, and `fupdate` essentially just performs `copy.copy()` and then `.update()`. ##### `fupdate` and named tuples @@ -2418,10 +2418,10 @@ assert out == A(42, 23) Named tuples export only a sequence interface, so they **cannot** be treated as mappings, even though their elements have names. -Support for ``namedtuple`` uses an extra feature of ``fupdate``, which is available for custom classes, too. When constructing the output sequence, ``fupdate`` first checks whether the type of the input sequence has a ``._make()`` method, and if so, hands the iterable containing the final data to that to construct the output. Otherwise the regular constructor is called (and it must accept a single iterable). +Support for `namedtuple` uses an extra feature of `fupdate`, which is available for custom classes, too. When constructing the output sequence, `fupdate` first checks whether the type of the input sequence has a `._make()` method, and if so, hands the iterable containing the final data to that to construct the output. Otherwise the regular constructor is called (and it must accept a single iterable). -### ``view``: writable, sliceable view into a sequence +### `view`: writable, sliceable view into a sequence A writable view into a sequence, with slicing, so you can take a slice of a slice (of a slice ...), and it reflects the original both ways: @@ -2446,32 +2446,32 @@ v[:] = 42 # scalar broadcast assert lst == [0, 1, 42, 42, 4] ``` -While ``fupdate`` lets you be more functional than Python otherwise allows, ``view`` lets you be more imperative than Python otherwise allows. +While `fupdate` lets you be more functional than Python otherwise allows, `view` lets you be more imperative than Python otherwise allows. We store slice specs, not actual indices, so this works also if the underlying sequence undergoes length changes. -Slicing a view returns a new view. Slicing anything else will usually shallow-copy, because the object being sliced does, before we get control. To slice lazily, first view the sequence itself and then slice that. The initial no-op view is optimized away, so it won't slow down accesses. Alternatively, pass a ``slice`` object into the ``view`` constructor. +Slicing a view returns a new view. Slicing anything else will usually shallow-copy, because the object being sliced does, before we get control. To slice lazily, first view the sequence itself and then slice that. The initial no-op view is optimized away, so it won't slow down accesses. Alternatively, pass a `slice` object into the `view` constructor. The view can be efficiently iterated over. As usual, iteration assumes that no inserts/deletes in the underlying sequence occur during the iteration. Getting/setting an item (subscripting) checks whether the index cache needs updating during each access, so it can be a bit slow. Setting a slice checks just once, and then updates the underlying iterable directly. Setting a slice to a scalar value broadcasts the scalar à la NumPy. -The ``unpythonic.collections`` module also provides the ``SequenceView`` and ``MutableSequenceView`` abstract base classes; ``view`` is a ``MutableSequenceView``. +The `unpythonic.collections` module also provides the `SequenceView` and `MutableSequenceView` abstract base classes; `view` is a `MutableSequenceView`. -There is also the read-only cousin ``roview``, which is like ``view``, except it has no ``__setitem__`` or ``reverse``. This can be useful for providing explicit read-only access to a sequence, when it is undesirable to have clients write into it. +There is also the read-only cousin `roview`, which is like `view`, except it has no `__setitem__` or `reverse`. This can be useful for providing explicit read-only access to a sequence, when it is undesirable to have clients write into it. -The constructor of the writable ``view`` checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. +The constructor of the writable `view` checks that the input is not read-only (`roview`, or a `Sequence` that is not also a `MutableSequence`) before allowing creation of the writable view. -### ``mogrify``: update a mutable container in-place +### `mogrify`: update a mutable container in-place **Changed in v0.14.3.** *`mogrify` now skips `nil`, actually making it useful for processing `ll` linked lists.* -Recurse on a given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like ``map`` does. +Recurse on a given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like `map` does. If the container is a mapping, the function is applied to the values; keys are left untouched. -Unlike ``map`` and its cousins, **``mogrify`` only supports a single input container**. Supporting multiple containers as input would require enforcing some compatibility constraints on their type and shape, because ``mogrify`` is not limited to sequences. +Unlike `map` and its cousins, **`mogrify` only supports a single input container**. Supporting multiple containers as input would require enforcing some compatibility constraints on their type and shape, because `mogrify` is not limited to sequences. ```python from unpythonic import mogrify @@ -2482,42 +2482,42 @@ assert lst2 == [2, 4, 6] assert lst2 is lst1 ``` -Containers are detected by checking for instances of ``collections.abc`` superclasses (also virtuals are ok). Supported abcs are ``MutableMapping``, ``MutableSequence``, ``MutableSet``, ``Mapping``, ``Sequence`` and ``Set``. Any value that does not match any of these is treated as an atom. Containers can be nested, with an arbitrary combination of the types supported. +Containers are detected by checking for instances of `collections.abc` superclasses (also virtuals are ok). Supported abcs are `MutableMapping`, `MutableSequence`, `MutableSet`, `Mapping`, `Sequence` and `Set`. Any value that does not match any of these is treated as an atom. Containers can be nested, with an arbitrary combination of the types supported. For convenience, we support some special cases: - - Any classes created by ``collections.namedtuple``; they do not conform to the standard constructor API for a ``Sequence``. + - Any classes created by `collections.namedtuple`; they do not conform to the standard constructor API for a `Sequence`. - Thus, to support also named tuples: for any immutable ``Sequence``, we first check for the presence of a ``._make()`` method, and if found, use it as the constructor. Otherwise we use the regular constructor. + Thus, to support also named tuples: for any immutable `Sequence`, we first check for the presence of a `._make()` method, and if found, use it as the constructor. Otherwise we use the regular constructor. - - ``str`` is treated as an atom, although technically a ``Sequence``. + - `str` is treated as an atom, although technically a `Sequence`. It does not conform to the exact same API (its constructor does not take an iterable), and often one does not want to treat strings as containers anyway. - If you want to process strings, implement it in your function that is called by ``mogrify``. You can e.g. `tuple(thestring)` and then call ``mogrify`` on that. + If you want to process strings, implement it in your function that is called by `mogrify`. You can e.g. `tuple(thestring)` and then call `mogrify` on that. - - The ``box``, `ThreadLocalBox` and `Some` containers from ``unpythonic.collections``. Although the first two are mutable, their update is not conveniently expressible by the ``collections.abc`` APIs. + - The `box`, `ThreadLocalBox` and `Some` containers from `unpythonic.collections`. Although the first two are mutable, their update is not conveniently expressible by the `collections.abc` APIs. - - The ``cons`` container from ``unpythonic.llist`` (including the ``ll``, ``llist`` linked lists). This is treated with the general tree strategy, so nested linked lists will be flattened, and the final ``nil`` is also processed. + - The `cons` container from `unpythonic.llist` (including the `ll`, `llist` linked lists). This is treated with the general tree strategy, so nested linked lists will be flattened, and the final `nil` is also processed. - Note that since ``cons`` is immutable, anyway, if you know you have a long linked list where you need to update the values, just iterate over it and produce a new copy - that will work as intended. + Note that since `cons` is immutable, anyway, if you know you have a long linked list where you need to update the values, just iterate over it and produce a new copy - that will work as intended. -### ``s``, ``imathify``, ``gmathify``: lazy mathematical sequences with infix arithmetic +### `s`, `imathify`, `gmathify`: lazy mathematical sequences with infix arithmetic **Changed in v0.14.3.** Added convenience mode to generate cyclic infinite sequences. **Changed in v0.14.3.** To improve descriptiveness, and for consistency with names of other abstractions in `unpythonic`, `m` has been renamed `imathify` and `mg` has been renamed `gmathify`. The old names will continue working in v0.14.x, and will be removed in v0.15.0. This is a one-time change; it is not likely that these names will be changed ever again. -We provide a compact syntax to create lazy constant, cyclic, arithmetic, geometric and power sequences: ``s(...)``. Numeric (``int``, ``float``, ``mpmath``) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. +We provide a compact syntax to create lazy constant, cyclic, arithmetic, geometric and power sequences: `s(...)`. Numeric (`int`, `float`, `mpmath`) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. -We also provide arithmetic operation support for iterables (termwise). To make any iterable infix math aware, use ``imathify(iterable)``. The arithmetic is lazy; it just plans computations, returning a new lazy mathematical sequence. To extract values, iterate over the result. (Note this implies that expressions consisting of thousands of operations will overflow Python's call stack. In practice this shouldn't be a problem.) +We also provide arithmetic operation support for iterables (termwise). To make any iterable infix math aware, use `imathify(iterable)`. The arithmetic is lazy; it just plans computations, returning a new lazy mathematical sequence. To extract values, iterate over the result. (Note this implies that expressions consisting of thousands of operations will overflow Python's call stack. In practice this shouldn't be a problem.) -The function versions of the arithmetic operations (also provided, à la the ``operator`` module) have an **s** prefix (short for mathematical **sequence**), because in Python the **i** prefix (which could stand for *iterable*) is already used to denote the in-place operators. +The function versions of the arithmetic operations (also provided, à la the `operator` module) have an **s** prefix (short for mathematical **sequence**), because in Python the **i** prefix (which could stand for *iterable*) is already used to denote the in-place operators. -We provide the [Cauchy product](https://en.wikipedia.org/wiki/Cauchy_product), and its generalization, the diagonal combination-reduction, for two (possibly infinite) iterables. Note ``cauchyprod`` **does not sum the series**; given the input sequences ``a`` and ``b``, the call ``cauchyprod(a, b)`` computes the elements of the output sequence ``c``. +We provide the [Cauchy product](https://en.wikipedia.org/wiki/Cauchy_product), and its generalization, the diagonal combination-reduction, for two (possibly infinite) iterables. Note `cauchyprod` **does not sum the series**; given the input sequences `a` and `b`, the call `cauchyprod(a, b)` computes the elements of the output sequence `c`. -We also provide ``gmathify``, a decorator to mathify a gfunc, so that it will ``imathify()`` the generator instances it makes. Combo with ``imemoize`` for great justice, e.g. ``a = gmathify(imemoize(myiterable))``, and then ``a()`` to instantiate a memoized-and-mathified copy. +We also provide `gmathify`, a decorator to mathify a gfunc, so that it will `imathify()` the generator instances it makes. Combo with `imemoize` for great justice, e.g. `a = gmathify(imemoize(myiterable))`, and then `a()` to instantiate a memoized-and-mathified copy. Finally, we provide ready-made generators that yield some common sequences (currently, the Fibonacci numbers, the triangular numbers, and the prime numbers). The prime generator is an FP-ized sieve of Eratosthenes. @@ -2553,7 +2553,7 @@ assert tuple(take(10, fibonacci())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) assert tuple(take(10, triangular())) == (1, 3, 6, 10, 15, 21, 28, 36, 45, 55) ``` -A math iterable (i.e. one that has infix math support) is an instance of the class ``imathify``: +A math iterable (i.e. one that has infix math support) is an instance of the class `imathify`: ```python a = s(1, 3, ...) @@ -2604,12 +2604,12 @@ s2 = px(s(2, 4, 6, ...)) # 2, 4*x, 6*x**2, ... assert tuple(take(3, cauchyprod(s1, s2))) == (2, 10*x, 28*x**2) ``` -**CAUTION**: Symbolic sequence detection is sensitive to the assumptions on the symbols, because very pythonically, ``SymPy`` only simplifies when the result is guaranteed to hold in the most general case under the given assumptions. +**CAUTION**: Symbolic sequence detection is sensitive to the assumptions on the symbols, because very pythonically, `SymPy` only simplifies when the result is guaranteed to hold in the most general case under the given assumptions. Inspired by Haskell. -### ``sym``, ``gensym``, ``Singleton``: symbols and singletons +### `sym`, `gensym`, `Singleton`: symbols and singletons **Added in v0.14.2**. @@ -2707,7 +2707,7 @@ Our `Singleton` abstraction is the result of these pythonifications applied to t #### When to use a singleton? -Most often, **don't**. ``Singleton`` is provided for the very rare occasion where it's the appropriate abstraction. There exist **at least** three categories of use cases where singleton-like instantiation semantics are desirable: +Most often, **don't**. `Singleton` is provided for the very rare occasion where it's the appropriate abstraction. There exist **at least** three categories of use cases where singleton-like instantiation semantics are desirable: 1. **A process-wide unique marker value**, which has no functionality other than being quickly and uniquely identifiable as that marker. - `sym` and `gensym` are the specific tools that cover this use case, depending on whether the intent is to allow that value to be independently "constructed" in several places yet always obtaining the same instance (`sym`), or if the implementation just happens to internally need a guaranteed-unique value that no value passed in from the outside could possibly clash with (`gensym`). For the latter case, sometimes a simple (and much faster) `nonce = object()` will do just as well, if you don't need the human-readable label and `pickle` support. @@ -2728,7 +2728,7 @@ I'm not completely sure if it's meaningful to provide a generic `Singleton` abst Tools related to control flow. -### ``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations +### `trampolined`, `jump`: tail call optimization (TCO) / explicit continuations Express algorithms elegantly without blowing the call stack - with explicit, clear syntax. @@ -2750,15 +2750,15 @@ Functions that use TCO **must** be `@trampolined`. Calling a trampolined functio Inside a trampolined function, a normal call `f(a, ..., kw=v, ...)` remains a normal call. -A tail call with target `f` is denoted `return jump(f, a, ..., kw=v, ...)`. This explicitly marks that it is indeed a tail call (due to the explicit ``return``). Note that `jump` is **a noun, not a verb**. The `jump(f, ...)` part just evaluates to a `jump` instance, which on its own does nothing. Returning it to the trampoline actually performs the tail call. +A tail call with target `f` is denoted `return jump(f, a, ..., kw=v, ...)`. This explicitly marks that it is indeed a tail call (due to the explicit `return`). Note that `jump` is **a noun, not a verb**. The `jump(f, ...)` part just evaluates to a `jump` instance, which on its own does nothing. Returning it to the trampoline actually performs the tail call. If the jump target has a trampoline, don't worry; the trampoline implementation will automatically strip it and jump into the actual entrypoint. -Trying to ``jump(...)`` without the ``return`` does nothing useful, and will **usually** print an *unclaimed jump* warning. It does this by checking a flag in the ``__del__`` method of ``jump``; any correctly used jump instance should have been claimed by a trampoline before it gets garbage-collected. +Trying to `jump(...)` without the `return` does nothing useful, and will **usually** print an *unclaimed jump* warning. It does this by checking a flag in the `__del__` method of `jump`; any correctly used jump instance should have been claimed by a trampoline before it gets garbage-collected. -(Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (``KeyboardInterrupt``). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by the trampoline.) +(Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (`KeyboardInterrupt`). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by the trampoline.) -The final result is just returned normally. This shuts down the trampoline, and returns the given value from the initial call (to a ``@trampolined`` function) that originally started that trampoline. +The final result is just returned normally. This shuts down the trampoline, and returns the given value from the initial call (to a `@trampolined` function) that originally started that trampoline. *Tail recursion in a lambda*: @@ -2771,7 +2771,7 @@ print(t(4)) # 24 Here the jump is just `jump` instead of `return jump`, since lambda does not use the `return` syntax. -To denote tail recursion in an anonymous function, use ``unpythonic.fun.withself``. The ``self`` argument is declared explicitly, but passed implicitly, just like the ``self`` argument of a method. +To denote tail recursion in an anonymous function, use `unpythonic.fun.withself`. The `self` argument is declared explicitly, but passed implicitly, just like the `self` argument of a method. *Mutual recursion with TCO*: @@ -2852,7 +2852,7 @@ The `return jump(...)` solution is essentially the same there (the syntax is `#( Clojure's trampoline system is thus more explicit and simple than ours (the trampoline doesn't need to detect and strip the tail-call target's trampoline, if it has one - because with Clojure's solution, it never does), at some cost to convenience at each use site. We have chosen to emphasize use-site convenience. -### ``looped``, ``looped_over``: loops in FP style (with TCO) +### `looped`, `looped_over`: loops in FP style (with TCO) *Functional loop with automatic tail call optimization* (for calls re-invoking the loop body): @@ -2896,13 +2896,13 @@ print(s) # 45 In `@looped`, the function name of the loop body is the name of the final result, like in `@call`. The final result of the loop is just returned normally. -The first parameter of the loop body is the magic parameter ``loop``. It is *self-ish*, representing a jump back to the loop body itself, starting a new iteration. Just like Python's ``self``, ``loop`` can have any name; it is passed positionally. +The first parameter of the loop body is the magic parameter `loop`. It is *self-ish*, representing a jump back to the loop body itself, starting a new iteration. Just like Python's `self`, `loop` can have any name; it is passed positionally. -Note that ``loop`` is **a noun, not a verb.** This is because the expression ``loop(...)`` is essentially the same as ``jump(...)`` to the loop body itself. However, it also inserts the magic parameter ``loop``, which can only be set up via this mechanism. +Note that `loop` is **a noun, not a verb.** This is because the expression `loop(...)` is essentially the same as `jump(...)` to the loop body itself. However, it also inserts the magic parameter `loop`, which can only be set up via this mechanism. -Additional arguments can be given to ``loop(...)``. When the loop body is called, any additional positional arguments are appended to the implicit ones, and can be anything. Additional arguments can also be passed by name. The initial values of any additional arguments **must** be declared as defaults in the formal parameter list of the loop body. The loop is automatically started by `@looped`, by calling the body with the magic ``loop`` as the only argument. +Additional arguments can be given to `loop(...)`. When the loop body is called, any additional positional arguments are appended to the implicit ones, and can be anything. Additional arguments can also be passed by name. The initial values of any additional arguments **must** be declared as defaults in the formal parameter list of the loop body. The loop is automatically started by `@looped`, by calling the body with the magic `loop` as the only argument. -Any loop variables such as ``i`` in the above example are **in scope only in the loop body**; there is no ``i`` in the surrounding scope. Moreover, it's a fresh ``i`` at each iteration; nothing is mutated by the looping mechanism. (But be careful if you use a mutable object instance as a loop variable. The loop body is just a function call like any other, so the usual rules apply.) +Any loop variables such as `i` in the above example are **in scope only in the loop body**; there is no `i` in the surrounding scope. Moreover, it's a fresh `i` at each iteration; nothing is mutated by the looping mechanism. (But be careful if you use a mutable object instance as a loop variable. The loop body is just a function call like any other, so the usual rules apply.) FP loops don't have to be pure: @@ -2919,7 +2919,7 @@ assert out == [0, 1, 2, 3] Keep in mind, though, that this pure-Python FP looping mechanism is slow, so it may make sense to use it only when "the FP-ness" (no mutation, scoping) is important. -Also be aware that `@looped` is specifically neither a ``for`` loop nor a ``while`` loop; instead, it is a general looping mechanism that can express both kinds of loops. +Also be aware that `@looped` is specifically neither a `for` loop nor a `while` loop; instead, it is a general looping mechanism that can express both kinds of loops. *Typical `while True` loop in FP style*: @@ -2937,7 +2937,7 @@ def _(loop): #### FP loop over an iterable -In Python, loops often run directly over the elements of an iterable, which markedly improves readability compared to dealing with indices. Enter ``@looped_over``: +In Python, loops often run directly over the elements of an iterable, which markedly improves readability compared to dealing with indices. Enter `@looped_over`: ```python @looped_over(range(10), acc=0) @@ -2946,7 +2946,7 @@ def s(loop, x, acc): assert s == 45 ``` -The ``@looped_over`` decorator is essentially sugar. Behaviorally equivalent code: +The `@looped_over` decorator is essentially sugar. Behaviorally equivalent code: ```python @call @@ -2963,11 +2963,11 @@ def s(iterable=range(10)): assert s == 45 ``` -In ``@looped_over``, the loop body takes three magic positional parameters. The first parameter ``loop`` works like in ``@looped``. The second parameter ``x`` is the current element. The third parameter ``acc`` is initialized to the ``acc`` value given to ``@looped_over``, and then (functionally) updated at each iteration, taking as the new value the first positional argument given to ``loop(...)``, if any positional arguments were given. Otherwise ``acc`` retains its last value. +In `@looped_over`, the loop body takes three magic positional parameters. The first parameter `loop` works like in `@looped`. The second parameter `x` is the current element. The third parameter `acc` is initialized to the `acc` value given to `@looped_over`, and then (functionally) updated at each iteration, taking as the new value the first positional argument given to `loop(...)`, if any positional arguments were given. Otherwise `acc` retains its last value. -If ``acc`` is a mutable object, mutating it is allowed. For example, if ``acc`` is a list, it is perfectly fine to ``acc.append(...)`` and then just ``loop()`` with no arguments, allowing ``acc`` to retain its last value. To be exact, keeping the last value means *the binding of the name ``acc`` does not change*, so when the next iteration starts, the name ``acc`` still points to the same object that was mutated. This strategy can be used to pythonically construct a list in an FP loop. +If `acc` is a mutable object, mutating it is allowed. For example, if `acc` is a list, it is perfectly fine to `acc.append(...)` and then just `loop()` with no arguments, allowing `acc` to retain its last value. To be exact, keeping the last value means *the binding of the name `acc` does not change*, so when the next iteration starts, the name `acc` still points to the same object that was mutated. This strategy can be used to pythonically construct a list in an FP loop. -Additional arguments can be given to ``loop(...)``. The same notes as above apply. For example, here we have the additional parameters ``fruit`` and ``number``. The first one is passed positionally, and the second one by name: +Additional arguments can be given to `loop(...)`. The same notes as above apply. For example, here we have the additional parameters `fruit` and `number`. The first one is passed positionally, and the second one by name: ```python @looped_over(range(10), acc=0) @@ -2979,11 +2979,11 @@ def s(loop, x, acc, fruit="pear", number=23): assert s == 45 ``` -The loop body is called once for each element in the iterable. When the iterable runs out of elements, the last ``acc`` value that was given to ``loop(...)`` becomes the return value of the loop. If the iterable is empty, the body never runs; then the return value of the loop is the initial value of ``acc``. +The loop body is called once for each element in the iterable. When the iterable runs out of elements, the last `acc` value that was given to `loop(...)` becomes the return value of the loop. If the iterable is empty, the body never runs; then the return value of the loop is the initial value of `acc`. -To terminate the loop early, just ``return`` your final result normally, like in ``@looped``. (It can be anything, does not need to be ``acc``.) +To terminate the loop early, just `return` your final result normally, like in `@looped`. (It can be anything, does not need to be `acc`.) -Multiple input iterables work somewhat like in Python's ``for``, except any sequence unpacking must be performed inside the body: +Multiple input iterables work somewhat like in Python's `for`, except any sequence unpacking must be performed inside the body: ```python @looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=()) @@ -3013,16 +3013,16 @@ def outer_result(outer_loop, y, outer_acc): assert outer_result == ((1, 2), (2, 4), (3, 6)) ``` -If you feel the trailing commas ruin the aesthetics, see ``unpythonic.misc.pack``. +If you feel the trailing commas ruin the aesthetics, see `unpythonic.misc.pack`. #### Accumulator type and runtime cost As [the reference warns (note 6)](https://docs.python.org/3/library/stdtypes.html#common-sequence-operations), repeated concatenation of tuples has an O(n²) runtime cost, because each concatenation creates a new tuple, which needs to copy all of the already existing elements. To keep the runtime O(n), there are two options: - - *Pythonic solution*: Destructively modify a mutable sequence. Particularly, ``list`` is a dynamic array that has a low amortized cost for concatenation (most often O(1), with the occasional O(n) when the allocated storage grows). - - *Unpythonic solution*: ``cons`` a linked list, and reverse it at the end. Cons cells are immutable; consing a new element to the front costs O(1). Reversing the list costs O(n). + - *Pythonic solution*: Destructively modify a mutable sequence. Particularly, `list` is a dynamic array that has a low amortized cost for concatenation (most often O(1), with the occasional O(n) when the allocated storage grows). + - *Unpythonic solution*: `cons` a linked list, and reverse it at the end. Cons cells are immutable; consing a new element to the front costs O(1). Reversing the list costs O(n). -Mutable sequence (Python ``list``): +Mutable sequence (Python `list`): ```python @looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=[]) @@ -3048,9 +3048,9 @@ def p(loop, item, acc): assert p == ll('1a', '2b', '3c') ``` -Note the unpythonic use of the ``lreverse`` function as a decorator. ``@looped_over`` overwrites the def'd name by the return value of the loop; then ``lreverse`` takes that as input, and overwrites once more. Thus ``p`` becomes the final list. +Note the unpythonic use of the `lreverse` function as a decorator. `@looped_over` overwrites the def'd name by the return value of the loop; then `lreverse` takes that as input, and overwrites once more. Thus `p` becomes the final list. -To get the output as a tuple, we can add ``tuple`` to the decorator chain: +To get the output as a tuple, we can add `tuple` to the decorator chain: ```python @tuple @@ -3065,15 +3065,15 @@ assert p == ('1a', '2b', '3c') This works in both solutions. The cost is an additional O(n) step. -#### ``break`` +#### `break` -The main way to exit an FP loop (also early) is, at any time, to just ``return`` the final result normally. +The main way to exit an FP loop (also early) is, at any time, to just `return` the final result normally. If you want to exit the function *containing* the loop from inside the loop, see **escape continuations** below. -#### ``continue`` +#### `continue` -The main way to *continue* an FP loop is, at any time, to ``loop(...)`` with the appropriate arguments that will make it proceed to the next iteration. Or package the appropriate `loop(...)` expression into your own function ``cont``, and then use ``cont(...)``: +The main way to *continue* an FP loop is, at any time, to `loop(...)` with the appropriate arguments that will make it proceed to the next iteration. Or package the appropriate `loop(...)` expression into your own function `cont`, and then use `cont(...)`: ```python @looped @@ -3090,15 +3090,15 @@ print(s) # 35 This approach separates the computations of the new values for the iteration counter and the accumulator. -#### Prepackaged ``break`` and ``continue`` +#### Prepackaged `break` and `continue` -See ``@breakably_looped`` (offering `brk`) and ``@breakably_looped_over`` (offering `brk` and `cnt`). +See `@breakably_looped` (offering `brk`) and `@breakably_looped_over` (offering `brk` and `cnt`). The point of `brk(value)` over just `return value` is that `brk` is first-class, so it can be passed on to functions called by the loop body (so that those functions then have the power to directly terminate the loop). -In ``@looped``, a library-provided ``cnt`` wouldn't make sense, since all parameters except ``loop`` are user-defined. *The client code itself defines what it means to proceed to the "next" iteration*. Really the only way in a construct with this degree of flexibility is for the client code to fill in all the arguments itself. +In `@looped`, a library-provided `cnt` wouldn't make sense, since all parameters except `loop` are user-defined. *The client code itself defines what it means to proceed to the "next" iteration*. Really the only way in a construct with this degree of flexibility is for the client code to fill in all the arguments itself. -Because ``@looped_over`` is a more specific abstraction, there the concept of *continue* is much more clear-cut. We define `cnt` to mean *proceed to take the next element from the iterable, keeping the current value of `acc`*. Essentially `cnt` is a partially applied `loop(...)` with the first positional argument set to the current value of `acc`. +Because `@looped_over` is a more specific abstraction, there the concept of *continue* is much more clear-cut. We define `cnt` to mean *proceed to take the next element from the iterable, keeping the current value of `acc`*. Essentially `cnt` is a partially applied `loop(...)` with the first positional argument set to the current value of `acc`. #### FP loops using a lambda as body @@ -3110,9 +3110,9 @@ s = looped(lambda loop, acc=0, i=0: print(s) ``` -It's not just a decorator; in Lisps, a construct like this would likely be named ``call/looped``. +It's not just a decorator; in Lisps, a construct like this would likely be named `call/looped`. -We can also use ``let`` to make local definitions: +We can also use `let` to make local definitions: ```python s = looped(lambda loop, acc=0, i=0: @@ -3132,9 +3132,9 @@ s = r10(lambda loop, x, acc: assert s == 45 ``` -If you **really** need to make that into an expression, bind ``r10`` using ``let`` (if you use ``letrec``, keeping in mind it is a callable), or to make your code unreadable, just inline it. +If you **really** need to make that into an expression, bind `r10` using `let` (if you use `letrec`, keeping in mind it is a callable), or to make your code unreadable, just inline it. -With ``curry``, this is also a possible solution: +With `curry`, this is also a possible solution: ```python s = curry(looped_over, range(10), 0, @@ -3143,11 +3143,11 @@ s = curry(looped_over, range(10), 0, assert s == 45 ``` -### ``gtrampolined``: generators with TCO +### `gtrampolined`: generators with TCO -In ``unpythonic``, a generator can tail-chain into another generator. This is like invoking ``itertools.chain``, but as a tail call from inside the generator - so the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. +In `unpythonic`, a generator can tail-chain into another generator. This is like invoking `itertools.chain`, but as a tail call from inside the generator - so the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. -Python provides a convenient hook to build things like this, in the guise of ``return``: +Python provides a convenient hook to build things like this, in the guise of `return`: ```python from unpythonic import gtco, take, last @@ -3160,7 +3160,7 @@ assert tuple(take(6, gtco(march()))) == (1, 2, 1, 2, 1, 2) last(take(10000, gtco(march()))) # no crash ``` -Note the calls to ``gtco`` at the use sites. For convenience, we provide ``@gtrampolined``, which automates that: +Note the calls to `gtco` at the use sites. For convenience, we provide `@gtrampolined`, which automates that: ```python from unpythonic import gtrampolined, take, last @@ -3173,7 +3173,7 @@ assert tuple(take(10, ones())) == (1,) * 10 last(take(10000, ones())) # no crash ``` -It is safe to tail-chain into a ``@gtrampolined`` generator; the system strips the TCO target's trampoline if it has one. +It is safe to tail-chain into a `@gtrampolined` generator; the system strips the TCO target's trampoline if it has one. Like all tail calls, this works for any *iterative* process. In contrast, this **does not work**: @@ -3188,7 +3188,7 @@ def fibos(): # see numerics.py print(tuple(take(10, fibos()))) # --> (1, 1, 2), only 3 terms?! ``` -This sequence (technically iterable, but in the mathematical sense) is recursively defined, and the ``return`` shuts down the generator before it can yield more terms into ``scanl``. With ``yield from`` instead of ``return`` the second example works (but since it is recursive, it eventually blows the call stack). +This sequence (technically iterable, but in the mathematical sense) is recursively defined, and the `return` shuts down the generator before it can yield more terms into `scanl`. With `yield from` instead of `return` the second example works (but since it is recursive, it eventually blows the call stack). This particular example can be converted into a linear process with a different higher-order function, no TCO needed: @@ -3203,7 +3203,7 @@ last(take(10000, fibos())) # no crash ``` -### ``catch``, ``throw``: escape continuations (ec) +### `catch`, `throw`: escape continuations (ec) **Changed in v0.14.2.** *These constructs were previously named `setescape`, `escape`. The names have been changed to match the standard naming for this feature in several Lisps. Starting in 0.14.2, using the old names emits a `FutureWarning`, and the old names will be removed in 0.15.0.* @@ -3222,11 +3222,11 @@ def f(): assert f() == "hello from g" ``` -**CAUTION**: The implementation is based on exceptions, so catch-all ``except:`` statements will intercept also throws, breaking the escape mechanism. As you already know, be specific in which exception types you catch in an `except` clause! +**CAUTION**: The implementation is based on exceptions, so catch-all `except:` statements will intercept also throws, breaking the escape mechanism. As you already know, be specific in which exception types you catch in an `except` clause! -In Lisp terms, `@catch` essentially captures the escape continuation (ec) of the function decorated with it. The nearest (dynamically) surrounding ec can then be invoked by `throw(value)`. When the `throw` is performed, the function decorated with `@catch` immediately terminates, returning ``value``. +In Lisp terms, `@catch` essentially captures the escape continuation (ec) of the function decorated with it. The nearest (dynamically) surrounding ec can then be invoked by `throw(value)`. When the `throw` is performed, the function decorated with `@catch` immediately terminates, returning `value`. -In Python terms, a throw means just raising a specific type of exception; the usual rules concerning ``try/except/else/finally`` and ``with`` blocks apply. It is a function call, so it works also in lambdas. +In Python terms, a throw means just raising a specific type of exception; the usual rules concerning `try/except/else/finally` and `with` blocks apply. It is a function call, so it works also in lambdas. Escaping the function surrounding an FP loop, from inside the loop: @@ -3242,7 +3242,7 @@ def f(): f() # --> 15 ``` -For more control, both ``@catch`` points and ``throw`` instances can be tagged: +For more control, both `@catch` points and `throw` instances can be tagged: ```python @catch(tags="foo") # catch point tags can be single value or tuple (tuples OR'd, like isinstance()) @@ -3262,24 +3262,24 @@ def foo(): assert foo() == 15 ``` -For details on tagging, especially how untagged and tagged throw and catch points interact, and how to make one-to-one connections, see the docstring for ``@catch``. +For details on tagging, especially how untagged and tagged throw and catch points interact, and how to make one-to-one connections, see the docstring for `@catch`. **Etymology** This feature is known as `catch`/`throw` in several Lisps, e.g. in Emacs Lisp and in Common Lisp (as well as some of its ancestors). This terminology is independent of the use of `throw`/`catch` in C++/Java for the exception handling mechanism. Common Lisp also provides a lexically scoped variant (`BLOCK`/`RETURN-FROM`) that is more idiomatic [according to Seibel](http://www.gigamonkeys.com/book/the-special-operators.html). -#### ``call_ec``: first-class escape continuations +#### `call_ec`: first-class escape continuations -We provide ``call/ec`` (a.k.a. ``call-with-escape-continuation``), in Python spelled as ``call_ec``. It's a decorator that, like ``@call``, immediately runs the function and replaces the def'd name with the return value. The twist is that it internally sets up a catch point, and hands a **first-class escape continuation** to the callee. +We provide `call/ec` (a.k.a. `call-with-escape-continuation`), in Python spelled as `call_ec`. It's a decorator that, like `@call`, immediately runs the function and replaces the def'd name with the return value. The twist is that it internally sets up a catch point, and hands a **first-class escape continuation** to the callee. The function to be decorated **must** take one positional argument, the ec instance. -The ec instance itself is another function, which takes one positional argument: the value to send to the catch point. The ec instance and the catch point are connected one-to-one. No other ``@catch`` point will catch the ec instance, and the catch point catches only this particular ec instance and nothing else. +The ec instance itself is another function, which takes one positional argument: the value to send to the catch point. The ec instance and the catch point are connected one-to-one. No other `@catch` point will catch the ec instance, and the catch point catches only this particular ec instance and nothing else. -Any particular ec instance is only valid inside the dynamic extent of the ``call_ec`` invocation that created it. Attempting to call the ec later raises ``RuntimeError``. +Any particular ec instance is only valid inside the dynamic extent of the `call_ec` invocation that created it. Attempting to call the ec later raises `RuntimeError`. -This builds on ``@catch`` and ``throw``, so the caution about catch-all ``except:`` statements applies here, too. +This builds on `@catch` and `throw`, so the caution about catch-all `except:` statements applies here, too. ```python from unpythonic import call_ec @@ -3306,7 +3306,7 @@ def result(ec): assert result == 42 ``` -The ec doesn't have to be called from the lexical scope of the call_ec'd function, as long as the call occurs within the dynamic extent of the ``call_ec``. It's essentially a *return from me* for the original function: +The ec doesn't have to be called from the lexical scope of the call_ec'd function, as long as the call occurs within the dynamic extent of the `call_ec`. It's essentially a *return from me* for the original function: ```python def f(ec): @@ -3320,7 +3320,7 @@ def result(ec): assert result == 42 ``` -This also works with lambdas, by using ``call_ec()`` directly. No need for a trampoline: +This also works with lambdas, by using `call_ec()` directly. No need for a trampoline: ```python result = call_ec(lambda ec: @@ -3330,11 +3330,11 @@ result = call_ec(lambda ec: assert result == 42 ``` -Normally ``begin()`` would return the last value, but the ec overrides that; it is effectively a ``return`` for multi-expression lambdas! +Normally `begin()` would return the last value, but the ec overrides that; it is effectively a `return` for multi-expression lambdas! But wait, doesn't Python evaluate all the arguments of `begin(...)` before the `begin` itself has a chance to run? Why doesn't the example print also *never reached*? This is because escapes are implemented using exceptions. Evaluating the ec call raises an exception, preventing any further elements from being evaluated. -This usage is valid with named functions, too - ``call_ec`` is not only a decorator: +This usage is valid with named functions, too - `call_ec` is not only a decorator: ```python def f(ec): @@ -3349,30 +3349,30 @@ assert result == 42 ``` -### ``forall``: nondeterministic evaluation +### `forall`: nondeterministic evaluation We provide a simple variant of nondeterministic evaluation. This is essentially a toy that has no more power than list comprehensions or nested for loops. See also the easy-to-use [macro](macros.md) version with natural syntax and a clean implementation. -An important feature of McCarthy's [`amb` operator](https://rosettacode.org/wiki/Amb) is its nonlocality - being able to jump back to a choice point, even after the dynamic extent of the function where that choice point resides. If that sounds a lot like ``call/cc``, that's because that's how ``amb`` is usually implemented. See examples [in Ruby](http://www.randomhacks.net/2005/10/11/amb-operator/) and [in Racket](http://www.cs.toronto.edu/~david/courses/csc324_w15/extra/choice.html). +An important feature of McCarthy's [`amb` operator](https://rosettacode.org/wiki/Amb) is its nonlocality - being able to jump back to a choice point, even after the dynamic extent of the function where that choice point resides. If that sounds a lot like `call/cc`, that's because that's how `amb` is usually implemented. See examples [in Ruby](http://www.randomhacks.net/2005/10/11/amb-operator/) and [in Racket](http://www.cs.toronto.edu/~david/courses/csc324_w15/extra/choice.html). -Python can't do that, short of transforming the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), while applying TCO everywhere to prevent stack overflow. **If that's what you want**, see ``continuations`` in [the macros](macros.md). +Python can't do that, short of transforming the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), while applying TCO everywhere to prevent stack overflow. **If that's what you want**, see `continuations` in [the macros](macros.md). -This ``forall`` is essentially a tuple comprehension that: +This `forall` is essentially a tuple comprehension that: - Can have multiple body expressions (side effects also welcome!), by simply listing them in sequence. - Allows filters to be placed at any level of the nested looping. - Presents the source code in the same order as it actually runs. -The ``unpythonic.amb`` module defines four operators: +The `unpythonic.amb` module defines four operators: - - ``forall`` is the control structure, which marks a section with nondeterministic evaluation. - - ``choice`` binds a name: ``choice(x=range(3))`` essentially means ``for e.x in range(3):``. - - ``insist`` is a filter, which allows the remaining lines to run if the condition evaluates to truthy. - - ``deny`` is ``insist not``; it allows the remaining lines to run if the condition evaluates to falsey. + - `forall` is the control structure, which marks a section with nondeterministic evaluation. + - `choice` binds a name: `choice(x=range(3))` essentially means `for e.x in range(3):`. + - `insist` is a filter, which allows the remaining lines to run if the condition evaluates to truthy. + - `deny` is `insist not`; it allows the remaining lines to run if the condition evaluates to falsey. -Choice variables live in the environment, which is accessed via a ``lambda e: ...``, just like in ``letrec``. Lexical scoping is emulated. In the environment, each line only sees variables defined above it; trying to access a variable defined later raises ``AttributeError``. +Choice variables live in the environment, which is accessed via a `lambda e: ...`, just like in `letrec`. Lexical scoping is emulated. In the environment, each line only sees variables defined above it; trying to access a variable defined later raises `AttributeError`. -The last line in a ``forall`` describes one item of the output. The output items are collected into a tuple, which becomes the return value of the ``forall`` expression. +The last line in a `forall` describes one item of the output. The output items are collected into a tuple, which becomes the return value of the `forall` expression. ```python out = forall(choice(y=range(3)), @@ -3410,22 +3410,22 @@ out = forall(range(2), # do the rest twice! assert out == (1, 2, 3, 1, 2, 3) ``` -The initial ``range(2)`` causes the remaining lines to run twice - because it yields two output values - regardless of whether we bind the result to a variable or not. In effect, each line, if it returns more than one output, introduces a new nested loop at that point. +The initial `range(2)` causes the remaining lines to run twice - because it yields two output values - regardless of whether we bind the result to a variable or not. In effect, each line, if it returns more than one output, introduces a new nested loop at that point. -For more, see the docstring of ``forall``. +For more, see the docstring of `forall`. #### For haskellers The implementation is based on the List monad, and a bastardized variant of do-notation. Quick vocabulary: - - ``forall(...)`` = ``do ...`` (for a List monad) - - ``choice(x=foo)`` = ``x <- foo``, where ``foo`` is an iterable - - ``insist x`` = ``guard x`` - - ``deny x`` = ``guard (not x)`` - - Last line = implicit ``return ...`` + - `forall(...)` = `do ...` (for a List monad) + - `choice(x=foo)` = `x <- foo`, where `foo` is an iterable + - `insist x` = `guard x` + - `deny x` = `guard (not x)` + - Last line = implicit `return ...` -### ``handlers``, ``restarts``: conditions and restarts +### `handlers`, `restarts`: conditions and restarts **Added in v0.14.2**. @@ -3522,7 +3522,7 @@ To create a simple handler that does not take an argument, and just invokes a pr Following Common Lisp terminology, *a named function that invokes a specific restart* - whether it is intended to act as a handler or to be called from one - is termed a *restart function*. (This is somewhat confusing, as a *restart function* is not a function that implements a restart, but a function that *invokes* a specific one.) The `use_value` function mentioned above is an example. -For a detailed API reference, see the module ``unpythonic.conditions``. +For a detailed API reference, see the module `unpythonic.conditions`. #### High-level signaling protocols @@ -3571,7 +3571,7 @@ What we provide here is essentially a rewrite, based on studying that implementa The core idea can be expressed in fewer than 100 lines of Python; ours is (as of v0.14.2) 151 lines, not counting docstrings, comments, or blank lines. The main reason our module is over 700 lines are the docstrings. -### ``generic``, ``typed``, ``isoftype``: multiple dispatch +### `generic`, `typed`, `isoftype`: multiple dispatch **Added in v0.14.2**. @@ -3591,7 +3591,7 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* -The ``generic`` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: ``augment`` adds a new multimethod to an existing generic function, ``typed`` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and ``isoftype`` (which powers the first three) is the big sister of ``isinstance``, with support for many (but unfortunately not all) features of the ``typing`` standard library module. +The `generic` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: `augment` adds a new multimethod to an existing generic function, `typed` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and `isoftype` (which powers the first three) is the big sister of `isinstance`, with support for many (but unfortunately not all) features of the `typing` standard library module. For what kind of things can be done with this, see particularly the [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) example in [`unpythonic.tests.test_dispatch`](../unpythonic/tests/test_dispatch.py). @@ -3607,9 +3607,9 @@ The term *multimethod* distinguishes them from the OOP sense of *method*, alread **CAUTION**: Code using the `with lazify` macro cannot usefully use `@generic` or `@typed`, because all arguments of each function call will be wrapped in a promise (`unpythonic.lazyutil.Lazy`) that carries no type information on its contents. -#### ``generic``: multiple dispatch with type annotation syntax +#### `generic`: multiple dispatch with type annotation syntax -The ``generic`` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. This not only kills boilerplate, but makes the dispatch extensible, since the dispatcher lives outside the original function definition. There is no need to monkey-patch the original to add a new case. +The `generic` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. This not only kills boilerplate, but makes the dispatch extensible, since the dispatcher lives outside the original function definition. There is no need to monkey-patch the original to add a new case. If several multimethods of the same generic function match the arguments given, the most recently registered multimethod wins. @@ -3686,9 +3686,9 @@ assert kittify(x=1, y=2) == "int" assert kittify(x=1.0, y=2.0) == "float" ``` -See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the ``typing`` stdlib module are supported, see ``isoftype`` below. +See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see `isoftype` below. -##### ``@generic`` and OOP +##### `@generic` and OOP As of version 0.14.3, `@generic` and `@typed` can decorate instance methods, class methods and static methods (beside regular functions as in 0.14.2). @@ -3717,9 +3717,9 @@ The machinery itself is also missing some advanced features, such as matching th If you need multiple dispatch, but not the other features of `unpythonic`, see the [multipledispatch](https://github.com/mrocklin/multipledispatch) library, which likely runs faster. -#### ``typed``: add run-time type checks with type annotation syntax +#### `typed`: add run-time type checks with type annotation syntax -The ``typed`` decorator creates a one-multimethod pony, which automatically enforces its argument types. Just like with ``generic``, the type specification may use features from the `typing` stdlib module. +The `typed` decorator creates a one-multimethod pony, which automatically enforces its argument types. Just like with `generic`, the type specification may use features from the `typing` stdlib module. ```python import typing @@ -3742,14 +3742,14 @@ assert jack("foo") == "foo" jack(3.14) # TypeError ``` -For which features of the ``typing`` stdlib module are supported, see ``isoftype`` below. +For which features of the `typing` stdlib module are supported, see `isoftype` below. -#### ``isoftype``: the big sister of ``isinstance`` +#### `isoftype`: the big sister of `isinstance` -Type check object instances against type specifications at run time. This is the machinery that powers ``generic`` and ``typed``. This goes beyond ``isinstance`` in that many (but unfortunately not all) features of the ``typing`` standard library module are supported. +Type check object instances against type specifications at run time. This is the machinery that powers `generic` and `typed`. This goes beyond `isinstance` in that many (but unfortunately not all) features of the `typing` standard library module are supported. -Any checks on the type arguments of the meta-utilities defined in the ``typing`` stdlib module are performed recursively using `isoftype` itself, in order to allow compound abstract specifications. +Any checks on the type arguments of the meta-utilities defined in the `typing` stdlib module are performed recursively using `isoftype` itself, in order to allow compound abstract specifications. Some examples: @@ -3814,7 +3814,7 @@ See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. **CAUTION**: Callables are just checked for being callable; no further analysis is done. Type-checking callables properly requires a much more complex type checker. -**CAUTION**: The `isoftype` function is one big hack. In Python 3.6 through 3.9, there is no consistent way to handle a type specification at run time. We must access some private attributes of the ``typing`` meta-utilities, because that seems to be the only way to get what we need to do this. +**CAUTION**: The `isoftype` function is one big hack. In Python 3.6 through 3.9, there is no consistent way to handle a type specification at run time. We must access some private attributes of the `typing` meta-utilities, because that seems to be the only way to get what we need to do this. If you need a run-time type checker, but not the other features of `unpythonic`, see the [`typeguard`](https://github.com/agronholm/typeguard) library. @@ -3823,7 +3823,7 @@ If you need a run-time type checker, but not the other features of `unpythonic`, Utilities for dealing with exceptions. -### ``raisef``, ``tryf``: ``raise`` and ``try`` as functions +### `raisef`, `tryf`: `raise` and `try` as functions **Changed in v0.14.3**. *Now we have also `tryf`.* @@ -3857,7 +3857,7 @@ The exception handler is a function. It may optionally accept one argument, the Functions can also be specified for the `else` and `finally` behavior; see the docstring of `unpythonic.misc.tryf` for details. -### ``equip_with_traceback`` +### `equip_with_traceback` **Added in v0.14.3**. @@ -3873,7 +3873,7 @@ The traceback is automatically extracted from the call stack of the calling thre Optionally, you can cull a number of the topmost frames by passing the optional argument `stacklevel=...`. Typically, for direct use of this function `stacklevel` should be the default `1` (so it excludes `equip_with_traceback` itself, but shows all stack levels from your code), and for use in a utility function that itself is called from your code, it should be `2` (so it excludes the utility function, too). -### ``async_raise``: inject an exception to another thread +### `async_raise`: inject an exception to another thread **Added in v0.14.2**. @@ -4001,11 +4001,11 @@ If you use the conditions-and-restarts system, see also `resignal_in`, `resignal ## Function call and return value tools -### ``def`` as a code block: ``@call`` +### `def` as a code block: `@call` Fuel for different thinking. Compare `call-with-something` in Lisps - but without parameters, so just `call`. A `def` is really just a new lexical scope to hold code to run later... or right now! -At the top level of a module, this is seldom useful, but keep in mind that Python allows nested function definitions. Used with an inner ``def``, this becomes a versatile tool. +At the top level of a module, this is seldom useful, but keep in mind that Python allows nested function definitions. Used with an inner `def`, this becomes a versatile tool. *Make temporaries fall out of scope as soon as no longer needed*: @@ -4033,7 +4033,7 @@ def result(): print(result) # (6, 7) ``` -(But see ``@catch``, ``throw``, and ``call_ec``.) +(But see `@catch`, `throw`, and `call_ec`.) Compare the sweet-exp Racket: @@ -4048,7 +4048,7 @@ define result displayln result ; (6 7) ``` -Noting [what ``let/ec`` does](https://docs.racket-lang.org/reference/cont.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%2Fec%29%29), using ``call_ec`` we can make the Python even closer to the Racket: +Noting [what `let/ec` does](https://docs.racket-lang.org/reference/cont.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%2Fec%29%29), using `call_ec` we can make the Python even closer to the Racket: ```python @call_ec @@ -4092,12 +4092,12 @@ Essentially the implementation is just `def call(thunk): return thunk()`. The po Note [the grammar](https://docs.python.org/3/reference/grammar.html) requires a newline after a decorator. -**NOTE**: ``call`` can also be used as a normal function: ``call(f, *a, **kw)`` is the same as ``f(*a, **kw)``. This is occasionally useful. +**NOTE**: `call` can also be used as a normal function: `call(f, *a, **kw)` is the same as `f(*a, **kw)`. This is occasionally useful. -### ``@callwith``: freeze arguments, choose function later +### `@callwith`: freeze arguments, choose function later -If you need to pass arguments when using ``@call`` as a decorator, use its cousin ``@callwith``: +If you need to pass arguments when using `@call` as a decorator, use its cousin `@callwith`: ```python from unpythonic import callwith @@ -4108,7 +4108,7 @@ def result(x): assert result == 9 ``` -Like ``call``, it can also be called normally. It's essentially an argument freezer: +Like `call`, it can also be called normally. It's essentially an argument freezer: ```python def myadd(a, b): @@ -4120,13 +4120,13 @@ assert apply23(myadd) == 5 assert apply23(mymul) == 6 ``` -When called normally, the two-step application is mandatory. The first step stores the given arguments. It returns a function ``f(callable)``. When ``f`` is called, it calls its ``callable`` argument, passing in the arguments stored in the first step. +When called normally, the two-step application is mandatory. The first step stores the given arguments. It returns a function `f(callable)`. When `f` is called, it calls its `callable` argument, passing in the arguments stored in the first step. -In other words, ``callwith`` is similar to ``functools.partial``, but without specializing to any particular function. The function to be called is given later, in the second step. +In other words, `callwith` is similar to `functools.partial`, but without specializing to any particular function. The function to be called is given later, in the second step. -Hence, ``callwith(2, 3)(myadd)`` means "make a function that passes in two positional arguments, with values ``2`` and ``3``. Then call this function for the callable ``myadd``". But if we instead write``callwith(2, 3, myadd)``, it means "make a function that passes in three positional arguments, with values ``2``, ``3`` and ``myadd`` - not what we want in the above example. +Hence, `callwith(2, 3)(myadd)` means "make a function that passes in two positional arguments, with values `2` and `3`. Then call this function for the callable `myadd`". But if we instead write`callwith(2, 3, myadd)`, it means "make a function that passes in three positional arguments, with values `2`, `3` and `myadd` - not what we want in the above example. -If you want to specialize some arguments now and some later, combine with ``partial``: +If you want to specialize some arguments now and some later, combine with `partial`: ```python from functools import partial @@ -4145,7 +4145,7 @@ assert apply234(mul3) == 24 If the code above feels weird, it should. Arguments are gathered first, and the function to which they will be passed is chosen in the last step. -Another use case of ``callwith`` is ``map``, if we want to vary the function instead of the data: +Another use case of `callwith` is `map`, if we want to vary the function instead of the data: ```python m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) @@ -4308,9 +4308,9 @@ Test floating-point numbers for near-equality. Beside the built-in `float`, we s Anything else, for example `SymPy` expressions, strings, and containers (regardless of content), is tested for exact equality. -For ``mpmath.mpf``, we just delegate to ``mpmath.almosteq``, with the given tolerance. +For `mpmath.mpf`, we just delegate to `mpmath.almosteq`, with the given tolerance. -For ``float``, we use the strategy suggested in [the floating point guide](https://floating-point-gui.de/errors/comparison/), because naive absolute and relative comparisons against a tolerance fail in commonly encountered situations. +For `float`, we use the strategy suggested in [the floating point guide](https://floating-point-gui.de/errors/comparison/), because naive absolute and relative comparisons against a tolerance fail in commonly encountered situations. ### `fixpoint`: arithmetic fixed-point finder @@ -4379,7 +4379,7 @@ In `partition_int_triangular`, the `lower` and `upper` parameters work exactly t **CAUTION**: The number of possible partitions grows very quickly with `n`, so in practice these functions are only useful for small numbers, or with a lower limit that is not too much smaller than `n / 2`. -### ``ulp``: unit in last place +### `ulp`: unit in last place **Added in v0.14.2.** @@ -4414,7 +4414,7 @@ When `x` is a round number in base-10, the ULP is not, because the usual kind of Stuff that didn't fit elsewhere. -### ``callsite_filename`` +### `callsite_filename` **Added in v0.14.3**. @@ -4423,16 +4423,16 @@ Stuff that didn't fit elsewhere. Return the filename from which this function is being called. Useful as a building block for debug utilities and similar. -### ``safeissubclass`` +### `safeissubclass` **Added in v0.14.3**. Convenience function. Like `issubclass(cls)`, but if `cls` is not a class, swallow the `TypeError` and return `False`. -### ``pack``: multi-arg constructor for tuple +### `pack`: multi-arg constructor for tuple -The default ``tuple`` constructor accepts a single iterable. But sometimes one needs to pass in the elements separately. Most often a literal tuple such as ``(1, 2, 3)`` is then the right solution, but there are situations that do not admit a literal tuple. Enter ``pack``: +The default `tuple` constructor accepts a single iterable. But sometimes one needs to pass in the elements separately. Most often a literal tuple such as `(1, 2, 3)` is then the right solution, but there are situations that do not admit a literal tuple. Enter `pack`: ```python from unpythonic import pack @@ -4443,13 +4443,13 @@ assert tuple(myzip(lol)) == ((1, 3, 5), (2, 4, 6)) ``` -### ``namelambda``: rename a function +### `namelambda`: rename a function -Rename any function object (including lambdas). The return value of ``namelambda`` is a modified copy; the original function object is not mutated. The input can be any function object (``isinstance(f, (types.LambdaType, types.FunctionType))``). It will be renamed even if it already has a name. +Rename any function object (including lambdas). The return value of `namelambda` is a modified copy; the original function object is not mutated. The input can be any function object (`isinstance(f, (types.LambdaType, types.FunctionType))`). It will be renamed even if it already has a name. This is mainly useful in those situations where you return a lambda as a closure, call it much later, and it happens to crash - so you can tell from the stack trace *which* of the *N* lambdas in your codebase it is. -For technical reasons, ``namelambda`` conforms to the parametric decorator API. Usage: +For technical reasons, `namelambda` conforms to the parametric decorator API. Usage: ```python from unpythonic import namelambda @@ -4463,7 +4463,7 @@ kaboom() # --> stack trace, showing the function name "kaboom" The first call returns a *foo-renamer*, which takes a function object and returns a copy that has its name changed to *foo*. -Technically, this updates ``__name__`` (the obvious place), ``__qualname__`` (used by ``repr()``), and ``__code__.co_name`` (used by stack traces). +Technically, this updates `__name__` (the obvious place), `__qualname__` (used by `repr()`), and `__code__.co_name` (used by stack traces). **CAUTION**: There is one pitfall: @@ -4475,10 +4475,10 @@ print(nested.__qualname__) # "outer" print(nested().__qualname__) # "..inner" ``` -The inner lambda does not see the outer's new name; the parent scope names are baked into a function's ``__qualname__`` too early for the outer rename to be in effect at that time. +The inner lambda does not see the outer's new name; the parent scope names are baked into a function's `__qualname__` too early for the outer rename to be in effect at that time. -### ``timer``: a context manager for performance testing +### `timer`: a context manager for performance testing ```python from unpythonic import timer @@ -4493,10 +4493,10 @@ with timer(p=True): # if p, auto-print result pass ``` -The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the ``with ... as ...`` syntax. After the context exits, the *Δt* is available in its ``dt`` attribute. +The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the `with ... as ...` syntax. After the context exits, the *Δt* is available in its `dt` attribute. -### ``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers +### `getattrrec`, `setattrrec`: access underlying data in an onion of wrappers ```python from unpythonic import getattrrec, setattrrec @@ -4517,7 +4517,7 @@ assert getattrrec(w, "x") == 23 ``` -### ``arities``, ``kwargs``, ``resolve_bindings``: Function signature inspection utilities +### `arities`, `kwargs`, `resolve_bindings`: Function signature inspection utilities **Added in v0.14.2**: `resolve_bindings`. *Get the parameter bindings a given callable would establish if it was called with the given args and kwargs. This is mainly of interest for implementing memoizers, since this allows them to see (e.g.) `f(1)` and `f(a=1)` as the same thing for `def f(a): pass`.* @@ -4525,9 +4525,9 @@ assert getattrrec(w, "x") == 23 *Now `tuplify_bindings` accepts an `inspect.BoundArguments` object instead of its previous input format. The function is only ever intended to be used to postprocess the output of `resolve_bindings`, so this change shouldn't affect your own code.* -Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by ``inspect``. +Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by `inspect`. -Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit ``self`` and ``cls`` are ignored). +Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit `self` and `cls` are ignored). ```python from unpythonic import (arities, arity_includes, UnknownArity, @@ -4582,16 +4582,16 @@ assert tuple(resolve_bindings(f, 1, c=3, b=2).items()) == (("a", 1), ("b", 2), ( assert tuple(resolve_bindings(f, c=3, b=2, a=1).items()) == (("a", 1), ("b", 2), ("c", 3)) ``` -We special-case the builtin functions that either fail to return any arity (are uninspectable) or report incorrect arity information, so that also their arities are reported correctly. Note we **do not** special-case the *methods* of any builtin classes, so e.g. ``list.append`` remains uninspectable. This limitation might or might not be lifted in a future version. +We special-case the builtin functions that either fail to return any arity (are uninspectable) or report incorrect arity information, so that also their arities are reported correctly. Note we **do not** special-case the *methods* of any builtin classes, so e.g. `list.append` remains uninspectable. This limitation might or might not be lifted in a future version. -If the arity cannot be inspected, and the function is not one of the special-cased builtins, the ``UnknownArity`` exception is raised. +If the arity cannot be inspected, and the function is not one of the special-cased builtins, the `UnknownArity` exception is raised. -These functions are internally used in various places in unpythonic, particularly ``curry``, ``fix``, and ``@generic``. The ``let`` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. +These functions are internally used in various places in unpythonic, particularly `curry`, `fix`, and `@generic`. The `let` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. -Inspired by various Racket functions such as ``(arity-includes?)`` and ``(procedure-keywords)``. +Inspired by various Racket functions such as `(arity-includes?)` and `(procedure-keywords)`. -### ``Popper``: a pop-while iterator +### `Popper`: a pop-while iterator Consider this highly artificial example: @@ -4607,7 +4607,7 @@ assert inp == deque([]) assert out == list(range(5)) ``` -``Popper`` condenses the ``while`` and ``pop`` into a ``for``, while allowing the loop body to mutate the input iterable in arbitrary ways (we never actually ``iter()`` it): +`Popper` condenses the `while` and `pop` into a `for`, while allowing the loop body to mutate the input iterable in arbitrary ways (we never actually `iter()` it): ```python from collections import deque @@ -4630,7 +4630,7 @@ assert inp == deque([]) assert out == [0, 10, 1, 11, 2, 12] ``` -``Popper`` comboes with other iterable utilities, such as ``window``: +`Popper` comboes with other iterable utilities, such as `window`: ```python from collections import deque @@ -4646,14 +4646,14 @@ assert inp == deque([]) assert out == [(0, 1), (1, 2), (2, 10), (10, 11), (11, 12)] ``` -(Although ``window`` invokes ``iter()`` on the ``Popper``, this works because the ``Popper`` never invokes ``iter()`` on the underlying container. Any mutations to the input container performed by the loop body will be understood by ``Popper`` and thus also seen by the ``window``. The first ``n`` elements, though, are read before the loop body gets control, because the window needs them to initialize itself.) +(Although `window` invokes `iter()` on the `Popper`, this works because the `Popper` never invokes `iter()` on the underlying container. Any mutations to the input container performed by the loop body will be understood by `Popper` and thus also seen by the `window`. The first `n` elements, though, are read before the loop body gets control, because the window needs them to initialize itself.) -One possible real use case for ``Popper`` is to split sequences of items, stored as lists in a deque, into shorter sequences where some condition is contiguously ``True`` or ``False``. When the condition changes state, just commit the current subsequence, and push the rest of that input sequence (still requiring analysis) back to the input deque, to be dealt with later. +One possible real use case for `Popper` is to split sequences of items, stored as lists in a deque, into shorter sequences where some condition is contiguously `True` or `False`. When the condition changes state, just commit the current subsequence, and push the rest of that input sequence (still requiring analysis) back to the input deque, to be dealt with later. -The argument to ``Popper`` (here ``lst``) contains the **remaining** items. Each iteration pops an element **from the left**. The loop terminates when ``lst`` is empty. +The argument to `Popper` (here `lst`) contains the **remaining** items. Each iteration pops an element **from the left**. The loop terminates when `lst` is empty. -The input container must support either ``popleft()`` or ``pop(0)``. This is fully duck-typed. At least ``collections.deque`` and any ``collections.abc.MutableSequence`` (including ``list``) are fine. +The input container must support either `popleft()` or `pop(0)`. This is fully duck-typed. At least `collections.deque` and any `collections.abc.MutableSequence` (including `list`) are fine. -Per-iteration efficiency is O(1) for ``collections.deque``, and O(n) for a ``list``. +Per-iteration efficiency is O(1) for `collections.deque`, and O(n) for a `list`. Named after [Karl Popper](https://en.wikipedia.org/wiki/Karl_Popper). From 0d44961428f2cdd3896d719250a15f62302ca8a8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 16:47:44 +0300 Subject: [PATCH 521/832] markdown: use single backticks --- doc/macros.md | 630 +++++++++++++++++++++++++------------------------- 1 file changed, 315 insertions(+), 315 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 5fd715e7..918781bb 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -11,9 +11,9 @@ - [Additional reading](readings.md) - [Contribution guidelines](../CONTRIBUTING.md) -# Language extensions using ``unpythonic.syntax`` +# Language extensions using `unpythonic.syntax` -Our extensions to the Python language are built on [``mcpyrate``](https://github.com/Technologicat/mcpyrate), from the PyPI package [``mcpyrate``](https://pypi.org/project/mcpyrate/). +Our extensions to the Python language are built on [`mcpyrate`](https://github.com/Technologicat/mcpyrate), from the PyPI package [`mcpyrate`](https://pypi.org/project/mcpyrate/). Because in Python macro expansion occurs *at import time*, Python programs whose main module uses macros, such as [our unit tests that contain usage examples](../unpythonic/syntax/test/), cannot be run directly. Instead, run them via `macropython`, included in `mcpyrate`. @@ -29,50 +29,50 @@ Because in Python macro expansion occurs *at import time*, Python programs whose ### Features [**Bindings**](#bindings) -- [``let``, ``letseq``, ``letrec`` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. -- [``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) -- [``let_syntax``, ``abbrev``: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. -- [Bonus: barebones ``let``](#bonus-barebones-let): pure AST transformation of ``let`` into a ``lambda``. +- [`let`, `letseq`, `letrec` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. +- [`dlet`, `dletseq`, `dletrec`, `blet`, `bletseq`, `bletrec`: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) +- [`let_syntax`, `abbrev`: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. +- [Bonus: barebones `let`](#bonus-barebones-let): pure AST transformation of `let` into a `lambda`. [**Sequencing**](#sequencing) -- [``do`` as a macro: stuff imperative code into an expression, *with style*](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) +- [`do` as a macro: stuff imperative code into an expression, *with style*](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) [**Tools for lambdas**](#tools-for-lambdas) -- [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. -- [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. -- [``fn``: underscore notation (quick lambdas) for Python](#f-underscore-notation-quick-lambdas-for-python) -- [``quicklambda``: expand quick lambdas first](#quicklambda-expand-quick-lambdas-first) -- [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) +- [`multilambda`: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. +- [`namedlambda`: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. +- [`fn`: underscore notation (quick lambdas) for Python](#f-underscore-notation-quick-lambdas-for-python) +- [`quicklambda`: expand quick lambdas first](#quicklambda-expand-quick-lambdas-first) +- [`envify`: make formal parameters live in an unpythonic `env`](#envify-make-formal-parameters-live-in-an-unpythonic-env) [**Language features**](#language-features) -- [``autocurry``: automatic currying for Python](#autocurry-automatic-currying-for-python) -- [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) - - [``lazy[]`` and ``lazyrec[]`` macros](#lazy-and-lazyrec-macros) +- [`autocurry`: automatic currying for Python](#autocurry-automatic-currying-for-python) +- [`lazify`: call-by-need for Python](#lazify-call-by-need-for-python) + - [`lazy[]` and `lazyrec[]` macros](#lazy-and-lazyrec-macros) - [Forcing promises manually](#forcing-promises-manually) - [Binding constructs and auto-lazification](#binding-constructs-and-auto-lazification) - [Note about TCO](#note-about-tco) -- [``tco``: automatic tail call optimization for Python](#tco-automatic-tail-call-optimization-for-python) +- [`tco`: automatic tail call optimization for Python](#tco-automatic-tail-call-optimization-for-python) - [TCO and continuations](#tco-and-continuations) -- [``continuations``: call/cc for Python](#continuations-callcc-for-python) +- [`continuations`: call/cc for Python](#continuations-callcc-for-python) - [General remarks on continuations](#general-remarks-on-continuations) - - [Differences between ``call/cc`` and certain other language features](#differences-between-callcc-and-certain-other-language-features) (generators, exceptions) - - [``call_cc`` API reference](#call_cc-api-reference) + - [Differences between `call/cc` and certain other language features](#differences-between-callcc-and-certain-other-language-features) (generators, exceptions) + - [`call_cc` API reference](#call_cc-api-reference) - [Combo notes](#combo-notes) - [Continuations as an escape mechanism](#continuations-as-an-escape-mechanism) - [What can be used as a continuation?](#what-can-be-used-as-a-continuation) - - [This isn't ``call/cc``!](#this-isnt-callcc) + - [This isn't `call/cc`!](#this-isnt-callcc) - [Why this syntax?](#why-this-syntax) -- [``prefix``: prefix function call syntax for Python](#prefix-prefix-function-call-syntax-for-python) -- [``autoreturn``: implicit ``return`` in tail position](#autoreturn-implicit-return-in-tail-position), like in Lisps. -- [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation) with monadic do-notation for Python. +- [`prefix`: prefix function call syntax for Python](#prefix-prefix-function-call-syntax-for-python) +- [`autoreturn`: implicit `return` in tail position](#autoreturn-implicit-return-in-tail-position), like in Lisps. +- [`forall`: nondeterministic evaluation](#forall-nondeterministic-evaluation) with monadic do-notation for Python. [**Convenience features**](#convenience-features) -- [``cond``: the missing ``elif`` for ``a if p else b``](#cond-the-missing-elif-for-a-if-p-else-b) -- [``aif``: anaphoric if](#aif-anaphoric-if), the test result is ``it``. -- [``autoref``: implicitly reference attributes of an object](#autoref-implicitly-reference-attributes-of-an-object) +- [`cond`: the missing `elif` for `a if p else b`](#cond-the-missing-elif-for-a-if-p-else-b) +- [`aif`: anaphoric if](#aif-anaphoric-if), the test result is `it`. +- [`autoref`: implicitly reference attributes of an object](#autoref-implicitly-reference-attributes-of-an-object) [**Testing and debugging**](#testing-and-debugging) -- [``unpythonic.test.fixtures``: a test framework for macro-enabled Python](#unpythonic-test-fixtures-a-test-framework-for-macro-enabled-python) +- [`unpythonic.test.fixtures`: a test framework for macro-enabled Python](#unpythonic-test-fixtures-a-test-framework-for-macro-enabled-python) - [Overview](#overview) - [Testing syntax quick reference](#testing-syntax-quick-reference) - [Expansion order](#expansion-order) @@ -83,10 +83,10 @@ Because in Python macro expansion occurs *at import time*, Python programs whose - [Advanced: building a custom test framework](#advanced-building-a-custom-test-framework) - [Why another test framework?](#why-another-test-framework) - [Etymology and roots](#etymology-and-roots) -- [``dbg``: debug-print expressions with source code](#dbg-debug-print-expressions-with-source-code) +- [`dbg`: debug-print expressions with source code](#dbg-debug-print-expressions-with-source-code) [**Other**](#other) -- [``nb``: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) +- [`nb`: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) [**Meta**](#meta) - [The xmas tree combo](#the-xmas-tree-combo): notes on the macros working together. @@ -97,11 +97,11 @@ Because in Python macro expansion occurs *at import time*, Python programs whose Macros that introduce new ways to bind identifiers. -### ``let``, ``letseq``, ``letrec`` as macros +### `let`, `letseq`, `letrec` as macros **Changed in v0.15.0.** *Added support for env-assignment syntax in the bindings subform. For consistency with other env-assignments, this is now the preferred syntax to establish let bindings. Additionally, the old lispy syntax now accepts also brackets, for consistency with the use of brackets for macro invocations.* -Properly lexically scoped ``let`` constructs, no boilerplate: +Properly lexically scoped `let` constructs, no boilerplate: ```python from unpythonic.syntax import macros, let, letseq, letrec @@ -127,13 +127,13 @@ let[x << 21][2 * x] There must be at least one binding; `let[][...]` is a syntax error, since Python's parser rejects an empty subscript slice. -Bindings are established using the `unpythonic` *env-assignment* syntax, ``name << value``. The let bindings can be rebound in the body with the same env-assignment syntax, e.g. ``x << 42``. +Bindings are established using the `unpythonic` *env-assignment* syntax, `name << value`. The let bindings can be rebound in the body with the same env-assignment syntax, e.g. `x << 42`. The same syntax for the bindings subform is used by: -- ``let``, ``letseq``, ``letrec`` (expressions) -- ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec`` (decorators) -- ``let_syntax``, ``abbrev`` (expression mode) +- `let`, `letseq`, `letrec` (expressions) +- `dlet`, `dletseq`, `dletrec`, `blet`, `bletseq`, `bletrec` (decorators) +- `let_syntax`, `abbrev` (expression mode) #### Haskelly let-in, let-where @@ -155,7 +155,7 @@ let[[x << 21] in 2 * x] let[2 * x, where[x << 21]] ``` -These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the ``...`` in `let[...]`. +These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the `...` in `let[...]`. Note the bindings subform is always enclosed by brackets. @@ -166,11 +166,11 @@ The `where` operator, if used, must be macro-imported. It may only appear at the >The bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. > ->These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the formats where the bindings subform is given in the macro arguments part, because there the body is in any case placed differently (it's the body of the function being decorated). +>These syntaxes are valid for all **expression forms** of `let`, namely: `let[]`, `letseq[]`, `letrec[]`, `let_syntax[]` and `abbrev[]`. The decorator variants (`dlet` et al., `blet` et al.) and the block variants (`with let_syntax`, `with abbrev`) support only the formats where the bindings subform is given in the macro arguments part, because there the body is in any case placed differently (it's the body of the function being decorated). > ->In the first variant above (the *let-in*), note that even there, the bindings block needs the brackets. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make the ``in`` refer to all of the bindings, the bindings block must be bracketed. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your brackets. +>In the first variant above (the *let-in*), note that even there, the bindings block needs the brackets. This is due to Python's precedence rules; `in` binds more strongly than the comma (which makes sense almost everywhere else), so to make the `in` refer to all of the bindings, the bindings block must be bracketed. If the `let` expander complains your code does not look like a `let` form and you have used *let-in*, check your brackets. > ->In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression. It's not only syntactically valid Python, it's also syntactically valid English (at least for mathematicians).) +>In the second variant (the *let-where*), note the comma between the body and `where`; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression. It's not only syntactically valid Python, it's also syntactically valid English (at least for mathematicians).)
#### Alternate syntaxes for the bindings subform @@ -207,7 +207,7 @@ let[(x, 42) in ...] let[..., where(x, 42)] ``` -Even though an expr macro invocation itself is always denoted using brackets, as of `unpythonic` v0.15.0 parentheses can still be used *to pass macro arguments*, hence ``let(...)[...]`` is still accepted. The code that interprets the AST for the let bindings accepts both lists and tuples for each key-value pair, and the top-level container for the bindings subform in a let-in or let-where can be either list or tuple, so whether brackets or parentheses are used does not matter there, either. +Even though an expr macro invocation itself is always denoted using brackets, as of `unpythonic` v0.15.0 parentheses can still be used *to pass macro arguments*, hence `let(...)[...]` is still accepted. The code that interprets the AST for the let bindings accepts both lists and tuples for each key-value pair, and the top-level container for the bindings subform in a let-in or let-where can be either list or tuple, so whether brackets or parentheses are used does not matter there, either. Still, brackets are now the preferred delimiter, for consistency between the bindings and body subforms. @@ -237,9 +237,9 @@ let[[y << x + y, y << 2]] ``` -The let macros implement this by inserting a ``do[...]`` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see [``do`` for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). +The let macros implement this by inserting a `do[...]` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the `let`; see [`do` for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). -Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a ``let`` form with only one body expression, use three sets of brackets: +Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a `let` form with only one body expression, use three sets of brackets: ```python let[x << 1, @@ -255,7 +255,7 @@ let[[[x, y]], y << 2]] ``` -The outermost brackets delimit the ``let`` form, the middle ones activate multiple-expression mode, and the innermost ones denote a list. +The outermost brackets delimit the `let` form, the middle ones activate multiple-expression mode, and the innermost ones denote a list. Only brackets are affected; parentheses are interpreted as usual, so returning a literal tuple works as expected: @@ -277,7 +277,7 @@ let[(x, y), The main difference of the `let` family to Python's own named expressions (a.k.a. walrus operator, added in Python 3.8) is that `x := 42` does not create a scope, but `let[(x, 42)][...]` does. The walrus operator assigns to the name `x` in the scope it appears in, whereas in the `let` expression, the `x` only exists in that expression. -``let`` and ``letrec`` expand into the ``unpythonic.lispylet`` constructs, implicitly inserting the necessary boilerplate: the ``lambda e: ...`` wrappers, quoting variable names in definitions, and transforming ``x`` to ``e.x`` for all ``x`` declared in the bindings. Assignment syntax ``x << 42`` transforms to ``e.set('x', 42)``. The implicit environment parameter ``e`` is actually named using a gensym, so lexically outer environments automatically show through. ``letseq`` expands into a chain of nested ``let`` expressions. +`let` and `letrec` expand into the `unpythonic.lispylet` constructs, implicitly inserting the necessary boilerplate: the `lambda e: ...` wrappers, quoting variable names in definitions, and transforming `x` to `e.x` for all `x` declared in the bindings. Assignment syntax `x << 42` transforms to `e.set('x', 42)`. The implicit environment parameter `e` is actually named using a gensym, so lexically outer environments automatically show through. `letseq` expands into a chain of nested `let` expressions. Nesting utilizes an inside-out macro expansion order: @@ -288,12 +288,12 @@ letrec[z << 1][[ print(z)]]] ``` -Hence the ``z`` in the inner scope expands to the inner environment's ``z``, which makes the outer expansion leave it alone. (This works by transforming only ``ast.Name`` nodes, stopping recursion when an ``ast.Attribute`` is encountered.) +Hence the `z` in the inner scope expands to the inner environment's `z`, which makes the outer expansion leave it alone. (This works by transforming only `ast.Name` nodes, stopping recursion when an `ast.Attribute` is encountered.) -### ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions +### `dlet`, `dletseq`, `dletrec`, `blet`, `bletseq`, `bletrec`: decorator versions -Similar to ``let``, ``letseq``, ``letrec``, these sugar the corresponding ``unpythonic.lispylet`` constructs, with the ``dletseq`` and ``bletseq`` constructs existing only as macros (expanding to nested ``dlet`` or ``blet``, respectively). +Similar to `let`, `letseq`, `letrec`, these sugar the corresponding `unpythonic.lispylet` constructs, with the `dletseq` and `bletseq` constructs existing only as macros (expanding to nested `dlet` or `blet`, respectively). Lexical scoping is respected; each environment is internally named using a gensym. Nesting is allowed. @@ -343,13 +343,13 @@ def result(): assert result == 4 ``` -**CAUTION**: assignment to the let environment uses the syntax ``name << value``, as always with ``unpythonic`` environments. The standard Python syntax ``name = value`` creates a local variable, as usual - *shadowing any variable with the same name from the ``let``*. +**CAUTION**: assignment to the let environment uses the syntax `name << value`, as always with `unpythonic` environments. The standard Python syntax `name = value` creates a local variable, as usual - *shadowing any variable with the same name from the `let`*. -The write of a ``name << value`` always occurs to the lexically innermost environment (as seen from the write site) that has that ``name``. If no lexically surrounding environment has that ``name``, *then* the expression remains untransformed, and means a left-shift (if ``name`` happens to be otherwise defined). +The write of a `name << value` always occurs to the lexically innermost environment (as seen from the write site) that has that `name`. If no lexically surrounding environment has that `name`, *then* the expression remains untransformed, and means a left-shift (if `name` happens to be otherwise defined). -**CAUTION**: formal parameters of a function definition, local variables, and any names declared as ``global`` or ``nonlocal`` in a given lexical scope shadow names from the ``let`` environment. Mostly, this applies *to the entirety of that lexical scope*. This is modeled after Python's standard scoping rules. +**CAUTION**: formal parameters of a function definition, local variables, and any names declared as `global` or `nonlocal` in a given lexical scope shadow names from the `let` environment. Mostly, this applies *to the entirety of that lexical scope*. This is modeled after Python's standard scoping rules. -As an exception to the rule, for the purposes of the scope analysis performed by ``unpythonic.syntax``, creations and deletions *of lexical local variables* take effect from the next statement, and remain in effect for the **lexically** remaining part of the current scope. This allows ``x = ...`` to see the old bindings on the RHS, as well as allows the client code to restore access to a surrounding env's ``x`` (by deleting a local ``x`` shadowing it) when desired. +As an exception to the rule, for the purposes of the scope analysis performed by `unpythonic.syntax`, creations and deletions *of lexical local variables* take effect from the next statement, and remain in effect for the **lexically** remaining part of the current scope. This allows `x = ...` to see the old bindings on the RHS, as well as allows the client code to restore access to a surrounding env's `x` (by deleting a local `x` shadowing it) when desired. To clarify, here's a sampling from the unit tests: @@ -400,7 +400,7 @@ else: ``` -### ``let_syntax``, ``abbrev``: syntactic local bindings +### `let_syntax`, `abbrev`: syntactic local bindings **Note v0.15.0.** *Now that we use `mcpyrate` as the macro expander, `let_syntax` and `abbrev` are not really needed. We are keeping them mostly for backwards compatibility, and because they exercise a different feature set in the macro expander, making the existence of these constructs particularly useful for system testing.* @@ -410,7 +410,7 @@ else: These constructs allow to locally splice code at macro expansion time (it's almost like inlining functions): -#### ``let_syntax`` +#### `let_syntax` ```python from unpythonic.syntax import macros, let_syntax, block, expr @@ -467,28 +467,28 @@ with let_syntax: assert lst == [7, 8, 9]*2 ``` -After macro expansion completes, ``let_syntax`` has zero runtime overhead; it completely disappears in macro expansion. +After macro expansion completes, `let_syntax` has zero runtime overhead; it completely disappears in macro expansion. The `expr` and `block` operators, if used, must be macro-imported. They may only appear in `with expr` and `with block` subforms at the top level of a `with let_syntax` or `with abbrev`. In any invalid position, `expr` and `block` are both considered a syntax error at macro expansion time.
There are two kinds of substitutions: ->*Bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are **not** supported.) +>*Bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, `*args`, `**kwargs` and default values are **not** supported.) > ->When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, ``f[a]`` is a template with a formal parameter ``a``. But when used as a block macro, the formal parameters are declared on the ``block`` or ``expr`` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use ``with block as ...:`` or ``with expr as ...:`` with no macro arguments. +>When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, `f[a]` is a template with a formal parameter `a`. But when used as a block macro, the formal parameters are declared on the `block` or `expr` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use `with block as ...:` or `with expr as ...:` with no macro arguments. > ->In the body of ``let_syntax``, a bare name substitution is invoked by name (just like a variable). A template substitution is invoked like an expr macro. Any instances of the formal parameters of the template get replaced by the argument values from the use site, at macro expansion time. +>In the body of `let_syntax`, a bare name substitution is invoked by name (just like a variable). A template substitution is invoked like an expr macro. Any instances of the formal parameters of the template get replaced by the argument values from the use site, at macro expansion time. > ->Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each ``a`` in the body of ``twice`` separately expands to a copy of whatever code was given as the macro argument ``a``. +>Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each `a` in the body of `twice` separately expands to a copy of whatever code was given as the macro argument `a`. > ->When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. (The single expression can be an explicit ``do[]`` if multiple expressions are needed.) When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. +>When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. (The single expression can be an explicit `do[]` if multiple expressions are needed.) When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. > ->(If you know about Python ASTs, don't worry about the ``ast.Expr`` wrapper needed to place an expression in a statement position; this is handled automatically.) +>(If you know about Python ASTs, don't worry about the `ast.Expr` wrapper needed to place an expression in a statement position; this is handled automatically.)

-**HINT**: If you get a compiler error that some sort of statement was encountered where an expression was expected, check your uses of ``let_syntax``. The most likely reason is that a substitution is trying to splice a block of statements into an expression position. +**HINT**: If you get a compiler error that some sort of statement was encountered where an expression was expected, check your uses of `let_syntax`. The most likely reason is that a substitution is trying to splice a block of statements into an expression position.

Expansion of this macro is a two-step process: @@ -500,20 +500,20 @@ The `expr` and `block` operators, if used, must be macro-imported. They may only > >Within each step, the substitutions are applied **in definition order**: > -> - If the bindings are ``[x << y, y << z]``, then an ``x`` at the use site transforms to ``z``. So does a ``y`` at the use site. -> - But if the bindings are ``[y << z, x << y]``, then an ``x`` at the use site transforms to ``y``, and only an explicit ``y`` at the use site transforms to ``z``. +> - If the bindings are `[x << y, y << z]`, then an `x` at the use site transforms to `z`. So does a `y` at the use site. +> - But if the bindings are `[y << z, x << y]`, then an `x` at the use site transforms to `y`, and only an explicit `y` at the use site transforms to `z`. > >Even in block templates, arguments are always expressions, because invoking a template uses the subscript syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above.

-Nesting ``let_syntax`` is allowed. Lexical scoping is supported (inner definitions of substitutions shadow outer ones). +Nesting `let_syntax` is allowed. Lexical scoping is supported (inner definitions of substitutions shadow outer ones). -When used as an expr macro, all bindings are registered first, and then the body is evaluated. When used as a block macro, a new binding (substitution declaration) takes effect from the next statement onward, and remains active for the lexically remaining part of the ``with let_syntax:`` block. +When used as an expr macro, all bindings are registered first, and then the body is evaluated. When used as a block macro, a new binding (substitution declaration) takes effect from the next statement onward, and remains active for the lexically remaining part of the `with let_syntax:` block. #### `abbrev` -The ``abbrev`` macro is otherwise exactly like ``let_syntax``, but it expands outside-in. Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the ``abbrev`` itself expands before any macros invoked in its body. This allows things like: +The `abbrev` macro is otherwise exactly like `let_syntax`, but it expands outside-in. Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the `abbrev` itself expands before any macros invoked in its body. This allows things like: ```python abbrev[m << macrowithverylongname][ @@ -526,18 +526,18 @@ abbrev[m[tree1] if m[tree2] else m[tree3], which can be useful when writing macros. -**CAUTION**: ``let_syntax`` is essentially a toy macro system within the real macro system. The usual caveats of macro systems apply. Especially, ``let_syntax`` and ``abbrev`` support absolutely no form of hygiene. Be very, very careful to avoid name conflicts. +**CAUTION**: `let_syntax` is essentially a toy macro system within the real macro system. The usual caveats of macro systems apply. Especially, `let_syntax` and `abbrev` support absolutely no form of hygiene. Be very, very careful to avoid name conflicts. -The ``let_syntax`` macro is meant for simple local substitutions where the elimination of repetition can shorten the code and improve its readability, in cases where the final "unrolled" code should be written out at compile time. If you need to do something complex (or indeed save a definition and reuse it somewhere else, non-locally), write a real macro directly in `mcpyrate`. +The `let_syntax` macro is meant for simple local substitutions where the elimination of repetition can shorten the code and improve its readability, in cases where the final "unrolled" code should be written out at compile time. If you need to do something complex (or indeed save a definition and reuse it somewhere else, non-locally), write a real macro directly in `mcpyrate`. -This was inspired by Racket's [``let-syntax``](https://docs.racket-lang.org/reference/let.html) and [``with-syntax``](https://docs.racket-lang.org/reference/stx-patterns.html). +This was inspired by Racket's [`let-syntax`](https://docs.racket-lang.org/reference/let.html) and [`with-syntax`](https://docs.racket-lang.org/reference/stx-patterns.html). -### Bonus: barebones ``let`` +### Bonus: barebones `let` -As a bonus, we provide classical simple ``let`` and ``letseq``, wholly implemented as AST transformations, providing true lexical variables but no assignment support (because in Python, assignment is a statement) or multi-expression body support. Just like in Lisps, this version of ``letseq`` (Scheme/Racket ``let*``) expands into a chain of nested ``let`` expressions, which expand to lambdas. +As a bonus, we provide classical simple `let` and `letseq`, wholly implemented as AST transformations, providing true lexical variables but no assignment support (because in Python, assignment is a statement) or multi-expression body support. Just like in Lisps, this version of `letseq` (Scheme/Racket `let*`) expands into a chain of nested `let` expressions, which expand to lambdas. -These are provided in the separate module ``unpythonic.syntax.simplelet``, and are not part of the `unpythonic.syntax` macro API. For simplicity, they support only the lispy list syntax in the bindings subform (using brackets, specifically!), and no haskelly syntax at all: +These are provided in the separate module `unpythonic.syntax.simplelet`, and are not part of the `unpythonic.syntax` macro API. For simplicity, they support only the lispy list syntax in the bindings subform (using brackets, specifically!), and no haskelly syntax at all: ```python from unpythonic.syntax.simplelet import macros, let, letseq @@ -552,9 +552,9 @@ letseq[[x, 1]][...] Macros that run multiple expressions, in sequence, in place of one expression. -### ``do`` as a macro: stuff imperative code into an expression, *with style* +### `do` as a macro: stuff imperative code into an expression, *with style* -We provide an ``expr`` macro wrapper for ``unpythonic.seq.do``, with some extra features. +We provide an `expr` macro wrapper for `unpythonic.seq.do`, with some extra features. This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see [the functions in `unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). @@ -575,11 +575,11 @@ y = do[local[a << 17], True] ``` -Local variables are declared and initialized with ``local[var << value]``, where ``var`` is a bare name. To explicitly denote "no value", just use ``None``. ``delete[...]`` allows deleting a ``local[...]`` binding. This uses ``env.pop()`` internally, so a ``delete[...]`` returns the value the deleted local variable had at the time of deletion. (So if you manually use the ``do()`` function in some code without macros, feel free to ``env.pop()`` in a do-item if needed.) +Local variables are declared and initialized with `local[var << value]`, where `var` is a bare name. To explicitly denote "no value", just use `None`. `delete[...]` allows deleting a `local[...]` binding. This uses `env.pop()` internally, so a `delete[...]` returns the value the deleted local variable had at the time of deletion. (So if you manually use the `do()` function in some code without macros, feel free to `env.pop()` in a do-item if needed.) The `local[]` and `delete[]` declarations may only appear at the top level of a `do[]`, `do0[]`, or implicit `do` (extra bracket syntax, e.g. for the body of a `let` form). In any invalid position, `local[]` and `delete[]` are considered a syntax error at macro expansion time. -A ``local`` declaration comes into effect in the expression following the one where it appears, capturing the declared name as a local variable for the **lexically** remaining part of the ``do``. In a ``local``, the RHS still sees the previous bindings, so this is valid (although maybe not readable): +A `local` declaration comes into effect in the expression following the one where it appears, capturing the declared name as a local variable for the **lexically** remaining part of the `do`. In a `local`, the RHS still sees the previous bindings, so this is valid (although maybe not readable): ```python result = [] @@ -589,32 +589,32 @@ let[lst << []][[result.append(lst), # the let "lst" assert result == [[], [1]] ``` -Already declared local variables are updated with ``var << value``. Updating variables in lexically outer environments (e.g. a ``let`` surrounding a ``do``) uses the same syntax. +Already declared local variables are updated with `var << value`. Updating variables in lexically outer environments (e.g. a `let` surrounding a `do`) uses the same syntax.

The reason we require local variables to be declared is to allow write access to lexically outer environments. ->Assignments are recognized anywhere inside the ``do``; but note that any ``let`` constructs nested *inside* the ``do``, that define variables of the same name, will (inside the ``let``) shadow those of the ``do`` - as expected of lexical scoping. +>Assignments are recognized anywhere inside the `do`; but note that any `let` constructs nested *inside* the `do`, that define variables of the same name, will (inside the `let`) shadow those of the `do` - as expected of lexical scoping. > ->The necessary boilerplate (notably the ``lambda e: ...`` wrappers) is inserted automatically, so the expressions in a ``do[]`` are only evaluated when the underlying ``seq.do`` actually runs. +>The necessary boilerplate (notably the `lambda e: ...` wrappers) is inserted automatically, so the expressions in a `do[]` are only evaluated when the underlying `seq.do` actually runs. > ->When running, ``do`` behaves like ``letseq``; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites (this is afterall an imperative tool). +>When running, `do` behaves like `letseq`; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites (this is afterall an imperative tool). > ->We also provide a ``do0`` macro, which returns the value of the first expression, instead of the last. +>We also provide a `do0` macro, which returns the value of the first expression, instead of the last.

-**CAUTION**: ``do[]`` supports local variable deletion, but the ``let[]`` constructs don't, by design. When ``do[]`` is used implicitly with the extra bracket syntax, any ``delete[]`` refers to the scope of the implicit ``do[]``, not any surrounding ``let[]`` scope. +**CAUTION**: `do[]` supports local variable deletion, but the `let[]` constructs don't, by design. When `do[]` is used implicitly with the extra bracket syntax, any `delete[]` refers to the scope of the implicit `do[]`, not any surrounding `let[]` scope. ## Tools for lambdas Macros that introduce additional features for Python's lambdas. -### ``multilambda``: supercharge your lambdas +### `multilambda`: supercharge your lambdas -**Multiple expressions**: use ``[...]`` to denote a multiple-expression body. The macro implements this by inserting a ``do``. +**Multiple expressions**: use `[...]` to denote a multiple-expression body. The macro implements this by inserting a `do`. -**Local variables**: available in a multiple-expression body. For details on usage, see ``do``. +**Local variables**: available in a multiple-expression body. For details on usage, see `do`. ```python from unpythonic.syntax import macros, multilambda, let @@ -647,10 +647,10 @@ with multilambda: assert t() == [1, 2] ``` -In the second example, returning ``x`` separately is redundant, because the assignment to the let environment already returns the new value, but it demonstrates the usage of multiple expressions in a lambda. +In the second example, returning `x` separately is redundant, because the assignment to the let environment already returns the new value, but it demonstrates the usage of multiple expressions in a lambda. -### ``namedlambda``: auto-name your lambdas +### `namedlambda`: auto-name your lambdas **Changed in v0.15.0.** *When `namedlambda` encounters a lambda definition it cannot infer a name for, it instead injects source location info into the name, provided that the AST node for that particular `lambda` has a line number for it. The result looks like ``.* @@ -684,34 +684,34 @@ with namedlambda: assert d["g"].__name__ == "g" ``` -Lexically inside a ``with namedlambda`` block, any literal ``lambda`` that is assigned to a name using one of the supported assignment forms is named to have the name of the LHS of the assignment. The name is captured at macro expansion time. +Lexically inside a `with namedlambda` block, any literal `lambda` that is assigned to a name using one of the supported assignment forms is named to have the name of the LHS of the assignment. The name is captured at macro expansion time. -Decorated lambdas are also supported, as is a ``curry`` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/test/test_lambdatools.py) for detailed examples. +Decorated lambdas are also supported, as is a `curry` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/test/test_lambdatools.py) for detailed examples. -The naming is performed using the function ``unpythonic.misc.namelambda``, which will return a modified copy with its ``__name__``, ``__qualname__`` and ``__code__.co_name`` changed. The original function object is not mutated. +The naming is performed using the function `unpythonic.misc.namelambda`, which will return a modified copy with its `__name__`, `__qualname__` and `__code__.co_name` changed. The original function object is not mutated. **Supported assignment forms**: - - Single-item assignment to a local name, ``f = lambda ...: ...`` + - Single-item assignment to a local name, `f = lambda ...: ...` - - **Added in v0.15.0**: Named expressions (a.k.a. walrus operator, Python 3.8+), ``f := lambda ...: ...`` + - **Added in v0.15.0**: Named expressions (a.k.a. walrus operator, Python 3.8+), `f := lambda ...: ...` - - Expression-assignment to an unpythonic environment, ``f << (lambda ...: ...)`` + - Expression-assignment to an unpythonic environment, `f << (lambda ...: ...)` - Env-assignments are processed lexically, just like regular assignments. This should not cause problems, because left-shifting by a literal lambda most often makes no sense (whence, that syntax is *almost* guaranteed to mean an env-assignment). - - Let bindings, ``let[[f << (lambda ...: ...)] in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). + - Let bindings, `let[[f << (lambda ...: ...)] in ...]`, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). - - **Added in v0.14.2**: Named argument in a function call, as in ``foo(f=lambda ...: ...)``. + - **Added in v0.14.2**: Named argument in a function call, as in `foo(f=lambda ...: ...)`. - - **Added in v0.14.2**: In a dictionary literal ``{...}``, an item with a literal string key, as in ``{"f": lambda ...: ...}``. + - **Added in v0.14.2**: In a dictionary literal `{...}`, an item with a literal string key, as in `{"f": lambda ...: ...}`. Support for other forms of assignment may or may not be added in a future version. We will maintain a list here; but if you want the gritty details, see the `_namedlambda` syntax transformer in [`unpythonic.syntax.lambdatools`](../unpythonic/syntax/lambdatools.py). -### ``fn``: underscore notation (quick lambdas) for Python +### `fn`: underscore notation (quick lambdas) for Python **Changed in v0.15.0.** *Up to 0.14.x, the `f[]` macro used to be provided by `macropy`, but now that we use `mcpyrate`, we provide this ourselves. Note that the name of the construct is now `fn[]`.* -The syntax ``fn[...]`` creates a lambda, where each underscore `_` in the ``...`` part introduces a new parameter: +The syntax `fn[...]` creates a lambda, where each underscore `_` in the `...` part introduces a new parameter: ```python from unpythonic.syntax import macros, fn @@ -721,7 +721,7 @@ double = fn[_ * 2] # --> double = lambda x: x * 2 mul = fn[_ * _] # --> mul = lambda x, y: x * y ``` -The macro does not descend into any nested ``fn[]``, to allow the macro expander itself to expand those separately. +The macro does not descend into any nested `fn[]`, to allow the macro expander itself to expand those separately. We have named the construct `fn`, because `f` is often used as a function name in code examples, local temporaries, and similar. Also, `fn[]` is a less ambiguous abbreviation for a syntactic construct that means *function*, while remaining shorter than the equivalent `lambda`. @@ -735,13 +735,13 @@ Because in `mcpyrate`, macros can be as-imported, you can rename `fn` at import It is sufficient that `fn` has been macro-imported by the time when the `with quicklambda` expands. So it is possible, for example, for a dialect template to macro-import just `quicklambda` and inject an invocation for it, and leave macro-importing `fn` to the user code. The `Lispy` variant of the [Lispython dialect](dialects/lispython.md) does exactly this. -### ``quicklambda``: expand quick lambdas first +### `quicklambda`: expand quick lambdas first -To be able to transform correctly, the block macros in ``unpythonic.syntax`` that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all ``lambda`` definitions written with Python's standard ``lambda``. +To be able to transform correctly, the block macros in `unpythonic.syntax` that transform lambdas (e.g. `multilambda`, `tco`) need to see all `lambda` definitions written with Python's standard `lambda`. -However, the ``fn`` macro uses the syntax ``fn[...]``, which (to the analyzer) does not look like a lambda definition. The `quicklambda` block macro changes the expansion order, forcing any ``fn[...]`` lexically inside the block to expand before any other macros do. +However, the `fn` macro uses the syntax `fn[...]`, which (to the analyzer) does not look like a lambda definition. The `quicklambda` block macro changes the expansion order, forcing any `fn[...]` lexically inside the block to expand before any other macros do. -Any expression of the form ``fn[...]``, where ``fn`` is any name bound in the current macro expander to the macro `unpythonic.syntax.fn`, is understood as a quick lambda. (In plain English, this respects as-imports of the macro ``fn``.) +Any expression of the form `fn[...]`, where `fn` is any name bound in the current macro expander to the macro `unpythonic.syntax.fn`, is understood as a quick lambda. (In plain English, this respects as-imports of the macro `fn`.) Example - a quick multilambda: @@ -756,7 +756,7 @@ with quicklambda, multilambda: assert func(1, 2) == 3 ``` -This is of course rather silly, as an unnamed formal parameter can only be mentioned once. If we are giving names to them, a regular ``lambda`` is shorter to write. A more realistic combo is: +This is of course rather silly, as an unnamed formal parameter can only be mentioned once. If we are giving names to them, a regular `lambda` is shorter to write. A more realistic combo is: ```python with quicklambda, tco: @@ -770,9 +770,9 @@ with quicklambda, tco: ``` -### ``envify``: make formal parameters live in an unpythonic ``env`` +### `envify`: make formal parameters live in an unpythonic `env` -When a function whose definition (``def`` or ``lambda``) is lexically inside a ``with envify`` block is entered, it copies references to its arguments into an unpythonic ``env``. At macro expansion time, all references to the formal parameters are redirected to that environment. This allows rebinding, from an expression position, names that were originally the formal parameters. +When a function whose definition (`def` or `lambda`) is lexically inside a `with envify` block is entered, it copies references to its arguments into an unpythonic `env`. At macro expansion time, all references to the formal parameters are redirected to that environment. This allows rebinding, from an expression position, names that were originally the formal parameters. Wherever could *that* be useful? For an illustrative caricature, consider [PG's accumulator puzzle](http://paulgraham.com/icad.html). @@ -787,11 +787,11 @@ def foo(n): return accumulate ``` -This avoids allocating an extra place to store the accumulator ``n``. If you want optimal bytecode, this is the best solution in Python 3. +This avoids allocating an extra place to store the accumulator `n`. If you want optimal bytecode, this is the best solution in Python 3. -But what if, instead, we consider the readability of the unexpanded source code? The definition of ``accumulate`` requires many lines for something that simple. What if we wanted to make it a lambda? Because all forms of assignment are statements in Python, the above solution is not admissible for a lambda, even with macros. +But what if, instead, we consider the readability of the unexpanded source code? The definition of `accumulate` requires many lines for something that simple. What if we wanted to make it a lambda? Because all forms of assignment are statements in Python, the above solution is not admissible for a lambda, even with macros. -So if we want to use a lambda, we have to create an ``env``, so that we can write into it. Let's use the let-over-lambda idiom: +So if we want to use a lambda, we have to create an `env`, so that we can write into it. Let's use the let-over-lambda idiom: ```python def foo(n0): @@ -799,9 +799,9 @@ def foo(n0): (lambda i: n << n + i)] ``` -Already better, but the ``let`` is used only for (in effect) altering the passed-in value of ``n0``; we don't place any other variables into the ``let`` environment. Considering the source text already introduces an ``n0`` which is just used to initialize ``n``, that's an extra element that could be eliminated. +Already better, but the `let` is used only for (in effect) altering the passed-in value of `n0`; we don't place any other variables into the `let` environment. Considering the source text already introduces an `n0` which is just used to initialize `n`, that's an extra element that could be eliminated. -Enter the ``envify`` macro, which automates this: +Enter the `envify` macro, which automates this: ```python with envify: @@ -809,7 +809,7 @@ with envify: return lambda i: n << n + i ``` -Combining with ``autoreturn`` yields the fewest-elements optimal solution to the accumulator puzzle: +Combining with `autoreturn` yields the fewest-elements optimal solution to the accumulator puzzle: ```python with autoreturn, envify: @@ -817,13 +817,13 @@ with autoreturn, envify: lambda i: n << n + i ``` -The ``with`` block adds a few elements, but if desired, it can be refactored into the definition of a custom dialect in [Pydialect](https://github.com/Technologicat/pydialect). +The `with` block adds a few elements, but if desired, it can be refactored into the definition of a custom dialect in [Pydialect](https://github.com/Technologicat/pydialect). ## Language features To boldly go where Python without macros just won't. Changing the rules by code-walking and making significant rewrites. -### ``autocurry``: automatic currying for Python +### `autocurry`: automatic currying for Python **Changed in v0.15.0.** *The macro is now named `autocurry`, to avoid shadowing the `curry` function.* @@ -847,21 +847,21 @@ with autocurry: assert add3(1)(2)(3) == 6 ``` -*Lexically* inside a ``with autocurry`` block: +*Lexically* inside a `with autocurry` block: - - All **function calls** and **function definitions** (``def``, ``lambda``) are automatically curried, somewhat like in Haskell, or in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy). + - All **function calls** and **function definitions** (`def`, `lambda`) are automatically curried, somewhat like in Haskell, or in `#lang` [`spicy`](https://github.com/Technologicat/spicy). - - Function calls are autocurried, and run ``unpythonic.fun.curry`` in a special mode that no-ops on uninspectable functions (triggering a standard function call with the given args immediately) instead of raising ``TypeError`` as usual. + - Function calls are autocurried, and run `unpythonic.fun.curry` in a special mode that no-ops on uninspectable functions (triggering a standard function call with the given args immediately) instead of raising `TypeError` as usual. -**CAUTION**: Some built-ins are uninspectable or may report their arities incorrectly; in those cases, ``curry`` may fail, occasionally in mysterious ways. The function ``unpythonic.arity.arities``, which ``unpythonic.fun.curry`` internally uses, has a workaround for the inspectability problems of all built-ins in the top-level namespace (as of Python 3.7), but e.g. methods of built-in types are not handled. +**CAUTION**: Some built-ins are uninspectable or may report their arities incorrectly; in those cases, `curry` may fail, occasionally in mysterious ways. The function `unpythonic.arity.arities`, which `unpythonic.fun.curry` internally uses, has a workaround for the inspectability problems of all built-ins in the top-level namespace (as of Python 3.7), but e.g. methods of built-in types are not handled. Manual uses of the `curry` decorator (on both `def` and `lambda`) are detected, and in such cases the macro skips adding the decorator. -### ``lazify``: call-by-need for Python +### `lazify`: call-by-need for Python **Changed in v0.15.0.** *Up to 0.14.x, the `lazy[]` macro, that is used together with `with lazify`, used to be provided by `macropy`, but now that we use `mcpyrate`, we provide it ourselves. If you use `lazy[]`, change your import of that macro to `from unpythonic.syntax import macros, lazy`*. -Also known as *lazy functions*. Like [lazy/racket](https://docs.racket-lang.org/lazy/index.html), but for Python. Note if you want *lazy sequences* instead, Python already provides those; just use the generator facility (and decorate your gfunc with ``unpythonic.gmemoize`` if needed). +Also known as *lazy functions*. Like [lazy/racket](https://docs.racket-lang.org/lazy/index.html), but for Python. Note if you want *lazy sequences* instead, Python already provides those; just use the generator facility (and decorate your gfunc with `unpythonic.gmemoize` if needed). Lazy function example: @@ -882,11 +882,11 @@ with lazify: assert f(21, 1/0) == 42 ``` -In a ``with lazify`` block, function arguments are evaluated only when actually used, at most once each, and in the order in which they are actually used (regardless of the ordering of the formal parameters that receive them). Delayed values (*promises*) are automatically evaluated (*forced*) on access. Automatic lazification applies to arguments in function calls and to let-bindings, since they play a similar role. **No other binding forms are auto-lazified.** +In a `with lazify` block, function arguments are evaluated only when actually used, at most once each, and in the order in which they are actually used (regardless of the ordering of the formal parameters that receive them). Delayed values (*promises*) are automatically evaluated (*forced*) on access. Automatic lazification applies to arguments in function calls and to let-bindings, since they play a similar role. **No other binding forms are auto-lazified.** -Automatic lazification uses the ``lazyrec[]`` macro (see below), which recurses into certain types of container literals, so that the lazification will not interfere with unpacking. +Automatic lazification uses the `lazyrec[]` macro (see below), which recurses into certain types of container literals, so that the lazification will not interfere with unpacking. -Note ``my_if`` in the example is a regular function, not a macro. Only the ``with lazify`` is imbued with any magic. Essentially, the above code expands into: +Note `my_if` in the example is a regular function, not a macro. Only the `with lazify` is imbued with any magic. Essentially, the above code expands into: ```python from unpythonic.syntax import macros, lazy @@ -907,47 +907,47 @@ def f(a, b): assert f(lazy[21], lazy[1/0]) == 42 ``` -plus some clerical details to allow mixing lazy and strict code. This second example relies on the magic of closures to capture f's ``a`` and ``b`` into the ``lazy[]`` promises. +plus some clerical details to allow mixing lazy and strict code. This second example relies on the magic of closures to capture f's `a` and `b` into the `lazy[]` promises. -Like ``with continuations``, no state or context is associated with a ``with lazify`` block, so lazy functions defined in one block may call those defined in another. +Like `with continuations`, no state or context is associated with a `with lazify` block, so lazy functions defined in one block may call those defined in another. Lazy code is allowed to call strict functions and vice versa, without requiring any additional effort. -Comboing with other block macros in ``unpythonic.syntax`` is supported, including ``autocurry`` and ``continuations``. See the [meta](#meta) section of this README for the correct ordering. +Comboing with other block macros in `unpythonic.syntax` is supported, including `autocurry` and `continuations`. See the [meta](#meta) section of this README for the correct ordering. -For more details, see the docstring of ``unpythonic.syntax.lazify``. +For more details, see the docstring of `unpythonic.syntax.lazify`. -Inspired by Haskell, Racket's ``(delay)`` and ``(force)``, and [lazy/racket](https://docs.racket-lang.org/lazy/index.html). +Inspired by Haskell, Racket's `(delay)` and `(force)`, and [lazy/racket](https://docs.racket-lang.org/lazy/index.html). -**CAUTION**: The functions in ``unpythonic.fun`` are lazify-aware (so that e.g. ``curry`` and ``compose`` work with lazy functions), as are ``call`` and ``callwith`` in ``unpythonic.misc``, but a large part of ``unpythonic`` is not. Keep in mind that any call to a strict (regular Python) function will evaluate all of its arguments. +**CAUTION**: The functions in `unpythonic.fun` are lazify-aware (so that e.g. `curry` and `compose` work with lazy functions), as are `call` and `callwith` in `unpythonic.misc`, but a large part of `unpythonic` is not. Keep in mind that any call to a strict (regular Python) function will evaluate all of its arguments. -#### ``lazy[]`` and ``lazyrec[]`` macros +#### `lazy[]` and `lazyrec[]` macros **Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. Note that a lazy value now no longer has a `__call__` operator; instead, it has a `force()` method. The utility `unpythonic.lazyutil.force` (previously exported in `unpythonic.syntax`; now moved to the top-level namespace of `unpythonic`) abstracts away this detail.* -We provide the macros ``unpythonic.syntax.lazy``, which explicitly lazifies a single expression, and ``unpythonic.syntax.lazyrec``, which can be used to lazify expressions inside container literals, recursively. +We provide the macros `unpythonic.syntax.lazy`, which explicitly lazifies a single expression, and `unpythonic.syntax.lazyrec`, which can be used to lazify expressions inside container literals, recursively. -Essentially, ``lazy[...]`` achieves the same result as ``memoize(lambda: ...)``, with the practical difference that a ``lazy[]`` promise ``p`` is evaluated by calling ``unpythonic.lazyutil.force(p)`` or ``p.force()``. In ``unpythonic``, the promise datatype (``unpythonic.lazyutil.Lazy``) does not have a ``__call__`` method, because the word ``force`` better conveys the intent. +Essentially, `lazy[...]` achieves the same result as `memoize(lambda: ...)`, with the practical difference that a `lazy[]` promise `p` is evaluated by calling `unpythonic.lazyutil.force(p)` or `p.force()`. In `unpythonic`, the promise datatype (`unpythonic.lazyutil.Lazy`) does not have a `__call__` method, because the word `force` better conveys the intent. -It is preferable to use the ``force`` function instead of the ``.force`` method, because the function will also pass through any non-promise value, whereas (obviously) a non-promise value will not have a ``.force`` method. Using the function, you can ``force`` a value just to be sure, without caring whether that value was a promise. The ``force`` function is available in the top-level namespace of ``unpythonic``. +It is preferable to use the `force` function instead of the `.force` method, because the function will also pass through any non-promise value, whereas (obviously) a non-promise value will not have a `.force` method. Using the function, you can `force` a value just to be sure, without caring whether that value was a promise. The `force` function is available in the top-level namespace of `unpythonic`. -The ``lazify`` subsystem expects the ``lazy[...]`` notation in its analyzer, and will not recognize ``memoize(lambda: ...)`` as a delayed value. +The `lazify` subsystem expects the `lazy[...]` notation in its analyzer, and will not recognize `memoize(lambda: ...)` as a delayed value. -The ``lazyrec[]`` macro allows code like ``tpl = lazyrec[(1*2*3, 4*5*6)]``. Each item becomes wrapped with ``lazy[]``, but the container itself is left alone, to avoid interfering with unpacking. Because ``lazyrec[]`` is a macro and must work by names only, it supports a fixed set of container types: ``list``, ``tuple``, ``set``, ``dict``, ``frozenset``, ``unpythonic.collections.frozendict``, ``unpythonic.collections.box``, and ``unpythonic.llist.cons`` (specifically, the constructors ``cons``, ``ll`` and ``llist``). +The `lazyrec[]` macro allows code like `tpl = lazyrec[(1*2*3, 4*5*6)]`. Each item becomes wrapped with `lazy[]`, but the container itself is left alone, to avoid interfering with unpacking. Because `lazyrec[]` is a macro and must work by names only, it supports a fixed set of container types: `list`, `tuple`, `set`, `dict`, `frozenset`, `unpythonic.collections.frozendict`, `unpythonic.collections.box`, and `unpythonic.llist.cons` (specifically, the constructors `cons`, `ll` and `llist`). -The `unpythonic` containers **must be from-imported** for ``lazyrec[]`` to recognize them. Either use ``from unpythonic import xxx`` (**recommended**), where ``xxx`` is a container type, or import the ``containers`` subpackage by ``from unpythonic import containers``, and then use ``containers.xxx``. (The analyzer only looks inside at most one level of attributes. This may change in the future.) +The `unpythonic` containers **must be from-imported** for `lazyrec[]` to recognize them. Either use `from unpythonic import xxx` (**recommended**), where `xxx` is a container type, or import the `containers` subpackage by `from unpythonic import containers`, and then use `containers.xxx`. (The analyzer only looks inside at most one level of attributes. This may change in the future.) -(The analysis in ``lazyrec[]`` must work by names only, because in an eager language any lazification must be performed as a syntax transformation before the code actually runs, so the analysis must be performed statically - and locally, because ``lazyrec[]`` is an expr macro. [Fexprs](https://fexpr.blogspot.com/2011/04/fexpr.html) (along with [a new calculus to go with them](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html)) are the clean, elegant solution, but this requires redesigning the whole language from ground up. Of course, if you're fine with a language not particularly designed for extensibility, and lazy evaluation is your top requirement, you could just use Haskell.) +(The analysis in `lazyrec[]` must work by names only, because in an eager language any lazification must be performed as a syntax transformation before the code actually runs, so the analysis must be performed statically - and locally, because `lazyrec[]` is an expr macro. [Fexprs](https://fexpr.blogspot.com/2011/04/fexpr.html) (along with [a new calculus to go with them](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html)) are the clean, elegant solution, but this requires redesigning the whole language from ground up. Of course, if you're fine with a language not particularly designed for extensibility, and lazy evaluation is your top requirement, you could just use Haskell.) #### Forcing promises manually **Changed in v0.15.0.** *The functions `force1` and `force` now live in the top-level namespace of `unpythonic`, no longer in `unpythonic.syntax`.* -This is mainly useful if you ``lazy[]`` or ``lazyrec[]`` something explicitly, and want to compute its value outside a ``with lazify`` block. +This is mainly useful if you `lazy[]` or `lazyrec[]` something explicitly, and want to compute its value outside a `with lazify` block. -We provide the functions ``force1`` and ``force``. Using ``force1``, if ``x`` is a ``lazy[]`` promise, it will be forced, and the resulting value is returned. If ``x`` is not a promise, ``x`` itself is returned, à la Racket. The function ``force``, in addition, descends into containers (recursively). When an atom ``x`` (i.e. anything that is not a container) is encountered, it is processed using ``force1``. +We provide the functions `force1` and `force`. Using `force1`, if `x` is a `lazy[]` promise, it will be forced, and the resulting value is returned. If `x` is not a promise, `x` itself is returned, à la Racket. The function `force`, in addition, descends into containers (recursively). When an atom `x` (i.e. anything that is not a container) is encountered, it is processed using `force1`. -Mutable containers are updated in-place; for immutables, a new instance is created, but as a side effect the promise objects **in the input container** will be forced. Any container with a compatible ``collections.abc`` is supported. (See ``unpythonic.collections.mogrify`` for details.) In addition, as special cases ``unpythonic.collections.box`` and ``unpythonic.llist.cons`` are supported. +Mutable containers are updated in-place; for immutables, a new instance is created, but as a side effect the promise objects **in the input container** will be forced. Any container with a compatible `collections.abc` is supported. (See `unpythonic.collections.mogrify` for details.) In addition, as special cases `unpythonic.collections.box` and `unpythonic.llist.cons` are supported. #### Binding constructs and auto-lazification @@ -959,7 +959,7 @@ a = 2*a print(a) # 20, right? ``` -If we chose to auto-lazify assignments, then assuming a ``with lazify`` around the example, it would expand to: +If we chose to auto-lazify assignments, then assuming a `with lazify` around the example, it would expand to: ```python from unpythonic.syntax import macros, lazy @@ -970,9 +970,9 @@ a = lazy[2*force(a)] print(force(a)) ``` -In the second assignment, the ``lazy[]`` sets up a promise, which will force ``a`` *at the time when the containing promise is forced*, but at that time the name ``a`` points to a promise, which will force... +In the second assignment, the `lazy[]` sets up a promise, which will force `a` *at the time when the containing promise is forced*, but at that time the name `a` points to a promise, which will force... -The fundamental issue is that ``a = 2*a`` is an imperative update. Therefore, to avoid this infinite loop trap for the unwary, assignments are not auto-lazified. Note that if we use two different names, this works just fine: +The fundamental issue is that `a = 2*a` is an imperative update. Therefore, to avoid this infinite loop trap for the unwary, assignments are not auto-lazified. Note that if we use two different names, this works just fine: ```python from unpythonic.syntax import macros, lazy @@ -983,21 +983,21 @@ b = lazy[2*force(a)] print(force(b)) ``` -because now at the time when ``b`` is forced, the name ``a`` still points to the value we intended it to. +because now at the time when `b` is forced, the name `a` still points to the value we intended it to. -If you're sure you have *new definitions* and not *imperative updates*, just manually use ``lazy[]`` (or ``lazyrec[]``, as appropriate) on the RHS. Or if it's fine to use eager evaluation, just omit the ``lazy[]``, thus allowing Python to evaluate the RHS immediately. +If you're sure you have *new definitions* and not *imperative updates*, just manually use `lazy[]` (or `lazyrec[]`, as appropriate) on the RHS. Or if it's fine to use eager evaluation, just omit the `lazy[]`, thus allowing Python to evaluate the RHS immediately. -Beside function calls (which bind the parameters of the callee to the argument values of the call) and assignments, there are many other binding constructs in Python. For a full list, see [here](http://excess.org/article/2014/04/bar-foo/), or locally [here](../unpythonic/syntax/scopeanalyzer.py), in function ``get_names_in_store_context``. Particularly noteworthy in the context of lazification are the ``for`` loop and the ``with`` context manager. +Beside function calls (which bind the parameters of the callee to the argument values of the call) and assignments, there are many other binding constructs in Python. For a full list, see [here](http://excess.org/article/2014/04/bar-foo/), or locally [here](../unpythonic/syntax/scopeanalyzer.py), in function `get_names_in_store_context`. Particularly noteworthy in the context of lazification are the `for` loop and the `with` context manager. -In Python's ``for``, the loop counter is an imperatively updated single name. In many use cases a rapid update is desirable for performance reasons, and in any case, the whole point of the loop is (almost always) to read the counter (and do something with the value) at least once per iteration. So it is much simpler, faster, and equally correct not to lazify there. +In Python's `for`, the loop counter is an imperatively updated single name. In many use cases a rapid update is desirable for performance reasons, and in any case, the whole point of the loop is (almost always) to read the counter (and do something with the value) at least once per iteration. So it is much simpler, faster, and equally correct not to lazify there. -In ``with``, the whole point of a context manager is that it is eagerly initialized when the ``with`` block is entered (and finalized when the block exits). Since our lazy code can transparently use both bare values and promises (due to the semantics of our ``force1``), and the context manager would have to be eagerly initialized anyway, we can choose not to lazify there. +In `with`, the whole point of a context manager is that it is eagerly initialized when the `with` block is entered (and finalized when the block exits). Since our lazy code can transparently use both bare values and promises (due to the semantics of our `force1`), and the context manager would have to be eagerly initialized anyway, we can choose not to lazify there. #### Note about TCO -To borrow a term from PG's On Lisp, to make ``lazify`` *pay-as-you-go*, a special mode in ``unpythonic.tco.trampolined`` is automatically enabled by ``with lazify`` to build lazify-aware trampolines in order to avoid a drastic performance hit (~10x) in trampolines built for regular strict code. +To borrow a term from PG's On Lisp, to make `lazify` *pay-as-you-go*, a special mode in `unpythonic.tco.trampolined` is automatically enabled by `with lazify` to build lazify-aware trampolines in order to avoid a drastic performance hit (~10x) in trampolines built for regular strict code. -The idea is that the mode is enabled while any function definitions in the ``with lazify`` block run, so they get a lazify-aware trampoline when the ``trampolined`` decorator is applied. This should be determined lexically, but that's complicated to do API-wise, so we currently enable the mode for the dynamic extent of the ``with lazify``. Usually this is close enough; the main case where this can behave unexpectedly is: +The idea is that the mode is enabled while any function definitions in the `with lazify` block run, so they get a lazify-aware trampoline when the `trampolined` decorator is applied. This should be determined lexically, but that's complicated to do API-wise, so we currently enable the mode for the dynamic extent of the `with lazify`. Usually this is close enough; the main case where this can behave unexpectedly is: ```python @trampolined # strict trampoline @@ -1024,12 +1024,12 @@ TCO chains with an arbitrary mix of lazy and strict functions should work as lon Tail-calling from a strict function into a lazy function should work, because all arguments are evaluated at the strict side before the call is made. -But tail-calling ``strict -> lazy -> strict`` will fail in some cases. The second strict callee may get promises instead of values, because the strict trampoline does not have the ``maybe_force_args`` (the mechanism ``with lazify`` uses to force the args when lazy code calls into strict code). +But tail-calling `strict -> lazy -> strict` will fail in some cases. The second strict callee may get promises instead of values, because the strict trampoline does not have the `maybe_force_args` (the mechanism `with lazify` uses to force the args when lazy code calls into strict code). -The reason we have this hack is that it allows the performance of strict code using unpythonic's TCO machinery, not even caring that a ``lazify`` exists, to be unaffected by the additional machinery used to support automatic lazy-strict interaction. +The reason we have this hack is that it allows the performance of strict code using unpythonic's TCO machinery, not even caring that a `lazify` exists, to be unaffected by the additional machinery used to support automatic lazy-strict interaction. -### ``tco``: automatic tail call optimization for Python +### `tco`: automatic tail call optimization for Python ```python from unpythonic.syntax import macros, tco @@ -1051,42 +1051,42 @@ with tco: assert evenp(10000) is True ``` -All function definitions (``def`` and ``lambda``) lexically inside the block undergo TCO transformation. The functions are automatically ``@trampolined``, and any tail calls in their return values are converted to ``jump(...)`` for the TCO machinery. Here *return value* is defined as: +All function definitions (`def` and `lambda`) lexically inside the block undergo TCO transformation. The functions are automatically `@trampolined`, and any tail calls in their return values are converted to `jump(...)` for the TCO machinery. Here *return value* is defined as: - - In a ``def``, the argument expression of ``return``, or of a call to a known escape continuation. + - In a `def`, the argument expression of `return`, or of a call to a known escape continuation. - - In a ``lambda``, the whole body, as well as the argument expression of a call to a known escape continuation. + - In a `lambda`, the whole body, as well as the argument expression of a call to a known escape continuation. -What is a *known escape continuation* is explained below, in the section [TCO and ``call_ec``](#tco-and-call_ec). +What is a *known escape continuation* is explained below, in the section [TCO and `call_ec`](#tco-and-call_ec). -To find the tail position inside a compound return value, this recursively handles any combination of ``a if p else b``, ``and``, ``or``; and from ``unpythonic.syntax``, ``do[]``, ``let[]``, ``letseq[]``, ``letrec[]``. Support for ``do[]`` includes also any ``multilambda`` blocks that have already expanded when ``tco`` is processed. The macros ``aif[]`` and ``cond[]`` are also supported, because they expand into a combination of ``let[]``, ``do[]``, and ``a if p else b``. +To find the tail position inside a compound return value, this recursively handles any combination of `a if p else b`, `and`, `or`; and from `unpythonic.syntax`, `do[]`, `let[]`, `letseq[]`, `letrec[]`. Support for `do[]` includes also any `multilambda` blocks that have already expanded when `tco` is processed. The macros `aif[]` and `cond[]` are also supported, because they expand into a combination of `let[]`, `do[]`, and `a if p else b`. -**CAUTION**: In an ``and``/``or`` expression, only the last item of the whole expression is in tail position. This is because in general, it is impossible to know beforehand how many of the items will be evaluated. +**CAUTION**: In an `and`/`or` expression, only the last item of the whole expression is in tail position. This is because in general, it is impossible to know beforehand how many of the items will be evaluated. -**CAUTION**: In a ``def`` you still need the ``return``; it marks a return value. If you want the tail position to imply a ``return``, use the combo ``with autoreturn, tco`` (on ``autoreturn``, see below). +**CAUTION**: In a `def` you still need the `return`; it marks a return value. If you want the tail position to imply a `return`, use the combo `with autoreturn, tco` (on `autoreturn`, see below). -TCO is based on a strategy similar to MacroPy's ``tco`` macro, but using unpythonic's TCO machinery, and working together with the macros introduced by ``unpythonic.syntax``. The semantics are slightly different; by design, ``unpythonic`` requires an explicit ``return`` to mark tail calls in a ``def``. A call that is strictly speaking in tail position, but lacks the ``return``, is not TCO'd, and Python's implicit ``return None`` then shuts down the trampoline, returning ``None`` as the result of the TCO chain. +TCO is based on a strategy similar to MacroPy's `tco` macro, but using unpythonic's TCO machinery, and working together with the macros introduced by `unpythonic.syntax`. The semantics are slightly different; by design, `unpythonic` requires an explicit `return` to mark tail calls in a `def`. A call that is strictly speaking in tail position, but lacks the `return`, is not TCO'd, and Python's implicit `return None` then shuts down the trampoline, returning `None` as the result of the TCO chain. #### TCO and continuations -The ``tco`` macro detects and skips any ``with continuations`` blocks inside the ``with tco`` block, because ``continuations`` already implies TCO. This is done **for the specific reason** of allowing the [Lispython dialect](https://github.com/Technologicat/pydialect) to use ``with continuations``, because the dialect itself implies a ``with tco`` for the whole module (so the user code has no way to exit the TCO context). +The `tco` macro detects and skips any `with continuations` blocks inside the `with tco` block, because `continuations` already implies TCO. This is done **for the specific reason** of allowing the [Lispython dialect](https://github.com/Technologicat/pydialect) to use `with continuations`, because the dialect itself implies a `with tco` for the whole module (so the user code has no way to exit the TCO context). -The ``tco`` and ``continuations`` macros actually share a lot of the code that implements TCO; ``continuations`` just hooks into some callbacks to perform additional processing. +The `tco` and `continuations` macros actually share a lot of the code that implements TCO; `continuations` just hooks into some callbacks to perform additional processing. -#### TCO and ``call_ec`` +#### TCO and `call_ec` -(Mainly of interest for lambdas, which have no ``return``, and for "multi-return" from a nested function.) +(Mainly of interest for lambdas, which have no `return`, and for "multi-return" from a nested function.) It is important to recognize a call to an escape continuation as such, because the argument given to an escape continuation is essentially a return value. If this argument is itself a call, it needs the TCO transformation to be applied to it. -For escape continuations in ``tco`` and ``continuations`` blocks, only basic uses of ``call_ec`` are supported, for automatically harvesting names referring to an escape continuation. In addition, the literal function names ``ec``, ``brk`` and ``throw`` are always *understood as referring to* an escape continuation. +For escape continuations in `tco` and `continuations` blocks, only basic uses of `call_ec` are supported, for automatically harvesting names referring to an escape continuation. In addition, the literal function names `ec`, `brk` and `throw` are always *understood as referring to* an escape continuation. -The name ``ec``, ``brk`` or ``throw`` alone is not sufficient to make a function into an escape continuation, even though ``tco`` (and ``continuations``) will think of it as such. The function also needs to actually implement some kind of an escape mechanism. An easy way to get an escape continuation, where this has already been done for you, is to use ``call_ec``. Another such mechanism is the ``catch``/``throw`` pair. +The name `ec`, `brk` or `throw` alone is not sufficient to make a function into an escape continuation, even though `tco` (and `continuations`) will think of it as such. The function also needs to actually implement some kind of an escape mechanism. An easy way to get an escape continuation, where this has already been done for you, is to use `call_ec`. Another such mechanism is the `catch`/`throw` pair. -See the docstring of ``unpythonic.syntax.tco`` for details. +See the docstring of `unpythonic.syntax.tco` for details. -### ``continuations``: call/cc for Python +### `continuations`: call/cc for Python *Where control flow is your playground.* @@ -1117,19 +1117,19 @@ If you are new to continuations, see the [short and easy Python-based explanatio We essentially provide a very loose pythonification of Paul Graham's continuation-passing macros, chapter 20 in [On Lisp](http://paulgraham.com/onlisp.html). -The approach differs from native continuation support (such as in Scheme or Racket) in that the continuation is captured only where explicitly requested with ``call_cc[]``. This lets most of the code work as usual, while performing the continuation magic where explicitly desired. +The approach differs from native continuation support (such as in Scheme or Racket) in that the continuation is captured only where explicitly requested with `call_cc[]`. This lets most of the code work as usual, while performing the continuation magic where explicitly desired. -As a consequence of the approach, our continuations are [*delimited*](https://en.wikipedia.org/wiki/Delimited_continuation) in the very crude sense that the captured continuation ends at the end of the body where the *currently dynamically outermost* ``call_cc[]`` was used. Notably, in `unpythonic`, a continuation eventually terminates and returns a value, without hijacking the rest of the whole-program execution. +As a consequence of the approach, our continuations are [*delimited*](https://en.wikipedia.org/wiki/Delimited_continuation) in the very crude sense that the captured continuation ends at the end of the body where the *currently dynamically outermost* `call_cc[]` was used. Notably, in `unpythonic`, a continuation eventually terminates and returns a value, without hijacking the rest of the whole-program execution. -Hence, if porting some code that uses ``call/cc`` from Racket to Python, in the Python version the ``call_cc[]`` may be need to be placed further out to capture the relevant part of the computation. For example, see ``amb`` in the demonstration below; a Scheme or Racket equivalent usually has the ``call/cc`` placed inside the ``amb`` operator itself, whereas in Python we must place the ``call_cc[]`` at the call site of ``amb``. +Hence, if porting some code that uses `call/cc` from Racket to Python, in the Python version the `call_cc[]` may be need to be placed further out to capture the relevant part of the computation. For example, see `amb` in the demonstration below; a Scheme or Racket equivalent usually has the `call/cc` placed inside the `amb` operator itself, whereas in Python we must place the `call_cc[]` at the call site of `amb`. Observe that while our outermost `call_cc` already somewhat acts like a prompt (in the sense of delimited continuations), we are currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and terminate the capture there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. For various possible program topologies that continuations may introduce, see [these clarifying pictures](callcc_topology.pdf). -For full documentation, see the docstring of ``unpythonic.syntax.continuations``. The unit tests [[1]](../unpythonic/syntax/test/test_conts.py) [[2]](../unpythonic/syntax/test/test_conts_escape.py) [[3]](../unpythonic/syntax/test/test_conts_gen.py) [[4]](../unpythonic/syntax/test/test_conts_topo.py) may also be useful as usage examples. +For full documentation, see the docstring of `unpythonic.syntax.continuations`. The unit tests [[1]](../unpythonic/syntax/test/test_conts.py) [[2]](../unpythonic/syntax/test/test_conts_escape.py) [[3]](../unpythonic/syntax/test/test_conts_gen.py) [[4]](../unpythonic/syntax/test/test_conts_topo.py) may also be useful as usage examples. -**Note on debugging**: If a function containing a ``call_cc[]`` crashes below the ``call_cc[]``, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so you can pinpoint the source code line where the error occurred. (For a function ``f``, it is named ``f_cont_``) But be aware that especially in complex macro combos (e.g. ``continuations, curry, lazify``), the other block macros may spit out many internal function calls *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. +**Note on debugging**: If a function containing a `call_cc[]` crashes below the `call_cc[]`, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so you can pinpoint the source code line where the error occurred. (For a function `f`, it is named `f_cont_`) But be aware that especially in complex macro combos (e.g. `continuations, curry, lazify`), the other block macros may spit out many internal function calls *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. **Note on exceptions**: Raising an exception, or [signaling and restarting](features.md#handlers-restarts-conditions-and-restarts), will partly unwind the call stack, so the continuation *from the level that raised the exception* will be cancelled. This is arguably exactly the expected behavior. @@ -1184,88 +1184,88 @@ with continuations: print(fail()) print(fail()) ``` -Code within a ``with continuations`` block is treated specially. +Code within a `with continuations` block is treated specially.

Roughly: -> - Each function definition (``def`` or ``lambda``) in a ``with continuations`` block has an implicit formal parameter ``cc``, **even if not explicitly declared** in the formal parameter list. -> - The continuation machinery will set the default value of ``cc`` to the default continuation (``identity``), which just returns its arguments. -> - The default value allows these functions to be called also normally without passing a ``cc``. In effect, the function will then return normally. -> - If ``cc`` is not declared explicitly, it is implicitly declared as a by-name-only parameter named ``cc``, and the default value is set automatically. -> - If ``cc`` is declared explicitly, the default value is set automatically if ``cc`` is in a position that can accept a default value, and no default has been set by the user. +> - Each function definition (`def` or `lambda`) in a `with continuations` block has an implicit formal parameter `cc`, **even if not explicitly declared** in the formal parameter list. +> - The continuation machinery will set the default value of `cc` to the default continuation (`identity`), which just returns its arguments. +> - The default value allows these functions to be called also normally without passing a `cc`. In effect, the function will then return normally. +> - If `cc` is not declared explicitly, it is implicitly declared as a by-name-only parameter named `cc`, and the default value is set automatically. +> - If `cc` is declared explicitly, the default value is set automatically if `cc` is in a position that can accept a default value, and no default has been set by the user. > - Positions that can accept a default value are the last positional parameter that has no default, and a by-name-only parameter in any syntactically allowed position. -> - Having a hidden parameter is somewhat magic, but overall improves readability, as this allows declaring ``cc`` only where actually explicitly needed. -> - **CAUTION**: Usability trap: in nested function definitions, each ``def`` and ``lambda`` comes with **its own** implicit ``cc``. -> - In the above ``amb`` example, the local variable is named ``ourcc``, so that the continuation passed in from outside (into the ``lambda``, by closure) will have a name different from the ``cc`` implicitly introduced by the ``lambda`` itself. +> - Having a hidden parameter is somewhat magic, but overall improves readability, as this allows declaring `cc` only where actually explicitly needed. +> - **CAUTION**: Usability trap: in nested function definitions, each `def` and `lambda` comes with **its own** implicit `cc`. +> - In the above `amb` example, the local variable is named `ourcc`, so that the continuation passed in from outside (into the `lambda`, by closure) will have a name different from the `cc` implicitly introduced by the `lambda` itself. > - This is possibly subject to change in a future version (pending the invention of a better API), but for now just be aware of this gotcha. -> - Beside ``cc``, there's also a mechanism to keep track of the captured tail of a computation, which is important to have edge cases work correctly. See the note on **pcc** (*parent continuation*) in the docstring of ``unpythonic.syntax.continuations``, and [the pictures](callcc_topology.pdf). +> - Beside `cc`, there's also a mechanism to keep track of the captured tail of a computation, which is important to have edge cases work correctly. See the note on **pcc** (*parent continuation*) in the docstring of `unpythonic.syntax.continuations`, and [the pictures](callcc_topology.pdf). > -> - In a function definition inside the ``with continuations`` block: +> - In a function definition inside the `with continuations` block: > - Most of the language works as usual; especially, any non-tail function calls can be made as usual. -> - ``return value`` or ``return v0, ..., vn`` is actually a tail-call into ``cc``, passing the given value(s) as arguments. -> - As in other parts of ``unpythonic``, returning a `Values` means returning multiple-return-values. -> - This is important if the return value is received by the assignment targets of a ``call_cc[]``. If you get a ``TypeError`` concerning the arguments of a function with a name ending in ``_cont``, check your ``call_cc[]`` invocations and the ``return`` in the call_cc'd function. +> - `return value` or `return v0, ..., vn` is actually a tail-call into `cc`, passing the given value(s) as arguments. +> - As in other parts of `unpythonic`, returning a `Values` means returning multiple-return-values. +> - This is important if the return value is received by the assignment targets of a `call_cc[]`. If you get a `TypeError` concerning the arguments of a function with a name ending in `_cont`, check your `call_cc[]` invocations and the `return` in the call_cc'd function. > - **Changed in v0.15.0.** *Up to v0.14.3, multiple return values used to be represented as a `tuple`. Now returning a `tuple` means returning one value that is a tuple.* -> - ``return func(...)`` is actually a tail-call into ``func``, passing along (by default) the current value of ``cc`` to become its ``cc``. -> - Hence, the tail call is inserted between the end of the current function body and the start of the continuation ``cc``. -> - To override which continuation to use, you can specify the ``cc=...`` kwarg, as in ``return func(..., cc=mycc)``. -> - The ``cc`` argument, if passed explicitly, **must be passed by name**. -> - **CAUTION**: This is **not** enforced, as the machinery does not analyze positional arguments in any great detail. The machinery will most likely break in unintuitive ways (or at best, raise a mysterious ``TypeError``) if this rule is violated. -> - The function ``func`` must be a defined in a ``with continuations`` block, so that it knows what to do with the named argument ``cc``. -> - Attempting to tail-call a regular function breaks the TCO chain and immediately returns to the original caller (provided the function even accepts a ``cc`` named argument). -> - Be careful: ``xs = list(args); return xs`` and ``return list(args)`` mean different things. -> - TCO is automatically applied to these tail calls. This uses the exact same machinery as the ``tco`` macro. +> - `return func(...)` is actually a tail-call into `func`, passing along (by default) the current value of `cc` to become its `cc`. +> - Hence, the tail call is inserted between the end of the current function body and the start of the continuation `cc`. +> - To override which continuation to use, you can specify the `cc=...` kwarg, as in `return func(..., cc=mycc)`. +> - The `cc` argument, if passed explicitly, **must be passed by name**. +> - **CAUTION**: This is **not** enforced, as the machinery does not analyze positional arguments in any great detail. The machinery will most likely break in unintuitive ways (or at best, raise a mysterious `TypeError`) if this rule is violated. +> - The function `func` must be a defined in a `with continuations` block, so that it knows what to do with the named argument `cc`. +> - Attempting to tail-call a regular function breaks the TCO chain and immediately returns to the original caller (provided the function even accepts a `cc` named argument). +> - Be careful: `xs = list(args); return xs` and `return list(args)` mean different things. +> - TCO is automatically applied to these tail calls. This uses the exact same machinery as the `tco` macro. > -> - The ``call_cc[]`` statement essentially splits its use site into *before* and *after* parts, where the *after* part (the continuation) can be run a second and further times, by later calling the callable that represents the continuation. This makes a computation resumable from a desired point. +> - The `call_cc[]` statement essentially splits its use site into *before* and *after* parts, where the *after* part (the continuation) can be run a second and further times, by later calling the callable that represents the continuation. This makes a computation resumable from a desired point. > - The continuation is essentially a closure. -> - Just like in Scheme/Racket, only the control state is checkpointed by ``call_cc[]``; any modifications to mutable data remain. -> - Assignment targets can be used to get the return value of the function called by ``call_cc[]``. -> - Just like in Scheme/Racket's ``call/cc``, the values that get bound to the ``call_cc[]`` assignment targets on second and further calls (when the continuation runs) are the arguments given to the continuation when it is called (whether implicitly or manually). -> - A first-class reference to the captured continuation is available in the function called by ``call_cc[]``, as its ``cc`` argument. -> - The continuation is a function that takes positional arguments, plus a named argument ``cc``. -> - The call signature for the positional arguments is determined by the assignment targets of the ``call_cc[]``. -> - The ``cc`` parameter is there only so that a continuation behaves just like any continuation-enabled function when tail-called, or when later used as the target of another ``call_cc[]``. -> - Basically everywhere else, ``cc`` points to the identity function - the default continuation just returns its arguments. +> - Just like in Scheme/Racket, only the control state is checkpointed by `call_cc[]`; any modifications to mutable data remain. +> - Assignment targets can be used to get the return value of the function called by `call_cc[]`. +> - Just like in Scheme/Racket's `call/cc`, the values that get bound to the `call_cc[]` assignment targets on second and further calls (when the continuation runs) are the arguments given to the continuation when it is called (whether implicitly or manually). +> - A first-class reference to the captured continuation is available in the function called by `call_cc[]`, as its `cc` argument. +> - The continuation is a function that takes positional arguments, plus a named argument `cc`. +> - The call signature for the positional arguments is determined by the assignment targets of the `call_cc[]`. +> - The `cc` parameter is there only so that a continuation behaves just like any continuation-enabled function when tail-called, or when later used as the target of another `call_cc[]`. +> - Basically everywhere else, `cc` points to the identity function - the default continuation just returns its arguments. > - This is unlike in Scheme or Racket, which implicitly capture the continuation at every expression. -> - Inside a ``def``, ``call_cc[]`` generates a tail call, thus terminating the original (parent) function. (Hence ``call_ec`` does not combo well with this.) -> - At the top level of the ``with continuations`` block, ``call_cc[]`` generates a normal call. In this case there is no return value for the block (for the continuation, either), because the use site of the ``call_cc[]`` is not inside a function. +> - Inside a `def`, `call_cc[]` generates a tail call, thus terminating the original (parent) function. (Hence `call_ec` does not combo well with this.) +> - At the top level of the `with continuations` block, `call_cc[]` generates a normal call. In this case there is no return value for the block (for the continuation, either), because the use site of the `call_cc[]` is not inside a function.
-#### Differences between ``call/cc`` and certain other language features +#### Differences between `call/cc` and certain other language features - - Unlike **generators**, ``call_cc[]`` allows resuming also multiple times from an earlier checkpoint, even after execution has already proceeded further. Generators can be easily built on top of ``call/cc``. [Python version](../unpythonic/syntax/test/test_conts_gen.py), [Racket version](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt). + - Unlike **generators**, `call_cc[]` allows resuming also multiple times from an earlier checkpoint, even after execution has already proceeded further. Generators can be easily built on top of `call/cc`. [Python version](../unpythonic/syntax/test/test_conts_gen.py), [Racket version](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt). - The Python version is a pattern that could be packaged into a macro with `mcpyrate`; the Racket version has been packaged as a macro. - Both versions are just demonstrations for teaching purposes. In production code, use the language's native functionality. - - Python's built-in generators have no restriction on where ``yield`` can be placed, and provide better performance. + - Python's built-in generators have no restriction on where `yield` can be placed, and provide better performance. - Racket's standard library provides [generators](https://docs.racket-lang.org/reference/Generators.html). - - Unlike **exceptions**, which only perform escapes, ``call_cc[]`` allows to jump back at an arbitrary time later, also after the dynamic extent of the original function where the ``call_cc[]`` appears. Escape continuations are a special case of continuations, so exceptions can be built on top of ``call/cc``. + - Unlike **exceptions**, which only perform escapes, `call_cc[]` allows to jump back at an arbitrary time later, also after the dynamic extent of the original function where the `call_cc[]` appears. Escape continuations are a special case of continuations, so exceptions can be built on top of `call/cc`. - [As explained in detail by Matthew Might](http://matt.might.net/articles/implementing-exceptions/), exceptions are fundamentally based on (escape) continuations; the *"unwinding the call stack"* mental image is ["not even wrong"](https://en.wikiquote.org/wiki/Wolfgang_Pauli). -So if all you want is generators or exceptions (or even resumable exceptions a.k.a. [conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), then a general ``call/cc`` mechanism is not needed. The point of ``call/cc`` is to provide the ability to *resume more than once* from *the same*, already executed point in the program. In other words, ``call/cc`` is a general mechanism for bookmarking the control state. +So if all you want is generators or exceptions (or even resumable exceptions a.k.a. [conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), then a general `call/cc` mechanism is not needed. The point of `call/cc` is to provide the ability to *resume more than once* from *the same*, already executed point in the program. In other words, `call/cc` is a general mechanism for bookmarking the control state. However, its usability leaves much to be desired. This has been noted e.g. in [Oleg Kiselyov: An argument against call/cc](http://okmij.org/ftp/continuations/against-callcc.html) and [John Shutt: Guarded continuations](http://fexpr.blogspot.com/2012/01/guarded-continuations.html). For example, Shutt writes: *The traditional Scheme device for acquiring a first-class continuation object is **call/cc**, which calls a procedure and passes to that procedure the continuation to which that call would normally return. Frankly, this was always a very clumsy way to work with continuations; one might almost suspect it was devised as an "esoteric programming language" feature, akin to INTERCAL's COME FROM statement.* -#### ``call_cc`` API reference +#### `call_cc` API reference -To keep things relatively straightforward, our ``call_cc[]`` is only allowed to appear **at the top level** of: +To keep things relatively straightforward, our `call_cc[]` is only allowed to appear **at the top level** of: - - the ``with continuations`` block itself - - a ``def`` or ``async def`` + - the `with continuations` block itself + - a `def` or `async def` -Nested defs are ok; here *top level* only means the top level of the *currently innermost* ``def``. +Nested defs are ok; here *top level* only means the top level of the *currently innermost* `def`. -If you need to place ``call_cc[]`` inside a loop, use ``@looped`` et al. from ``unpythonic.fploop``; this has the loop body represented as the top level of a ``def``. +If you need to place `call_cc[]` inside a loop, use `@looped` et al. from `unpythonic.fploop`; this has the loop body represented as the top level of a `def`. -Multiple ``call_cc[]`` statements in the same function body are allowed. These essentially create nested closures. +Multiple `call_cc[]` statements in the same function body are allowed. These essentially create nested closures. In any invalid position, `call_cc[]` is considered a syntax error at macro expansion time. **Syntax**: -In ``unpythonic``, ``call_cc`` is a **statement**, with the following syntaxes: +In `unpythonic`, `call_cc` is a **statement**, with the following syntaxes: ```python x = call_cc[func(...)] @@ -1281,23 +1281,23 @@ x0, ..., *xs = call_cc[f(...) if p else g(...)] call_cc[f(...) if p else g(...)] ``` -*NOTE*: ``*xs`` may need to be written as ``*xs,`` in order to explicitly make the LHS into a tuple. The variant without the comma seems to work when run from a ``.py`` file with the `macropython` bootstrapper from [`mcpyrate`](https://pypi.org/project/mcpyrate/), but fails in code run interactively in the `mcpyrate` REPL. +*NOTE*: `*xs` may need to be written as `*xs,` in order to explicitly make the LHS into a tuple. The variant without the comma seems to work when run from a `.py` file with the `macropython` bootstrapper from [`mcpyrate`](https://pypi.org/project/mcpyrate/), but fails in code run interactively in the `mcpyrate` REPL. -*NOTE*: ``f()`` and ``g()`` must be **literal function calls**. Sneaky trickery (such as calling indirectly via ``unpythonic.funutil.call`` or ``unpythonic.fun.curry``) is not supported. (The ``prefix`` and ``curry`` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the ``call_cc[]`` needs to patch the ``cc=...`` kwarg of the call being made. +*NOTE*: `f()` and `g()` must be **literal function calls**. Sneaky trickery (such as calling indirectly via `unpythonic.funutil.call` or `unpythonic.fun.curry`) is not supported. (The `prefix` and `curry` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the `call_cc[]` needs to patch the `cc=...` kwarg of the call being made. **Assignment targets**: - To destructure positional multiple-values (from a `Values` return value of the function called by the `call_cc`), use a tuple assignment target (comma-separated names, as usual). Destructuring *named* return values from a `call_cc` is currently not supported due to syntactic limitations. - - The last assignment target may be starred. It is transformed into the vararg (a.k.a. ``*args``, star-args) of the continuation function created by the `call_cc`. (It will capture a whole tuple, or any excess items, as usual.) + - The last assignment target may be starred. It is transformed into the vararg (a.k.a. `*args`, star-args) of the continuation function created by the `call_cc`. (It will capture a whole tuple, or any excess items, as usual.) - - To ignore the return value, just omit the assignment part. Useful if ``func`` was called only to perform its side-effects (the classic side effect is to stash ``cc`` somewhere for later use). + - To ignore the return value, just omit the assignment part. Useful if `func` was called only to perform its side-effects (the classic side effect is to stash `cc` somewhere for later use). **Conditional variant**: - - ``p`` is any expression. If truthy, ``f(...)`` is called, and if falsey, ``g(...)`` is called. + - `p` is any expression. If truthy, `f(...)` is called, and if falsey, `g(...)` is called. - - Each of ``f(...)``, ``g(...)`` may be ``None``. A ``None`` skips the function call, proceeding directly to the continuation. Upon skipping, all assignment targets (if any are present) are set to ``None``. The starred assignment target (if present) gets the empty tuple. + - Each of `f(...)`, `g(...)` may be `None`. A `None` skips the function call, proceeding directly to the continuation. Upon skipping, all assignment targets (if any are present) are set to `None`. The starred assignment target (if present) gets the empty tuple. The main use case of the conditional variant is for things like: @@ -1312,21 +1312,21 @@ with continuations: ... ``` -**Main differences to ``call/cc`` in Scheme and Racket**: +**Main differences to `call/cc` in Scheme and Racket**: -Compared to Scheme/Racket, where ``call/cc`` will capture also expressions occurring further up in the call stack, our ``call_cc`` may be need to be placed differently (further out, depending on what needs to be captured) due to the delimited nature of the continuations implemented here. +Compared to Scheme/Racket, where `call/cc` will capture also expressions occurring further up in the call stack, our `call_cc` may be need to be placed differently (further out, depending on what needs to be captured) due to the delimited nature of the continuations implemented here. -Scheme and Racket implicitly capture the continuation at every position, whereas we do it explicitly, only at the use sites of the ``call_cc[]`` macro. +Scheme and Racket implicitly capture the continuation at every position, whereas we do it explicitly, only at the use sites of the `call_cc[]` macro. -Also, since there are limitations to where a ``call_cc[]`` may appear, some code may need to be structured differently to do some particular thing, if porting code examples originally written in Scheme or Racket. +Also, since there are limitations to where a `call_cc[]` may appear, some code may need to be structured differently to do some particular thing, if porting code examples originally written in Scheme or Racket. -Unlike ``call/cc`` in Scheme/Racket, our ``call_cc`` takes **a function call** as its argument, not just a function reference. Also, there's no need for it to be a one-argument function; any other args can be passed in the call. The ``cc`` argument is filled implicitly and passed by name; any others are passed exactly as written in the client code. +Unlike `call/cc` in Scheme/Racket, our `call_cc` takes **a function call** as its argument, not just a function reference. Also, there's no need for it to be a one-argument function; any other args can be passed in the call. The `cc` argument is filled implicitly and passed by name; any others are passed exactly as written in the client code. #### Combo notes -**CAUTION**: Do not use ``with tco`` inside a ``with continuations`` block; ``continuations`` already implies TCO. The ``continuations`` macro **makes no attempt** to skip ``with tco`` blocks inside it. +**CAUTION**: Do not use `with tco` inside a `with continuations` block; `continuations` already implies TCO. The `continuations` macro **makes no attempt** to skip `with tco` blocks inside it. -If you need both ``continuations`` and ``multilambda`` simultaneously, the incantation is: +If you need both `continuations` and `multilambda` simultaneously, the incantation is: ```python with multilambda, continuations: @@ -1334,9 +1334,9 @@ with multilambda, continuations: assert f(42) == 1764 ``` -This works, because the ``continuations`` macro understands already expanded ``let[]`` and ``do[]``, and ``multilambda`` generates and expands a ``do[]``. (Any explicit use of ``do[]`` in a lambda body or in a ``return`` is also ok; recall that macros expand from inside out.) +This works, because the `continuations` macro understands already expanded `let[]` and `do[]`, and `multilambda` generates and expands a `do[]`. (Any explicit use of `do[]` in a lambda body or in a `return` is also ok; recall that macros expand from inside out.) -Similarly, if you need ``quicklambda``, apply it first: +Similarly, if you need `quicklambda`, apply it first: ```python with quicklambda, continuations: @@ -1344,13 +1344,13 @@ with quicklambda, continuations: assert g(42) == 1764 ``` -This ordering makes the ``f[...]`` notation expand into standard ``lambda`` notation before ``continuations`` is expanded. +This ordering makes the `f[...]` notation expand into standard `lambda` notation before `continuations` is expanded. -To enable both of these, use ``with quicklambda, multilambda, continuations`` (although the usefulness of this combo may be questionable). +To enable both of these, use `with quicklambda, multilambda, continuations` (although the usefulness of this combo may be questionable). #### Continuations as an escape mechanism -Pretty much by the definition of a continuation, in a ``with continuations`` block, a trick that *should* at first glance produce an escape is to set ``cc`` to the ``cc`` of the caller, and then return the desired value. There is however a subtle catch, due to the way we implement continuations. +Pretty much by the definition of a continuation, in a `with continuations` block, a trick that *should* at first glance produce an escape is to set `cc` to the `cc` of the caller, and then return the desired value. There is however a subtle catch, due to the way we implement continuations. First, consider this basic strategy, without any macros: @@ -1399,11 +1399,11 @@ with continuations: assert main2() == "not odd" ``` -In the first example, ``ec`` is the escape continuation of the ``result1``/``result2`` block, due to the placement of the ``call_ec``. In the second example, the ``cc`` inside ``double_odd`` is the implicitly passed ``cc``... which, naively, should represent the continuation of the current call into ``double_odd``. So far, so good. +In the first example, `ec` is the escape continuation of the `result1`/`result2` block, due to the placement of the `call_ec`. In the second example, the `cc` inside `double_odd` is the implicitly passed `cc`... which, naively, should represent the continuation of the current call into `double_odd`. So far, so good. -However, because the example code contains no ``call_cc[]`` statements, the actual value of ``cc``, anywhere in this example, is always just ``identity``. *It's not the actual continuation.* Even though we pass the ``cc`` of ``main1``/``main2`` as an explicit argument "``ec``" to use as an escape continuation (like the first example does with ``ec``), it is still ``identity`` - and hence cannot perform an escape. +However, because the example code contains no `call_cc[]` statements, the actual value of `cc`, anywhere in this example, is always just `identity`. *It's not the actual continuation.* Even though we pass the `cc` of `main1`/`main2` as an explicit argument "`ec`" to use as an escape continuation (like the first example does with `ec`), it is still `identity` - and hence cannot perform an escape. -We must ``call_cc[]`` to request a capture of the actual continuation: +We must `call_cc[]` to request a capture of the actual continuation: ```python from unpythonic.syntax import macros, continuations, call_cc @@ -1428,43 +1428,43 @@ with continuations: This variant performs as expected. -There's also a second, even subtler catch; instead of setting ``cc = ec`` and returning a value, just tail-calling ``ec`` with that value doesn't do what we want. This is because - as explained in the rules of the ``continuations`` macro, above - a tail-call is *inserted* between the end of the function, and whatever ``cc`` currently points to. +There's also a second, even subtler catch; instead of setting `cc = ec` and returning a value, just tail-calling `ec` with that value doesn't do what we want. This is because - as explained in the rules of the `continuations` macro, above - a tail-call is *inserted* between the end of the function, and whatever `cc` currently points to. -Most often that's exactly what we want, but in this particular case, it causes *both* continuations to run, in sequence. But if we overwrite ``cc``, then the function's original ``cc`` argument (the one given by ``call_cc[]``) is discarded, so it never runs - and we get the effect we want, *replacing* the ``cc`` by the ``ec``. +Most often that's exactly what we want, but in this particular case, it causes *both* continuations to run, in sequence. But if we overwrite `cc`, then the function's original `cc` argument (the one given by `call_cc[]`) is discarded, so it never runs - and we get the effect we want, *replacing* the `cc` by the `ec`. -Such subtleties arise essentially from the difference between a language that natively supports continuations (Scheme, Racket) and one that has continuations hacked on top of it as macros performing a CPS conversion only partially (like Python with ``unpythonic.syntax``, or Common Lisp with PG's continuation-passing macros). The macro approach works, but the programmer needs to be careful. +Such subtleties arise essentially from the difference between a language that natively supports continuations (Scheme, Racket) and one that has continuations hacked on top of it as macros performing a CPS conversion only partially (like Python with `unpythonic.syntax`, or Common Lisp with PG's continuation-passing macros). The macro approach works, but the programmer needs to be careful. #### What can be used as a continuation? -In ``unpythonic`` specifically, a continuation is just a function. ([As John Shutt has pointed out](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html), in general this is not true. The calculus underlying the language becomes much cleaner if continuations are defined as a separate control flow mechanism orthogonal to function application. Continuations are not intrinsically a whole-computation device, either.) +In `unpythonic` specifically, a continuation is just a function. ([As John Shutt has pointed out](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html), in general this is not true. The calculus underlying the language becomes much cleaner if continuations are defined as a separate control flow mechanism orthogonal to function application. Continuations are not intrinsically a whole-computation device, either.) The continuation function must be able to take as many positional arguments as the previous function in the TCO chain is trying to pass into it. Keep in mind that: - - In ``unpythonic``, multiple return values are represented as a `Values` object. So if your function does ``return Values(a, b)``, and that is being fed into the continuation, this implies that the continuation must be able to take two positional arguments. + - In `unpythonic`, multiple return values are represented as a `Values` object. So if your function does `return Values(a, b)`, and that is being fed into the continuation, this implies that the continuation must be able to take two positional arguments. **Changed in v0.15.0.** *Up to v0.14.3, a `tuple` used to represent multiple-return-values; now it denotes a single return value that is a tuple. The `Values` type allows not only multiple return values, but also **named** return values. These are fed as kwargs.* - - At the end of any function in Python, at least an implicit bare ``return`` always exists. It will try to pass in the value ``None`` to the continuation, so the continuation must be able to accept one positional argument. (This is handled automatically for continuations created by ``call_cc[]``. If no assignment targets are given, ``call_cc[]`` automatically creates one ignored positional argument that defaults to ``None``.) + - At the end of any function in Python, at least an implicit bare `return` always exists. It will try to pass in the value `None` to the continuation, so the continuation must be able to accept one positional argument. (This is handled automatically for continuations created by `call_cc[]`. If no assignment targets are given, `call_cc[]` automatically creates one ignored positional argument that defaults to `None`.) -If there is an arity mismatch, Python will raise ``TypeError`` as usual. (The actual error message may be unhelpful due to the macro transformations; look for a mismatch in the number of values between a ``return`` and the call signature of a function used as a continuation (most often, the ``f`` in a ``cc=f``).) +If there is an arity mismatch, Python will raise `TypeError` as usual. (The actual error message may be unhelpful due to the macro transformations; look for a mismatch in the number of values between a `return` and the call signature of a function used as a continuation (most often, the `f` in a `cc=f`).) -Usually, a function to be used as a continuation is defined inside the ``with continuations`` block. This automatically introduces the implicit ``cc`` parameter, and in general makes the source code undergo the transformations needed by the continuation machinery. +Usually, a function to be used as a continuation is defined inside the `with continuations` block. This automatically introduces the implicit `cc` parameter, and in general makes the source code undergo the transformations needed by the continuation machinery. -However, as the only exception to this rule, if the continuation is meant to act as the endpoint of the TCO chain - i.e. terminating the chain and returning to the original top-level caller - then it may be defined outside the ``with continuations`` block. Recall that in a ``with continuations`` block, returning an inert data value (i.e. not making a tail call) transforms into a tail-call into the ``cc`` (with the given data becoming its argument(s)); it does not set the ``cc`` argument of the continuation being called, or even require that it has a ``cc`` parameter that could accept one. +However, as the only exception to this rule, if the continuation is meant to act as the endpoint of the TCO chain - i.e. terminating the chain and returning to the original top-level caller - then it may be defined outside the `with continuations` block. Recall that in a `with continuations` block, returning an inert data value (i.e. not making a tail call) transforms into a tail-call into the `cc` (with the given data becoming its argument(s)); it does not set the `cc` argument of the continuation being called, or even require that it has a `cc` parameter that could accept one. -(Note also that a continuation that has no ``cc`` parameter cannot be used as the target of an explicit tail-call in the client code, since a tail-call in a ``with continuations`` block will attempt to supply a ``cc`` argument to the function being tail-called. Likewise, it cannot be used as the target of a ``call_cc[]``, since this will also attempt to supply a ``cc`` argument.) +(Note also that a continuation that has no `cc` parameter cannot be used as the target of an explicit tail-call in the client code, since a tail-call in a `with continuations` block will attempt to supply a `cc` argument to the function being tail-called. Likewise, it cannot be used as the target of a `call_cc[]`, since this will also attempt to supply a `cc` argument.) -These observations make ``unpythonic.fun.identity`` eligible as a continuation, even though it is defined elsewhere in the library and it has no ``cc`` parameter. +These observations make `unpythonic.fun.identity` eligible as a continuation, even though it is defined elsewhere in the library and it has no `cc` parameter. -#### This isn't ``call/cc``! +#### This isn't `call/cc`! -Strictly speaking, ``True``. The implementation is very different (much more than just [exposing a hidden parameter](https://www.ps.uni-saarland.de/~duchier/python/continuations.html)), not to mention it has to be a macro, because it triggers capture - something that would not need to be requested for separately, had we converted the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style). +Strictly speaking, `True`. The implementation is very different (much more than just [exposing a hidden parameter](https://www.ps.uni-saarland.de/~duchier/python/continuations.html)), not to mention it has to be a macro, because it triggers capture - something that would not need to be requested for separately, had we converted the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style). -The selective capture approach is however more efficient when we implement the continuation system in Python, indeed *on Python* (in the sense of [On Lisp](paulgraham.com/onlisp.html)), since we want to run most of the program the usual way with no magic attached. This way there is no need to sprinkle absolutely every statement and expression with a ``def`` or a ``lambda``. (Not to mention Python's ``lambda`` is underpowered due to the existence of some language features only as statements, so we would need to use a mixture of both, which is already unnecessarily complicated.) Function definitions are not intended as [the only control flow construct](https://dspace.mit.edu/handle/1721.1/5753) in Python, so the compiler likely wouldn't optimize heavily enough (i.e. eliminate **almost all** of the implicitly introduced function definitions), if we attempted to use them as such. +The selective capture approach is however more efficient when we implement the continuation system in Python, indeed *on Python* (in the sense of [On Lisp](paulgraham.com/onlisp.html)), since we want to run most of the program the usual way with no magic attached. This way there is no need to sprinkle absolutely every statement and expression with a `def` or a `lambda`. (Not to mention Python's `lambda` is underpowered due to the existence of some language features only as statements, so we would need to use a mixture of both, which is already unnecessarily complicated.) Function definitions are not intended as [the only control flow construct](https://dspace.mit.edu/handle/1721.1/5753) in Python, so the compiler likely wouldn't optimize heavily enough (i.e. eliminate **almost all** of the implicitly introduced function definitions), if we attempted to use them as such. Continuations only need to come into play when we explicitly request for one ([ZoP §2](https://www.python.org/dev/peps/pep-0020/)); this avoids introducing any more extra function definitions than needed. -The name is nevertheless ``call_cc``, because the resulting behavior is close enough to ``call/cc``. +The name is nevertheless `call_cc`, because the resulting behavior is close enough to `call/cc`. Note our implementation provides a rudimentary form of *delimited* continuations. See [Oleg Kiselyov: Undelimited continuations are co-values rather than functions](http://okmij.org/ftp/continuations/undelimited.html). Delimited continuations return a value and can be composed, so they at least resemble functions (even though are not, strictly speaking, actually functions), whereas undelimited continuations do not even return. (For two different debunkings of the continuations-are-functions myth, approaching the problem from completely different angles, see the above post by Oleg Kiselyov, and [John Shutt: Continuations and term-rewriting calculi](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html).) @@ -1472,7 +1472,7 @@ Racket provides a thought-out implementation of delimited continuations and [pro #### Why this syntax? -As for a function call in ``call_cc[...]`` vs. just a function reference: Typical lispy usage of ``call/cc`` uses an inline lambda, with the closure property passing in everything except ``cc``, but in Python ``def`` is a statement. A technically possible alternative syntax would be: +As for a function call in `call_cc[...]` vs. just a function reference: Typical lispy usage of `call/cc` uses an inline lambda, with the closure property passing in everything except `cc`, but in Python `def` is a statement. A technically possible alternative syntax would be: ```python with call_cc(f): # this syntax not supported! @@ -1482,17 +1482,17 @@ with call_cc(f): # this syntax not supported! but the expr macro variant provides better options for receiving multiple return values, and perhaps remains closer to standard Python. -The ``call_cc[]`` explicitly suggests that these are (almost) the only places where the ``cc`` argument obtains a non-default value. It also visually indicates the exact position of the checkpoint, while keeping to standard Python syntax. +The `call_cc[]` explicitly suggests that these are (almost) the only places where the `cc` argument obtains a non-default value. It also visually indicates the exact position of the checkpoint, while keeping to standard Python syntax. -(*Almost*: As explained above, a tail call passes along the current value of ``cc``, and ``cc`` can be set manually.) +(*Almost*: As explained above, a tail call passes along the current value of `cc`, and `cc` can be set manually.) -### ``prefix``: prefix function call syntax for Python +### `prefix`: prefix function call syntax for Python Write Python almost like Lisp! -Lexically inside a ``with prefix`` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the ``let`` macros and the top-level tuple in a ``do[]`` are left alone, but ``prefix`` recurses inside them (in the case of bindings, on each RHS). +Lexically inside a `with prefix` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the `let` macros and the top-level tuple in a `do[]` are left alone, but `prefix` recurses inside them (in the case of bindings, on each RHS). The rest is best explained by example: @@ -1545,7 +1545,7 @@ with prefix: If you use the `q`, `u` and `kw()` operators, they must be macro-imported. The `q`, `u` and `kw()` operators may only appear in a tuple inside a prefix block. In any invalid position, any of them is considered a syntax error at macro expansion time. -This comboes with ``autocurry`` for an authentic *Listhell* programming experience: +This comboes with `autocurry` for an authentic *Listhell* programming experience: ```python from unpythonic.syntax import macros, autocurry, prefix, q, u, kw @@ -1558,14 +1558,14 @@ with prefix, autocurry: # important: apply prefix first, then autocurry assert (mymap, double, (q, 1, 2, 3)) == ll(2, 4, 6) ``` -**CAUTION**: The ``prefix`` macro is experimental and not intended for use in production code. +**CAUTION**: The `prefix` macro is experimental and not intended for use in production code. -### ``autoreturn``: implicit ``return`` in tail position +### `autoreturn`: implicit `return` in tail position -In Lisps, a function implicitly returns the value of the expression in tail position (along the code path being executed). Python's ``lambda`` also behaves like this (the whole body is just one return-value expression), but ``def`` doesn't. +In Lisps, a function implicitly returns the value of the expression in tail position (along the code path being executed). Python's `lambda` also behaves like this (the whole body is just one return-value expression), but `def` doesn't. -Now ``def`` can, too: +Now `def` can, too: ```python from unpythonic.syntax import macros, autoreturn @@ -1589,33 +1589,33 @@ with autoreturn: assert g(42) == "something else" ``` -Each ``def`` function definition lexically within the ``with autoreturn`` block is examined, and if the last item within the body is an expression ``expr``, it is transformed into ``return expr``. Additionally: +Each `def` function definition lexically within the `with autoreturn` block is examined, and if the last item within the body is an expression `expr`, it is transformed into `return expr`. Additionally: - - If the last item is an ``if``/``elif``/``else`` block, the transformation is applied to the last item in each of its branches. + - If the last item is an `if`/`elif`/`else` block, the transformation is applied to the last item in each of its branches. - - If the last item is a ``with`` or ``async with`` block, the transformation is applied to the last item in its body. + - If the last item is a `with` or `async with` block, the transformation is applied to the last item in its body. - - If the last item is a ``try``/``except``/``else``/``finally`` block: - - **If** an ``else`` clause is present, the transformation is applied to the last item in it; **otherwise**, to the last item in the ``try`` clause. These are the positions that indicate a normal return (no exception was raised). - - In both cases, the transformation is applied to the last item in each of the ``except`` clauses. - - The ``finally`` clause is not transformed; the intention is it is usually a finalizer (e.g. to release resources) that runs after the interesting value is already being returned by ``try``, ``else`` or ``except``. + - If the last item is a `try`/`except`/`else`/`finally` block: + - **If** an `else` clause is present, the transformation is applied to the last item in it; **otherwise**, to the last item in the `try` clause. These are the positions that indicate a normal return (no exception was raised). + - In both cases, the transformation is applied to the last item in each of the `except` clauses. + - The `finally` clause is not transformed; the intention is it is usually a finalizer (e.g. to release resources) that runs after the interesting value is already being returned by `try`, `else` or `except`. If needed, the above rules are applied recursively to locate the tail position(s). -Any explicit ``return`` statements are left alone, so ``return`` can still be used as usual. +Any explicit `return` statements are left alone, so `return` can still be used as usual. -**CAUTION**: If the final ``else`` of an ``if``/``elif``/``else`` is omitted, as often in Python, then only the ``else`` item is in tail position with respect to the function definition - likely not what you want. So with ``autoreturn``, the final ``else`` should be written out explicitly, to make the ``else`` branch part of the same ``if``/``elif``/``else`` block. +**CAUTION**: If the final `else` of an `if`/`elif`/`else` is omitted, as often in Python, then only the `else` item is in tail position with respect to the function definition - likely not what you want. So with `autoreturn`, the final `else` should be written out explicitly, to make the `else` branch part of the same `if`/`elif`/`else` block. -**CAUTION**: ``for``, ``async for``, ``while`` are currently not analyzed; effectively, these are defined as always returning ``None``. If the last item in your function body is a loop, use an explicit return. +**CAUTION**: `for`, `async for`, `while` are currently not analyzed; effectively, these are defined as always returning `None`. If the last item in your function body is a loop, use an explicit return. -**CAUTION**: With ``autoreturn`` enabled, functions no longer return ``None`` by default; the whole point of this macro is to change the default return value. The default return value is ``None`` only if the tail position contains a statement other than ``if``, ``with``, ``async with`` or ``try``. +**CAUTION**: With `autoreturn` enabled, functions no longer return `None` by default; the whole point of this macro is to change the default return value. The default return value is `None` only if the tail position contains a statement other than `if`, `with`, `async with` or `try`. -If you wish to omit ``return`` in tail calls, this comboes with ``tco``; just apply ``autoreturn`` first (either ``with autoreturn, tco:`` or in nested format, ``with tco:``, ``with autoreturn:``). +If you wish to omit `return` in tail calls, this comboes with `tco`; just apply `autoreturn` first (either `with autoreturn, tco:` or in nested format, `with tco:`, `with autoreturn:`). -### ``forall``: nondeterministic evaluation +### `forall`: nondeterministic evaluation -Behaves the same as the multiple-body-expression tuple comprehension ``unpythonic.amb.forall``, but implemented purely by AST transformation, with real lexical variables. This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see ``unpythonic.syntax.forall``). +Behaves the same as the multiple-body-expression tuple comprehension `unpythonic.amb.forall`, but implemented purely by AST transformation, with real lexical variables. This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see `unpythonic.syntax.forall`). ```python from unpythonic.syntax import macros, forall, insist, deny @@ -1636,18 +1636,18 @@ assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), (8, 15, 17), (9, 12, 15), (12, 16, 20)) ``` -Assignment (with List-monadic magic) is ``var << iterable``. It is only valid at the top level of the ``forall`` (e.g. not inside any possibly nested ``let``). +Assignment (with List-monadic magic) is `var << iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). -``insist`` and ``deny`` are not really macros; they are just the functions from ``unpythonic.amb``, re-exported for convenience. +`insist` and `deny` are not really macros; they are just the functions from `unpythonic.amb`, re-exported for convenience. -The error raised by an undefined name in a ``forall`` section is ``NameError``. +The error raised by an undefined name in a `forall` section is `NameError`. ## Convenience features Small macros that are not essential but make some things easier or simpler. -### ``cond``: the missing ``elif`` for ``a if p else b`` +### `cond`: the missing `elif` for `a if p else b` Now lambdas too can have multi-branch conditionals, yet remain human-readable: @@ -1660,9 +1660,9 @@ answer = lambda x: cond[x == 2, "two", print(answer(42)) ``` -Syntax is ``cond[test1, then1, test2, then2, ..., otherwise]``. Expansion raises an error if the ``otherwise`` branch is missing. +Syntax is `cond[test1, then1, test2, then2, ..., otherwise]`. Expansion raises an error if the `otherwise` branch is missing. -Any part of ``cond`` may have multiple expressions by surrounding it with brackets: +Any part of `cond` may have multiple expressions by surrounding it with brackets: ```python cond[[pre1, ..., test1], [post1, ..., then1], @@ -1671,12 +1671,12 @@ cond[[pre1, ..., test1], [post1, ..., then1], [postn, ..., otherwise]] ``` -To denote a single expression that is a literal list, use an extra set of brackets: ``[[1, 2, 3]]``. +To denote a single expression that is a literal list, use an extra set of brackets: `[[1, 2, 3]]`. -### ``aif``: anaphoric if +### `aif`: anaphoric if -This is mainly of interest as a point of [comparison with Racket](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/aif.rkt); ``aif`` is about the simplest macro that relies on either the lack of hygiene or breaking thereof. +This is mainly of interest as a point of [comparison with Racket](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/aif.rkt); `aif` is about the simplest macro that relies on either the lack of hygiene or breaking thereof. ```python from unpythonic.syntax import macros, aif, it @@ -1686,9 +1686,9 @@ aif[2*21, print("it is falsey")] ``` -Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the ``then`` and ``otherwise`` parts of ``aif``, and anywhere else is considered a syntax error at macro expansion time. +Syntax is `aif[test, then, otherwise]`. The magic identifier `it` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the `then` and `otherwise` parts of `aif`, and anywhere else is considered a syntax error at macro expansion time. -Any part of ``aif`` may have multiple expressions by surrounding it with brackets (implicit ``do[]``): +Any part of `aif` may have multiple expressions by surrounding it with brackets (implicit `do[]`): ```python aif[[pre, ..., test], @@ -1696,12 +1696,12 @@ aif[[pre, ..., test], [post_false, ..., otherwise]] # "otherwise" branch ``` -To denote a single expression that is a literal list, use an extra set of brackets: ``[[1, 2, 3]]``. +To denote a single expression that is a literal list, use an extra set of brackets: `[[1, 2, 3]]`. -### ``autoref``: implicitly reference attributes of an object +### `autoref`: implicitly reference attributes of an object -Ever wish you could ``with(obj)`` to say ``x`` instead of ``obj.x`` to read attributes of an object? Enter the ``autoref`` block macro: +Ever wish you could `with(obj)` to say `x` instead of `obj.x` to read attributes of an object? Enter the `autoref` block macro: ```python from unpythonic.syntax import macros, autoref @@ -1715,13 +1715,13 @@ with autoref(e): assert c == 3 # no c in e, so just c ``` -The transformation is applied for names in ``Load`` context only, including names found in ``Attribute`` or ``Subscript`` nodes. +The transformation is applied for names in `Load` context only, including names found in `Attribute` or `Subscript` nodes. -Names in ``Store`` or ``Del`` context are not redirected. To write to or delete attributes of ``o``, explicitly refer to ``o.x``, as usual. +Names in `Store` or `Del` context are not redirected. To write to or delete attributes of `o`, explicitly refer to `o.x`, as usual. Nested autoref blocks are allowed (lookups are lexically scoped). -Reading with ``autoref`` can be convenient e.g. for data returned by [SciPy's ``.mat`` file loader](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html). +Reading with `autoref` can be convenient e.g. for data returned by [SciPy's `.mat` file loader](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html). See the [unit tests](../unpythonic/syntax/test/test_autoref.py) for more usage examples. @@ -1734,7 +1734,7 @@ This is similar to the JavaScript [`with` construct](https://developer.mozilla.o ## Testing and debugging -### ``unpythonic.test.fixtures``: a test framework for macro-enabled Python +### `unpythonic.test.fixtures`: a test framework for macro-enabled Python **Added in v0.14.3.** @@ -2007,7 +2007,7 @@ If nothing but such trivialities were captured, the failure message will instead To make testing/debugging macro code more convenient, the `the[]` mechanism automatically unparses an AST value into its source code representation for display in the test failure message. This is meant for debugging macro utilities, to which a test case hands some quoted code (i.e. code lifted into its AST representation using mcpyrate's `q[]` macro). See [`unpythonic.syntax.test.test_letdoutil`](unpythonic/syntax/test/test_letdoutil.py) for some examples. (Note the unparsing is done for display only; the raw value remains inspectable in the exception instance.) -**CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See ``mcpyrate.unparse``. +**CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See `mcpyrate.unparse`. **CAUTION**: The name of the `the[]` construct was inspired by Common Lisp, but the semantics are completely different. Common Lisp's `THE` is a return-type declaration (pythonistas would say *return-type annotation*), meant as a hint for the compiler to produce performance-optimized compiled code (see [chapter 32 of Peter Seibel's Practical Common Lisp](http://www.gigamonkeys.com/book/conclusion-whats-next.html)), whereas our `the[]` captures a value for test reporting. The only common factors are the name, and that neither construct changes the semantics of the marked code, much. In `unpythonic.test.fixtures`, the reason behind picking this name was that it doesn't change the flow of the source code as English that much, specifically to suggest, between the lines, that it doesn't change the semantics much. The reasoning behind CL's `THE` may be similar. @@ -2080,7 +2080,7 @@ A test framework can be reused across many different projects, and the error-cat Inspired by [Julia](https://julialang.org/)'s standard-library [`Test` package](https://docs.julialang.org/en/v1/stdlib/Test/), and [chapter 9 of Peter Seibel's Practical Common Lisp](http://www.gigamonkeys.com/book/practical-building-a-unit-test-framework.html). -### ``dbg``: debug-print expressions with source code +### `dbg`: debug-print expressions with source code **Changed in 0.14.2.** The `dbg[]` macro now works in the REPL, too. You can use `mcpyrate.repl.console` (a.k.a. `macropython -i` in the shell) or the IPython extension `mcpyrate.repl.iconsole`. @@ -2105,7 +2105,7 @@ z = dbg[25 + 17] # --> [file.py:15] (25 + 17): 42 assert z == 42 # surrounding an expression with dbg[...] doesn't alter its value ``` -**In the block variant**, just like in ``nb``, a custom print function can be supplied as the first positional argument. This avoids transforming any uses of built-in ``print``: +**In the block variant**, just like in `nb`, a custom print function can be supplied as the first positional argument. This avoids transforming any uses of built-in `print`: ```python prt = lambda *args, **kwargs: print(*args) @@ -2122,13 +2122,13 @@ with dbg[prt]: ``` -The reference to the custom print function (i.e. the argument to the ``dbg`` block) **must be a bare name**. Support for methods may or may not be added in a future version. +The reference to the custom print function (i.e. the argument to the `dbg` block) **must be a bare name**. Support for methods may or may not be added in a future version. -**In the expr variant**, to customize printing, just assign a function to the dynvar ``dbgprint_expr`` via `with dyn.let(dbgprint_expr=...)`. If no custom printer is set, a default implementation is used. +**In the expr variant**, to customize printing, just assign a function to the dynvar `dbgprint_expr` via `with dyn.let(dbgprint_expr=...)`. If no custom printer is set, a default implementation is used. -For details on implementing custom debug print functions, see the docstrings of ``unpythonic.syntax.dbgprint_block`` and ``unpythonic.syntax.dbgprint_expr``, which provide the default implementations. +For details on implementing custom debug print functions, see the docstrings of `unpythonic.syntax.dbgprint_block` and `unpythonic.syntax.dbgprint_expr`, which provide the default implementations. -**CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See ``mcpyrate.unparse``. +**CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See `mcpyrate.unparse`. Inspired by the [dbg macro in Rust](https://doc.rust-lang.org/std/macro.dbg.html). @@ -2136,9 +2136,9 @@ Inspired by the [dbg macro in Rust](https://doc.rust-lang.org/std/macro.dbg.html Stuff that didn't fit elsewhere. -### ``nb``: silly ultralight math notebook +### `nb`: silly ultralight math notebook -Mix regular code with math-notebook-like code in a ``.py`` file. To enable notebook mode, ``with nb``: +Mix regular code with math-notebook-like code in a `.py` file. To enable notebook mode, `with nb`: ```python from unpythonic.syntax import macros, nb @@ -2158,9 +2158,9 @@ with nb[pprint]: assert _ == 3 * x * y ``` -Expressions at the top level auto-assign the result to ``_``, and auto-print it if the value is not ``None``. Only expressions do that; for any statement that is not an expression, ``_`` retains its previous value. +Expressions at the top level auto-assign the result to `_`, and auto-print it if the value is not `None`. Only expressions do that; for any statement that is not an expression, `_` retains its previous value. -A custom print function can be supplied as the first positional argument to ``nb``. This is useful with SymPy (and [latex-input](https://github.com/clarkgrubb/latex-input) to use α, β, γ, ... as actual variable names). +A custom print function can be supplied as the first positional argument to `nb`. This is useful with SymPy (and [latex-input](https://github.com/clarkgrubb/latex-input) to use α, β, γ, ... as actual variable names). Obviously not intended for production use, although is very likely to work anywhere. @@ -2170,7 +2170,7 @@ Is this just a set of macros, a language extension, or a compiler for a new lang ### The xmas tree combo -The macros in ``unpythonic.syntax`` are designed to work together, but some care needs to be taken regarding the order in which they expand. This complexity unfortunately comes with any pick-and-mix-your-own-language kit, because some features inevitably interact. For example, it is possible to lazify [continuation-enabled](https://en.wikipedia.org/wiki/Continuation-passing_style) code, but running the transformations the other way around produces nonsense. +The macros in `unpythonic.syntax` are designed to work together, but some care needs to be taken regarding the order in which they expand. This complexity unfortunately comes with any pick-and-mix-your-own-language kit, because some features inevitably interact. For example, it is possible to lazify [continuation-enabled](https://en.wikipedia.org/wiki/Continuation-passing_style) code, but running the transformations the other way around produces nonsense. The correct **xmas tree invocation** is: @@ -2188,7 +2188,7 @@ We have taken into account that: [The dialect examples](dialects.md) use this ordering. -For simplicity, **the block macros make no attempt to prevent invalid combos**, unless there is a specific technical reason to do that for some particular combination. Be careful; e.g. don't nest several ``with tco`` blocks (lexically), that won't work. +For simplicity, **the block macros make no attempt to prevent invalid combos**, unless there is a specific technical reason to do that for some particular combination. Be careful; e.g. don't nest several `with tco` blocks (lexically), that won't work. As an example of a specific technical reason, the `tco` macro skips already expanded `with continuations` blocks lexically contained within the `with tco`. This allows the [Lispython dialect](dialects/lispython.md) to support `continuations`. @@ -2202,9 +2202,9 @@ prefix > autoreturn, quicklambda > multilambda > continuations or tco > ... ... > autocurry > namedlambda, autoref > lazify > envify ``` -The ``let_syntax`` (and ``abbrev``) block may be placed anywhere in the chain; just keep in mind what it does. +The `let_syntax` (and `abbrev`) block may be placed anywhere in the chain; just keep in mind what it does. -The ``dbg`` block can be run at any position after ``prefix`` and before ``tco`` (or ``continuations``). It must be able to see function calls in Python's standard format, for detecting calls to the print function. +The `dbg` block can be run at any position after `prefix` and before `tco` (or `continuations`). It must be able to see function calls in Python's standard format, for detecting calls to the print function. The correct ordering for **block macro invocations** - which is the actual user-facing part - is somewhat complicated by the fact that some of the above are two-pass macros. Consider this artificial example, where `mac` is a two-pass macro: @@ -2295,6 +2295,6 @@ In a basic Emacs setup, the snippet goes into the `~/.emacs` startup file, or if ### This is semantics, not syntax! -[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``. We just repurpose Python's existing syntax to give it new meanings. However, in [the Racket reference](https://docs.racket-lang.org/reference/), **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less obscure, and close enough to convey the intended meaning. +[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), `True`. We just repurpose Python's existing syntax to give it new meanings. However, in [the Racket reference](https://docs.racket-lang.org/reference/), **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name `unpythonic.syntax` is also shorter to type than `unpythonic.semantics`, less obscure, and close enough to convey the intended meaning. If you want custom *syntax* proper, or want to package a set of block macros as a custom language that extends Python, then you may be interested in our sister project [`mcpyrate`](https://github.com/Technologicat/mcpyrate). From faa7ca5af7e4a1ce64612b1a598acc9c77a1754b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 16:50:16 +0300 Subject: [PATCH 522/832] markdown: use single backticks --- doc/design-notes.md | 124 +++++++++++++++++++------------------- doc/dialects.md | 2 +- doc/dialects/lispython.md | 60 +++++++++--------- doc/dialects/listhell.md | 14 ++--- doc/dialects/pytkell.md | 38 ++++++------ doc/essays.md | 6 +- 6 files changed, 122 insertions(+), 122 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 80015eb4..06370ab8 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -19,7 +19,7 @@ - [Language Discontinuities](#language-discontinuities) - [`unpythonic` and the Killer Features of Common Lisp](#unpythonic-and-the-killer-features-of-common-lisp) - [Python is not a Lisp](#python-is-not-a-lisp) - - [On ``let`` and Python](#on-let-and-python) + - [On `let` and Python](#on-let-and-python) - [Assignment syntax](#assignment-syntax) - [TCO syntax and speed](#tco-syntax-and-speed) - [No Monads?](#no-monads) @@ -47,7 +47,7 @@ The library is split into **three layers**, providing **four kinds of features** We believe syntactic macros are [*the nuclear option of software engineering*](https://www.factual.com/blog/thinking-in-clojure-for-java-programmers-part-2/). Accordingly, we aim to [minimize macro magic](https://macropy3.readthedocs.io/en/latest/discussion.html#minimize-macro-magic). If a feature can be implemented - *with a level of usability on par with pythonic standards* - without resorting to macros, then it belongs in the pure-Python layer. (The one exception is when building the feature as a macro is the *simpler* solution. Consider `unpythonic.amb.forall` (overly complicated, to avoid macros) vs. `unpythonic.syntax.forall` (a clean macro-based design of the same feature) as an example. Keep in mind [ZoP](https://www.python.org/dev/peps/pep-0020/) §17 and §18.) -When that is not possible, we implement the actual feature as a pure-Python core, not meant for direct use, and provide a macro layer on top. The purpose of the macro layer is then to improve usability, by eliminating the [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet) from the user interface of the pure-Python core. Examples are *automatic* currying, *automatic* tail-call optimization, and (beside a much leaner syntax) lexical scoping for the ``let`` and ``do`` constructs. We believe a well-designed macro layer can bring a difference in user experience similar to that between programming in [Brainfuck](https://en.wikipedia.org/wiki/Brainfuck) (or to be fair, in Fortran or in Java) versus in Python. +When that is not possible, we implement the actual feature as a pure-Python core, not meant for direct use, and provide a macro layer on top. The purpose of the macro layer is then to improve usability, by eliminating the [accidental complexity](https://en.wikipedia.org/wiki/No_Silver_Bullet) from the user interface of the pure-Python core. Examples are *automatic* currying, *automatic* tail-call optimization, and (beside a much leaner syntax) lexical scoping for the `let` and `do` constructs. We believe a well-designed macro layer can bring a difference in user experience similar to that between programming in [Brainfuck](https://en.wikipedia.org/wiki/Brainfuck) (or to be fair, in Fortran or in Java) versus in Python. Finally, when the whole purpose of the feature is to automatically transform a piece of code into a particular style (`continuations`, `lazify`, `autoreturn`), or when run-time access to the original [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree) is essential to the purpose (`dbg`), then the feature belongs squarely in the macro layer, with no pure-Python core underneath. @@ -58,7 +58,7 @@ When to implement your own feature as a syntactic macro, see the discussion in C Making macros work together is nontrivial, essentially because *macros don't compose*. [As pointed out by John Shutt](https://fexpr.blogspot.com/2013/12/abstractive-power.html), in a multilayered language extension implemented with macros, the second layer of macros needs to understand all of the first layer. The issue is that the macro abstraction leaks the details of its expansion. Contrast with functions, which operate on values: the process that was used to arrive at a value doesn't matter. It's always possible for a function to take this value and transform it into another value, which can then be used as input for the next layer of functions. That's composability at its finest. -The need for interaction between macros may arise already in what *feels* like a single layer of abstraction; for example, it's not only that the block macros must understand ``let[]``, but some of them must understand other block macros. This is because what feels like one layer of abstraction is actually implemented as a number of separate macros, which run in a specific order. Thus, from the viewpoint of actually applying the macros, if the resulting software is to work correctly, the mere act of allowing combos between the block macros already makes them into a multilayer system. The compartmentalization of conceptually separate features into separate macros facilitates understanding and maintainability, but fails to reach the ideal of modularity. +The need for interaction between macros may arise already in what *feels* like a single layer of abstraction; for example, it's not only that the block macros must understand `let[]`, but some of them must understand other block macros. This is because what feels like one layer of abstraction is actually implemented as a number of separate macros, which run in a specific order. Thus, from the viewpoint of actually applying the macros, if the resulting software is to work correctly, the mere act of allowing combos between the block macros already makes them into a multilayer system. The compartmentalization of conceptually separate features into separate macros facilitates understanding and maintainability, but fails to reach the ideal of modularity. Therefore, any particular combination of macros that has not been specifically tested might not work. That said, if some particular combo doesn't work and *is not at least documented as such*, that's an error; please raise an issue. The unit tests should cover the combos that on the surface seem the most useful, but there's no guarantee that they cover everything that actually is useful somewhere. @@ -111,33 +111,33 @@ But for those of us that [don't like parentheses](https://srfi.schemers.org/srfi ## Python is not a Lisp -The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macros.md)) is to make Python lambdas slightly more useful - which was really the starting point for the whole `unpythonic` experiment. +The point behind providing `let` and `begin` (and the `let[]` and `do[]` [macros](macros.md)) is to make Python lambdas slightly more useful - which was really the starting point for the whole `unpythonic` experiment. -The oft-quoted single-expression limitation of the Python ``lambda`` is ultimately a herring, as this library demonstrates. The real problem is the statement/expression dichotomy. In Python, the looping constructs (`for`, `while`), the full power of `if`, and `return` are statements, so they cannot be used in lambdas. (This observation has been earlier made by others, too; see e.g. the [Wikipedia page on anonymous functions](https://en.wikipedia.org/wiki/Anonymous_function#Python).) We can work around some of this: +The oft-quoted single-expression limitation of the Python `lambda` is ultimately a herring, as this library demonstrates. The real problem is the statement/expression dichotomy. In Python, the looping constructs (`for`, `while`), the full power of `if`, and `return` are statements, so they cannot be used in lambdas. (This observation has been earlier made by others, too; see e.g. the [Wikipedia page on anonymous functions](https://en.wikipedia.org/wiki/Anonymous_function#Python).) We can work around some of this: - The expr macro `do[]` gives us sequencing, i.e. allows to use, in any expression position, multiple expressions that run in the specified order. - - The expr macro ``cond[]`` gives us a general ``if``/``elif``/``else`` expression. - - Without it, the expression form of `if` (that Python already has) could be used, but readability suffers if nested, since it has no ``elif``. Actually, [`and` and `or` are sufficient for full generality](https://www.ibm.com/developerworks/library/l-prog/), but readability suffers even more. - - So we use macros to define a ``cond`` expression, essentially duplicating a feature the language already almost has. See [our macros](macros.md). - - Functional looping (with TCO) gives us equivalents of ``for`` and ``while``. See the constructs in ``unpythonic.fploop``, particularly ``looped`` and ``breakably_looped``. - - ``unpythonic.ec.call_ec`` gives us ``return`` (the ec). - - ``unpythonic.misc.raisef`` gives us ``raise``, and ``unpythonic.misc.tryf`` gives us ``try``/``except``/``else``/``finally``. - - A lambda can be named, see ``unpythonic.misc.namelambda``. + - The expr macro `cond[]` gives us a general `if`/`elif`/`else` expression. + - Without it, the expression form of `if` (that Python already has) could be used, but readability suffers if nested, since it has no `elif`. Actually, [`and` and `or` are sufficient for full generality](https://www.ibm.com/developerworks/library/l-prog/), but readability suffers even more. + - So we use macros to define a `cond` expression, essentially duplicating a feature the language already almost has. See [our macros](macros.md). + - Functional looping (with TCO) gives us equivalents of `for` and `while`. See the constructs in `unpythonic.fploop`, particularly `looped` and `breakably_looped`. + - `unpythonic.ec.call_ec` gives us `return` (the ec). + - `unpythonic.misc.raisef` gives us `raise`, and `unpythonic.misc.tryf` gives us `try`/`except`/`else`/`finally`. + - A lambda can be named, see `unpythonic.misc.namelambda`. - There are some practical limitations on the fully qualified name of nested lambdas. - Note this does not bind the name to an identifier at the use site, so the name cannot be used to recurse. The point is that the name is available for inspection, and it will show in tracebacks. - - A lambda can recurse using ``unpythonic.fun.withself``. You will get a `self` argument that points to the lambda itself, and is passed implicitly, like `self` usually in Python. + - A lambda can recurse using `unpythonic.fun.withself`. You will get a `self` argument that points to the lambda itself, and is passed implicitly, like `self` usually in Python. - A lambda can define a class using the three-argument form of the builtin `type` function. For an example, see [Peter Corbett (2005): Statementless Python](https://gist.github.com/brool/1679908), a complete minimal Lisp interpreter implemented as a single Python expression. - A lambda can import a module using the builtin `__import__`, or better, `importlib.import_module`. - - A lambda can assert by using an if-expression and then ``raisef`` to actually raise the ``AssertionError``. + - A lambda can assert by using an if-expression and then `raisef` to actually raise the `AssertionError`. - Or use the `test[]` macro, which also shows the source code for the asserted expression if the assertion fails. - Technically, `test[]` will `signal` the `TestFailure` (part of the public API of `unpythonic.test.fixtures`), not raise it, but essentially, `test[]` is a more convenient assert that optionally hooks into a testing framework. The error signal, if unhandled, will automatically chain into raising a `ControlError` exception, which is often just fine. - - Context management (``with``) is currently **not** available for lambdas, even in ``unpythonic``. - - Aside from the `async` stuff, this is the last hold-out preventing full generality, so we will likely add an expression form of ``with`` in a future version. This is tracked in [issue #76](https://github.com/Technologicat/unpythonic/issues/76). + - Context management (`with`) is currently **not** available for lambdas, even in `unpythonic`. + - Aside from the `async` stuff, this is the last hold-out preventing full generality, so we will likely add an expression form of `with` in a future version. This is tracked in [issue #76](https://github.com/Technologicat/unpythonic/issues/76). Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. -## On ``let`` and Python +## On `let` and Python Why no `let*`, as a function? In Python, name lookup always occurs at runtime. Python gives us no compile-time guarantees that no binding refers to a later one - in [Racket](http://racket-lang.org/), this guarantee is the main difference between `let*` and `letrec`. @@ -147,9 +147,9 @@ In contrast, in a `let*` form, attempting such a definition is *a compile-time e Our `letrec` behaves like `let*` in that if `valexpr` is not a function, it may only refer to bindings above it. But this is only enforced at run time, and we allow mutually recursive function definitions, hence `letrec`. -Note the function versions of our `let` constructs, in the pure-Python API, are **not** properly lexically scoped; in case of nested ``let`` expressions, one must be explicit about which environment the names come from. +Note the function versions of our `let` constructs, in the pure-Python API, are **not** properly lexically scoped; in case of nested `let` expressions, one must be explicit about which environment the names come from. -The [macro versions](macros.md) of the `let` constructs **are** lexically scoped. The macros also provide a ``letseq[]`` that, similarly to Racket's ``let*``, gives a compile-time guarantee that no binding refers to a later one. +The [macro versions](macros.md) of the `let` constructs **are** lexically scoped. The macros also provide a `letseq[]` that, similarly to Racket's `let*`, gives a compile-time guarantee that no binding refers to a later one. Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). @@ -158,7 +158,7 @@ Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [ Why the clunky `e.set("foo", newval)` or `e << ("foo", newval)`, which do not directly mention `e.foo`? This is mainly because in Python, the language itself is not customizable. If we could define a new operator `e.foo newval` to transform to `e.set("foo", newval)`, this would be easily solved. -Our [macros](macros.md) essentially do exactly this, but by borrowing the ``<<`` operator to provide the syntax ``foo << newval``, because even with macros, it is not possible to define new [BinOp](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#BinOp)s in Python. That **is** possible essentially as a *reader macro* (as it's known in the Lisp world), to transform custom BinOps into some syntactically valid Python code before proceeding with the rest of the import machinery, but it seems as of this writing, no one has done this. +Our [macros](macros.md) essentially do exactly this, but by borrowing the `<<` operator to provide the syntax `foo << newval`, because even with macros, it is not possible to define new [BinOp](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#BinOp)s in Python. That **is** possible essentially as a *reader macro* (as it's known in the Lisp world), to transform custom BinOps into some syntactically valid Python code before proceeding with the rest of the import machinery, but it seems as of this writing, no one has done this. If you want a framework to play around with reader macros in Python, see [`mcpyrate`](https://github.com/Technologicat/mcpyrate). You'll still have to write a parser, where [Pyparsing](https://github.com/pyparsing/pyparsing) may help; but supporting something as complex as a customized version of the surface syntax of Python is still a lot of work, and may quickly go out of date. (You'll want to look at the official [full grammar specification](https://docs.python.org/3/reference/grammar.html), as well as the source code linked therein.) @@ -176,32 +176,32 @@ The current solution for the assignment syntax issue is to use macros, to have b ## TCO syntax and speed -Benefits and costs of ``return jump(...)``: +Benefits and costs of `return jump(...)`: - - Explicitly a tail call due to ``return``. - - The trampoline can be very simple and (relatively speaking) fast. Just a dumb ``jump`` record, a ``while`` loop, and regular function calls and returns. - - The cost is that ``jump`` cannot detect whether the user forgot the ``return``, leaving a possibility for bugs in the client code (causing an FP loop to immediately exit, returning ``None``). Unit tests of client code become very important. + - Explicitly a tail call due to `return`. + - The trampoline can be very simple and (relatively speaking) fast. Just a dumb `jump` record, a `while` loop, and regular function calls and returns. + - The cost is that `jump` cannot detect whether the user forgot the `return`, leaving a possibility for bugs in the client code (causing an FP loop to immediately exit, returning `None`). Unit tests of client code become very important. - This is somewhat mitigated by the check in `__del__`, but it can only print a warning, not stop the incorrect program from proceeding. - - We could mandate that trampolined functions must not return ``None``, but: - - Uniformity is lost between regular and trampolined functions, if only one kind may return ``None``. + - We could mandate that trampolined functions must not return `None`, but: + - Uniformity is lost between regular and trampolined functions, if only one kind may return `None`. - This breaks the *don't care about return value* use case, which is rather common when using side effects. - - Failing to terminate at the intended point may well fall through into what was intended as another branch of the client code, which may correctly have a ``return``. So this would not even solve the problem. + - Failing to terminate at the intended point may well fall through into what was intended as another branch of the client code, which may correctly have a `return`. So this would not even solve the problem. -The other simple-ish solution is to use exceptions, making the jump wrest control from the caller. Then ``jump(...)`` becomes a verb, but this approach is 2-5x slower, when measured with a do-nothing loop. (See the old default TCO implementation in v0.9.2.) +The other simple-ish solution is to use exceptions, making the jump wrest control from the caller. Then `jump(...)` becomes a verb, but this approach is 2-5x slower, when measured with a do-nothing loop. (See the old default TCO implementation in v0.9.2.) -Our [macros](macros.md) provide an easy-to use solution. Just wrap the relevant section of code in a ``with tco:``, to automatically apply TCO to code that looks exactly like standard Python. With the macro, function definitions (also lambdas) and returns are automatically converted. It also knows enough not to add a ``@trampolined`` if you have already declared a ``def`` as ``@looped`` (or any of the other TCO-enabling decorators in ``unpythonic.fploop``, or ``unpythonic.fix.fixtco``). +Our [macros](macros.md) provide an easy-to use solution. Just wrap the relevant section of code in a `with tco:`, to automatically apply TCO to code that looks exactly like standard Python. With the macro, function definitions (also lambdas) and returns are automatically converted. It also knows enough not to add a `@trampolined` if you have already declared a `def` as `@looped` (or any of the other TCO-enabling decorators in `unpythonic.fploop`, or `unpythonic.fix.fixtco`). For other libraries bringing TCO to Python, see: - [tco](https://github.com/baruchel/tco) by Thomas Baruchel, based on exceptions. - - [ActiveState recipe 474088](https://github.com/ActiveState/code/tree/master/recipes/Python/474088_Tail_Call_Optimization_Decorator), based on ``inspect``. - - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. + - [ActiveState recipe 474088](https://github.com/ActiveState/code/tree/master/recipes/Python/474088_Tail_Call_Optimization_Decorator), based on `inspect`. + - `recur.tco` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. + - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to `fn.py`. ## No Monads? -(Beside List inside ``forall``.) +(Beside List inside `forall`.) Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done elsewhere, see [OSlash](https://github.com/dbrattli/OSlash) if you need them. @@ -240,52 +240,52 @@ More on type systems: ## Detailed Notes on Macros - - ``continuations`` and ``tco`` are mutually exclusive, since ``continuations`` already implies TCO. - - However, the ``tco`` macro skips any ``with continuations`` blocks inside it, **for the specific reason** of allowing modules written in the [Lispython dialect](https://github.com/Technologicat/pydialect) (which implies TCO for the whole module) to use ``with continuations``. + - `continuations` and `tco` are mutually exclusive, since `continuations` already implies TCO. + - However, the `tco` macro skips any `with continuations` blocks inside it, **for the specific reason** of allowing modules written in the [Lispython dialect](https://github.com/Technologicat/pydialect) (which implies TCO for the whole module) to use `with continuations`. - - ``prefix``, ``autoreturn``, ``quicklambda`` and ``multilambda`` expand outside-in, because they change the semantics: - - ``prefix`` transforms things-that-look-like-tuples into function calls, - - ``autoreturn`` adds ``return`` statements where there weren't any, - - ``quicklambda`` transforms things-that-look-like-list-lookups into ``lambda`` function definitions, - - ``multilambda`` transforms things-that-look-like-lists (in the body of a ``lambda``) into sequences of multiple expressions, using ``do[]``. + - `prefix`, `autoreturn`, `quicklambda` and `multilambda` expand outside-in, because they change the semantics: + - `prefix` transforms things-that-look-like-tuples into function calls, + - `autoreturn` adds `return` statements where there weren't any, + - `quicklambda` transforms things-that-look-like-list-lookups into `lambda` function definitions, + - `multilambda` transforms things-that-look-like-lists (in the body of a `lambda`) into sequences of multiple expressions, using `do[]`. - Hence, a lexically outer block of one of these types *will expand first*, before any macros inside it are expanded. - This yields clean, standard-ish Python for the rest of the macros, which then don't need to worry about their input meaning something completely different from what it looks like. - - An already expanded ``do[]`` (including that inserted by `multilambda`) is accounted for by all ``unpythonic.syntax`` macros when handling expressions. + - An already expanded `do[]` (including that inserted by `multilambda`) is accounted for by all `unpythonic.syntax` macros when handling expressions. - For simplicity, this is **the only** type of sequencing understood by the macros. - - E.g. the more rudimentary ``unpythonic.seq.begin`` is not treated as a sequencing operation. This matters especially in ``tco``, where it is critically important to correctly detect a tail position in a return-value expression or (multi-)lambda body. + - E.g. the more rudimentary `unpythonic.seq.begin` is not treated as a sequencing operation. This matters especially in `tco`, where it is critically important to correctly detect a tail position in a return-value expression or (multi-)lambda body. - *Sequencing* is here meant in the Racket/Haskell sense of *running sub-operations in a specified order*, unrelated to Python's *sequences*. - - The TCO transformation knows about TCO-enabling decorators provided by ``unpythonic``, and adds the ``@trampolined`` decorator to a function definition only when it is not already TCO'd. - - This applies also to lambdas; they are decorated by directly wrapping them with a call: ``trampolined(lambda ...: ...)``. - - This allows ``with tco`` to work together with the functions in ``unpythonic.fploop``, which imply TCO. + - The TCO transformation knows about TCO-enabling decorators provided by `unpythonic`, and adds the `@trampolined` decorator to a function definition only when it is not already TCO'd. + - This applies also to lambdas; they are decorated by directly wrapping them with a call: `trampolined(lambda ...: ...)`. + - This allows `with tco` to work together with the functions in `unpythonic.fploop`, which imply TCO. - - Macros that transform lambdas (notably ``continuations`` and ``tco``): + - Macros that transform lambdas (notably `continuations` and `tco`): - Perform an outside-in pass to take note of all lambdas that appear in the code *before the expansion of any inner macros*. Then in an inside-out pass, *after the expansion of all inner macros*, only the recorded lambdas are transformed. - This mechanism distinguishes between explicit lambdas in the client code, and internal implicit lambdas automatically inserted by a macro. The latter are a technical detail that should not undergo the same transformations as user-written explicit lambdas. - - The identification is based on the ``id`` of the AST node instance. Hence, if you plan to write your own macros that work together with those in ``unpythonic.syntax``, avoid going overboard with FP. Modifying the tree in-place, preserving the original AST node instances as far as sensible, is just fine. - - For the interested reader, grep the source code for ``userlambdas``. - - Support a limited form of *decorated lambdas*, i.e. trees of the form ``f(g(h(lambda ...: ...)))``. + - The identification is based on the `id` of the AST node instance. Hence, if you plan to write your own macros that work together with those in `unpythonic.syntax`, avoid going overboard with FP. Modifying the tree in-place, preserving the original AST node instances as far as sensible, is just fine. + - For the interested reader, grep the source code for `userlambdas`. + - Support a limited form of *decorated lambdas*, i.e. trees of the form `f(g(h(lambda ...: ...)))`. - The macros will reorder a chain of lambda decorators (i.e. nested calls) to use the correct ordering, when only known decorators are used on a literal lambda. - - This allows some combos such as ``tco``, ``unpythonic.fploop.looped``, ``autocurry``. - - Only decorators provided by ``unpythonic`` are recognized, and only some of them are supported. For details, see ``unpythonic.regutil``. - - If you need to combo ``unpythonic.fploop.looped`` and ``unpythonic.ec.call_ec``, use ``unpythonic.fploop.breakably_looped``, which does exactly that. - - The problem with a direct combo is that the required ordering is the trampoline (inside ``looped``) outermost, then ``call_ec``, and then the actual loop, but because an escape continuation is only valid for the dynamic extent of the ``call_ec``, the whole loop must be run inside the dynamic extent of the ``call_ec``. - - ``unpythonic.fploop.breakably_looped`` internally inserts the ``call_ec`` at the right step, and gives you the ec as ``brk``. - - For the interested reader, look at ``unpythonic.syntax.util``. + - This allows some combos such as `tco`, `unpythonic.fploop.looped`, `autocurry`. + - Only decorators provided by `unpythonic` are recognized, and only some of them are supported. For details, see `unpythonic.regutil`. + - If you need to combo `unpythonic.fploop.looped` and `unpythonic.ec.call_ec`, use `unpythonic.fploop.breakably_looped`, which does exactly that. + - The problem with a direct combo is that the required ordering is the trampoline (inside `looped`) outermost, then `call_ec`, and then the actual loop, but because an escape continuation is only valid for the dynamic extent of the `call_ec`, the whole loop must be run inside the dynamic extent of the `call_ec`. + - `unpythonic.fploop.breakably_looped` internally inserts the `call_ec` at the right step, and gives you the ec as `brk`. + - For the interested reader, look at `unpythonic.syntax.util`. - - ``namedlambda`` is a two-pass macro. In the outside-in pass, it names lambdas inside ``let[]`` expressions before they are expanded away. The inside-out pass of ``namedlambda`` must run after ``autocurry`` to analyze and transform the auto-curried code produced by ``with autocurry``. + - `namedlambda` is a two-pass macro. In the outside-in pass, it names lambdas inside `let[]` expressions before they are expanded away. The inside-out pass of `namedlambda` must run after `autocurry` to analyze and transform the auto-curried code produced by `with autocurry`. - - ``autoref`` does not need in its output to be curried (hence after ``autocurry`` to gain some performance), but needs to run before ``lazify``, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what ``namedlambda`` does, so it does not matter in which exact order these two run. + - `autoref` does not need in its output to be curried (hence after `autocurry` to gain some performance), but needs to run before `lazify`, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what `namedlambda` does, so it does not matter in which exact order these two run. - - ``lazify`` is a rather invasive rewrite that needs to see the output from most of the other macros. + - `lazify` is a rather invasive rewrite that needs to see the output from most of the other macros. - - ``envify`` needs to see the output of ``lazify`` in order to shunt function args into an unpythonic ``env`` without triggering the implicit forcing. + - `envify` needs to see the output of `lazify` in order to shunt function args into an unpythonic `env` without triggering the implicit forcing. - - With MacroPy, it used to be so that some of the block macros could be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``autocurry`` and ``namedlambda``) required their own ``with`` statement. In `mcpyrate`, block macros can be comboed in the same ``with`` statement (and expansion order is *left-to-right*). + - With MacroPy, it used to be so that some of the block macros could be comboed as multiple context managers in the same `with` statement (expansion order is then *left-to-right*), whereas some (notably `autocurry` and `namedlambda`) required their own `with` statement. In `mcpyrate`, block macros can be comboed in the same `with` statement (and expansion order is *left-to-right*). - See the relevant [issue report](https://github.com/azazel75/macropy/issues/21) and [PR](https://github.com/azazel75/macropy/pull/22). - - When in doubt, you can use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. In ``mcpyrate``, this is almost equivalent to having the macros invoked in a single ``with`` statement, in the same order. - - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a ``with step_expansion:`` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. (And of course, while testing, try to keep the input as simple as possible.) + - When in doubt, you can use a separate `with` statement for each block macro that applies to the same section of code, and nest the blocks. In `mcpyrate`, this is almost equivalent to having the macros invoked in a single `with` statement, in the same order. + - Load the macro expansion debug utility `from mcpyrate.debug import macros, step_expansion`, and put a `with step_expansion:` around your use site. Then add your macro invocations one by one, and make sure the expansion looks like what you intended. (And of course, while testing, try to keep the input as simple as possible.) ## Miscellaneous notes diff --git a/doc/dialects.md b/doc/dialects.md index 3443416c..4a753df7 100644 --- a/doc/dialects.md +++ b/doc/dialects.md @@ -36,6 +36,6 @@ As examples of what can be done with a dialects system together with a kitchen-s - [**Listhell**: It's not Lisp, it's not Python, it's not Haskell](dialects/listhell.md) - [**Pytkell**: Because it's good to have a kell](dialects/pytkell.md) -All three dialects support `unpythonic`'s ``continuations`` block macro, to add ``call/cc`` to the language; but it is not enabled automatically. +All three dialects support `unpythonic`'s `continuations` block macro, to add `call/cc` to the language; but it is not enabled automatically. Mostly, these dialects are intended as a cross between teaching material and a (fully functional!) practical joke, but Lispython may occasionally come in handy. diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 21ccf599..4cf6d909 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -72,14 +72,14 @@ assert ll(1, 2, 3) == llist((1, 2, 3)) ## Features -In terms of ``unpythonic.syntax``, we implicitly enable ``autoreturn``, ``tco``, ``multilambda``, ``namedlambda``, and ``quicklambda`` for the whole module: +In terms of `unpythonic.syntax`, we implicitly enable `autoreturn`, `tco`, `multilambda`, `namedlambda`, and `quicklambda` for the whole module: - - In tail position, the ``return`` keyword can be omitted, like in Lisps. + - In tail position, the `return` keyword can be omitted, like in Lisps. - In a `def`, the last statement at the top level of the `def` is in tail position. - - If the tail position contains an expression, a ``return`` will be automatically injected, with that expression as the return value. + - If the tail position contains an expression, a `return` will be automatically injected, with that expression as the return value. - It is still legal to use `return` whenever you would in Python; this just makes the `return` keyword non-mandatory in places where a Lisp would not require it. - To be technically correct, Schemers and Racketeers should read this as, *"in places where a Lisp would not require explicitly invoking an escape continuation"*. - - Automatic tail-call optimization (TCO) for both ``def`` and ``lambda``. + - Automatic tail-call optimization (TCO) for both `def` and `lambda`. - In a `def`, the last statement at the top level of the `def` is in tail position. - Tail positions *inside an expression* that itself appears in tail position are: - Both the `body` and `orelse` branches of an if-expression. (Exactly one of them runs, hence both are in tail position.) @@ -88,13 +88,13 @@ In terms of ``unpythonic.syntax``, we implicitly enable ``autoreturn``, ``tco``, - The last item of a `do[]`. - The last item of an implicit `do[]` in a `let[]` where the body uses the extra bracket syntax. (All `let` constructs provided by `unpythonic.syntax` are supported.) - For the gritty details, see the syntax transformer `_transform_retexpr` in [`unpythonic.syntax.tailtools`](../../unpythonic/syntax/tailtools.py). - - Multiple-expression lambdas, using bracket syntax, for example ``lambda x: [expr0, ...]``. + - Multiple-expression lambdas, using bracket syntax, for example `lambda x: [expr0, ...]`. - Brackets denote a multiple-expression lambda body. Technically, the brackets create a `do[]` environment. - If you want your lambda to have one expression that is a literal list, double the brackets: `lambda x: [[5 * x]]`. - Lambdas are automatically named whenever the machinery can figure out a name from the surrounding context. - When not, source location is auto-injected into the name. -The multi-expression lambda syntax uses ``do[]``, so it also allows lambdas to manage local variables using ``local[name << value]`` and ``delete[name]``. See the documentation of ``do[]`` for details. +The multi-expression lambda syntax uses `do[]`, so it also allows lambdas to manage local variables using `local[name << value]` and `delete[name]`. See the documentation of `do[]` for details. If you need more stuff, `unpythonic` is effectively the standard library of Lispython, on top of what Python itself already provides. @@ -118,17 +118,17 @@ The main point of `Lispy`, compared to plain Python, is automatic TCO. The abili In the `Lispython` variant, we implicitly import some macros and functions to serve as dialect builtins, keeping in line with expectations for a ~language in the~ *somewhat distant relative of the* Lisp family: - - ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil``, ``prod``. - - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax``. + - `cons`, `car`, `cdr`, `ll`, `llist`, `nil`, `prod`. + - All `let[]` and `do[]` constructs from `unpythonic.syntax`. - The underscore: e.g. `fn[_ * 3]` becomes `lambda x: x * 3`, and `fn[_ * _]` becomes `lambda x, y: x * y`. - - ``dyn``, for dynamic assignment. - - ``Values``, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, `unfold`, `iterate`, the `pipe` family, the `compose` family, and the `with continuations` macro.) + - `dyn`, for dynamic assignment. + - `Values`, for returning multiple values and/or named return values. (This ties in to `unpythonic`'s function composition subsystem, e.g. `curry`, `unfold`, `iterate`, the `pipe` family, the `compose` family, and the `with continuations` macro.) -For detailed documentation of the language features, see [``unpythonic.syntax``](../macros.md), especially the macros ``tco``, ``autoreturn``, ``multilambda``, ``namedlambda``, ``quicklambda``, ``let`` and ``do``. +For detailed documentation of the language features, see [`unpythonic.syntax`](../macros.md), especially the macros `tco`, `autoreturn`, `multilambda`, `namedlambda`, `quicklambda`, `let` and `do`. -The dialect builtin ``let[]`` constructs are ``let``, ``letseq``, ``letrec``, the decorator versions ``dlet``, ``dletseq``, ``dletrec``, the block versions (decorator, call immediately, replace def'd name with result) ``blet``, ``bletseq``, ``bletrec``, and the code-splicing variants ``let_syntax`` and ``abbrev``. Bindings may be made using any syntax variant supported by ``unpythonic.syntax``. +The dialect builtin `let[]` constructs are `let`, `letseq`, `letrec`, the decorator versions `dlet`, `dletseq`, `dletrec`, the block versions (decorator, call immediately, replace def'd name with result) `blet`, `bletseq`, `bletrec`, and the code-splicing variants `let_syntax` and `abbrev`. Bindings may be made using any syntax variant supported by `unpythonic.syntax`. -The dialect builtin ``do[]`` constructs are ``do`` and ``do0``. +The dialect builtin `do[]` constructs are `do` and `do0`. ## What Lispython is @@ -148,41 +148,41 @@ Performance is only a secondary concern; performance-critical parts fare better The aforementioned block macros are enabled implicitly for the whole module; this is the essence of the Lispython dialect. Other block macros can still be invoked manually in the user code. -Of the other block macros in ``unpythonic.syntax``, code written in Lispython supports only ``continuations``. ``autoref`` should also be harmless enough (will expand too early, but shouldn't matter). +Of the other block macros in `unpythonic.syntax`, code written in Lispython supports only `continuations`. `autoref` should also be harmless enough (will expand too early, but shouldn't matter). -``prefix``, ``autocurry``, ``lazify`` and ``envify`` are **not compatible** with the ordering of block macros implicit in the Lispython dialect. +`prefix`, `autocurry`, `lazify` and `envify` are **not compatible** with the ordering of block macros implicit in the Lispython dialect. -``prefix`` is an outside-in macro that should expand first, so it should be placed in a lexically outer position with respect to the ones Lispython invokes implicitly; but nothing can be more outer than the dialect template. +`prefix` is an outside-in macro that should expand first, so it should be placed in a lexically outer position with respect to the ones Lispython invokes implicitly; but nothing can be more outer than the dialect template. The other three are inside-out macros that should expand later, so similarly, also they should be placed in a lexically outer position. -Basically, any block macro that can be invoked *lexically inside* a ``with tco`` block will work, the rest will not. +Basically, any block macro that can be invoked *lexically inside* a `with tco` block will work, the rest will not. -If you need e.g. a lazy Lispython, the way to do that is to make a copy of the dialect module, change the dialect template to import the ``lazify`` macro, and then include a ``with lazify`` in the appropriate position, outside the ``with namedlambda`` block. Other customizations can be made similarly. +If you need e.g. a lazy Lispython, the way to do that is to make a copy of the dialect module, change the dialect template to import the `lazify` macro, and then include a `with lazify` in the appropriate position, outside the `with namedlambda` block. Other customizations can be made similarly. ## Lispython and continuations (call/cc) -Just use ``with continuations`` from ``unpythonic.syntax`` where needed. See its documentation for usage. +Just use `with continuations` from `unpythonic.syntax` where needed. See its documentation for usage. -Lispython works with ``with continuations``, because: +Lispython works with `with continuations`, because: - - Nesting ``with continuations`` within a ``with tco`` block is allowed, for the specific reason of supporting continuations in Lispython. + - Nesting `with continuations` within a `with tco` block is allowed, for the specific reason of supporting continuations in Lispython. - The dialect's implicit ``with tco`` will just skip the ``with continuations`` block (``continuations`` implies TCO). + The dialect's implicit `with tco` will just skip the `with continuations` block (`continuations` implies TCO). - - ``autoreturn``, ``quicklambda`` and ``multilambda`` are outside-in macros, so although they will be in a lexically outer position with respect to the manually invoked ``with continuations`` in the user code, this is correct (because being on the outside, they run before ``continuations``, as they should). + - `autoreturn`, `quicklambda` and `multilambda` are outside-in macros, so although they will be in a lexically outer position with respect to the manually invoked `with continuations` in the user code, this is correct (because being on the outside, they run before `continuations`, as they should). - - The same applies to the outside-in pass of ``namedlambda``. Its inside-out pass, on the other hand, must come after ``continuations``, which it does, since the dialect's implicit ``with namedlambda`` is in a lexically outer position with respect to the ``with continuations``. + - The same applies to the outside-in pass of `namedlambda`. Its inside-out pass, on the other hand, must come after `continuations`, which it does, since the dialect's implicit `with namedlambda` is in a lexically outer position with respect to the `with continuations`. -Be aware, though, that the combination of the ``autoreturn`` implicit in the dialect and ``with continuations`` might have usability issues, because ``continuations`` handles tail calls specially (the target of a tail-call in a ``continuations`` block must be continuation-enabled; see the documentation of ``continuations``), and ``autoreturn`` makes it visually slightly less clear which positions are in fact tail calls (since no explicit ``return``). Also, the top level of a ``with continuations`` block may not use ``return`` - while Lispython's implicit ``autoreturn`` happily auto-injects a ``return`` to whatever is the last statement in any particular function. +Be aware, though, that the combination of the `autoreturn` implicit in the dialect and `with continuations` might have usability issues, because `continuations` handles tail calls specially (the target of a tail-call in a `continuations` block must be continuation-enabled; see the documentation of `continuations`), and `autoreturn` makes it visually slightly less clear which positions are in fact tail calls (since no explicit `return`). Also, the top level of a `with continuations` block may not use `return` - while Lispython's implicit `autoreturn` happily auto-injects a `return` to whatever is the last statement in any particular function. ## Why extend Python? [Racket](https://racket-lang.org/) is an excellent Lisp, especially with [sweet](https://docs.racket-lang.org/sweet/), sweet expressions [[1]](https://sourceforge.net/projects/readable/) [[2]](https://srfi.schemers.org/srfi-110/srfi-110.html) [[3]](https://srfi.schemers.org/srfi-105/srfi-105.html), not to mention extremely pythonic. The word is *rackety*; the syntax of the language comes with an air of Zen minimalism (as perhaps expected of a descendant of Scheme), but the focus on *batteries included* and understandability are remarkably similar to the pythonic ideal. Racket even has an IDE (DrRacket) and an equivalent of PyPI, and the documentation is simply stellar. -Python, on the other hand, has a slight edge in usability to the end-user programmer, and importantly, a huge ecosystem of libraries, second to ``None``. Python is where science happens (unless you're in CS). Python is an almost-Lisp that has delivered on [the productivity promise](http://paulgraham.com/icad.html) of Lisp. Python also gets many things right, such as well developed support for lazy sequences, and decorators. +Python, on the other hand, has a slight edge in usability to the end-user programmer, and importantly, a huge ecosystem of libraries, second to `None`. Python is where science happens (unless you're in CS). Python is an almost-Lisp that has delivered on [the productivity promise](http://paulgraham.com/icad.html) of Lisp. Python also gets many things right, such as well developed support for lazy sequences, and decorators. In certain other respects, Python the base language leaves something to be desired, if you have been exposed to Racket (or Haskell, but that's a different story). Writing macros is harder due to the irregular syntax, but thankfully macro expanders already exist, and any set of macros only needs to be created once. @@ -232,9 +232,9 @@ def foo(n): This is rather clean, but still needs the `nonlocal` declaration, which is a statement. -If we abbreviate ``accumulate`` as a lambda, it needs a ``let`` environment to write in, to use `unpythonic`'s expression-assignment (`name << value`). +If we abbreviate `accumulate` as a lambda, it needs a `let` environment to write in, to use `unpythonic`'s expression-assignment (`name << value`). -But see ``envify`` in ``unpythonic.syntax``, which shallow-copies function arguments into an `env` implicitly: +But see `envify` in `unpythonic.syntax`, which shallow-copies function arguments into an `env` implicitly: ```python from unpythonic.syntax import macros, envify @@ -251,9 +251,9 @@ with envify: foo = lambda n: lambda i: n << n + i ``` -``envify`` is not part of the Lispython dialect definition, because this particular, perhaps rarely used, feature is not really worth a global performance hit whenever a function is entered. +`envify` is not part of the Lispython dialect definition, because this particular, perhaps rarely used, feature is not really worth a global performance hit whenever a function is entered. -Note that ``envify`` is **not** compatible with Lispython, because it would need to appear in a lexically outer position compared to macros already invoked by the dialect template. If you need an envified Lispython, copy `unpythonic/dialects/lispython.py` and modify the template therein. [The xmas tree combo](../macros.md#the-xmas-tree-combo) says `envify` should come lexically after `multilambda`, but before `namedlambda`. +Note that `envify` is **not** compatible with Lispython, because it would need to appear in a lexically outer position compared to macros already invoked by the dialect template. If you need an envified Lispython, copy `unpythonic/dialects/lispython.py` and modify the template therein. [The xmas tree combo](../macros.md#the-xmas-tree-combo) says `envify` should come lexically after `multilambda`, but before `namedlambda`. ## CAUTION diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index b5e8f441..f2e018c9 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -47,16 +47,16 @@ assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ## Features -In terms of ``unpythonic.syntax``, we implicitly enable ``prefix`` and ``curry`` for the whole module. +In terms of `unpythonic.syntax`, we implicitly enable `prefix` and `curry` for the whole module. The following are dialect builtins: - - ``apply``, aliased to ``unpythonic.fun.apply`` - - ``compose``, aliased to unpythonic's currying right-compose ``composerc`` - - ``q``, ``u``, ``kw`` for the prefix syntax (note these are not `mcpyrate`'s - ``q`` and ``u``, but those from `unpythonic.syntax`, specifically for ``prefix``) + - `apply`, aliased to `unpythonic.fun.apply` + - `compose`, aliased to unpythonic's currying right-compose `composerc` + - `q`, `u`, `kw` for the prefix syntax (note these are not `mcpyrate`'s + `q` and `u`, but those from `unpythonic.syntax`, specifically for `prefix`) -For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md). +For detailed documentation of the language features, see [`unpythonic.syntax`](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md). If you need more stuff, `unpythonic` is effectively the standard library of Listhell, on top of what Python itself already provides. @@ -72,7 +72,7 @@ It's also a minimal example of how to make an AST-transforming dialect. ## Comboability -Only outside-in macros that should expand after ``autocurry`` (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``autocurry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Listhell dialect. +Only outside-in macros that should expand after `autocurry` (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before `autocurry` (there are two, namely `tco` and `continuations`) can be used in programs written in the Listhell dialect. ## Notes diff --git a/doc/dialects/pytkell.md b/doc/dialects/pytkell.md index f1325d9f..7025b3bf 100644 --- a/doc/dialects/pytkell.md +++ b/doc/dialects/pytkell.md @@ -70,29 +70,29 @@ assert x == 42 ## Features -In terms of ``unpythonic.syntax``, we implicitly enable ``autocurry`` and ``lazify`` for the whole module. +In terms of `unpythonic.syntax`, we implicitly enable `autocurry` and `lazify` for the whole module. We also import some macros and functions to serve as dialect builtins: - - All ``let[]`` and ``do[]`` constructs from ``unpythonic.syntax`` - - ``lazy[]`` and ``lazyrec[]`` for manual lazification of atoms and data structure literals, respectively - - If-elseif-else expression ``cond[]`` - - Nondeterministic evaluation ``forall[]`` (do-notation in the List monad) - - Function composition, ``compose`` (like Haskell's ``.`` operator), aliased to `unpythonic`'s currying right-compose ``composerc`` - - Linked list utilities ``cons``, ``car``, ``cdr``, ``ll``, ``llist``, ``nil`` - - Folds and scans ``foldl``, ``foldr``, ``scanl``, ``scanr`` - - Memoization ``memoize``, ``gmemoize``, ``imemoize``, ``fimemoize`` - - Functional updates ``fup`` and ``fupdate`` - - Immutable dict ``frozendict`` - - Mathematical sequences ``s``, ``imathify``, ``gmathify`` - - Iterable utilities ``islice`` (`unpythonic`'s version), ``take``, ``drop``, ``split_at``, ``first``, ``second``, ``nth``, ``last`` - - Function arglist reordering utilities ``flip``, ``rotate`` + - All `let[]` and `do[]` constructs from `unpythonic.syntax` + - `lazy[]` and `lazyrec[]` for manual lazification of atoms and data structure literals, respectively + - If-elseif-else expression `cond[]` + - Nondeterministic evaluation `forall[]` (do-notation in the List monad) + - Function composition, `compose` (like Haskell's `.` operator), aliased to `unpythonic`'s currying right-compose `composerc` + - Linked list utilities `cons`, `car`, `cdr`, `ll`, `llist`, `nil` + - Folds and scans `foldl`, `foldr`, `scanl`, `scanr` + - Memoization `memoize`, `gmemoize`, `imemoize`, `fimemoize` + - Functional updates `fup` and `fupdate` + - Immutable dict `frozendict` + - Mathematical sequences `s`, `imathify`, `gmathify` + - Iterable utilities `islice` (`unpythonic`'s version), `take`, `drop`, `split_at`, `first`, `second`, `nth`, `last` + - Function arglist reordering utilities `flip`, `rotate` -For detailed documentation of the language features, see [``unpythonic.syntax``](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md). +For detailed documentation of the language features, see [`unpythonic.syntax`](https://github.com/Technologicat/unpythonic/tree/master/doc/macros.md). -The builtin ``let[]`` constructs are ``let``, ``letseq``, ``letrec``, the decorator versions ``dlet``, ``dletseq``, ``dletrec``, the block versions (decorator, call immediately, replace `def`'d name with result) ``blet``, ``bletseq``, ``bletrec``. Bindings may be made using any syntax variant supported by ``unpythonic.syntax``. +The builtin `let[]` constructs are `let`, `letseq`, `letrec`, the decorator versions `dlet`, `dletseq`, `dletrec`, the block versions (decorator, call immediately, replace `def`'d name with result) `blet`, `bletseq`, `bletrec`. Bindings may be made using any syntax variant supported by `unpythonic.syntax`. -The builtin ``do[]`` constructs are ``do`` and ``do0``. +The builtin `do[]` constructs are `do` and `do0`. If you need more stuff, `unpythonic` is effectively the standard library of Pytkell, on top of what Python itself already provides. @@ -108,9 +108,9 @@ It's also a minimal example of how to make an AST-transforming dialect. ## Comboability -**Not** comboable with most of the block macros in ``unpythonic.syntax``, because ``autocurry`` and ``lazify`` appear in the dialect template, hence at the lexically outermost position. +**Not** comboable with most of the block macros in `unpythonic.syntax`, because `autocurry` and `lazify` appear in the dialect template, hence at the lexically outermost position. -Only outside-in macros that should expand after ``lazify`` has recorded its userlambdas (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before ``autocurry`` (there are two, namely ``tco`` and ``continuations``) can be used in programs written in the Pytkell dialect. +Only outside-in macros that should expand after `lazify` has recorded its userlambdas (currently, `unpythonic` provides no such macros) and inside-out macros that should expand before `autocurry` (there are two, namely `tco` and `continuations`) can be used in programs written in the Pytkell dialect. ## CAUTION diff --git a/doc/essays.md b/doc/essays.md index f865cf14..011a0a4f 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -49,7 +49,7 @@ While on the topic of usability, why are lambdas strictly anonymous? In cases wh On a point raised [here by the BDFL](https://www.artima.com/weblogs/viewpost.jsp?thread=147358), with respect to indentation-sensitive vs. indentation-insensitive parser modes; having seen [SRFI-110: Sweet-expressions (t-expressions)](https://srfi.schemers.org/srfi-110/srfi-110.html), I think Python is confusing matters by linking the parser mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. -It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) +It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose `lambda x: [expr0, expr1, ...]` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. @@ -113,7 +113,7 @@ To summarize; as someone already put it, `hoon` offers a glimpse into an alterna I think the perfect place to end this piece is to quote a few lines from the language definition [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), to give a flavor: -``` +`` ++ doos :: sleep until |= hap=path ^- (unit ,@da) (doze:(wink:(vent bud (dink (dint hap))) now 0 (beck ~)) now [hap ~]) @@ -154,7 +154,7 @@ I think the perfect place to end this piece is to quote a few lines from the lan [p.i.mor t.i.q.i.mor t.q.i.mor r.i.mor] [p.yub [[p.i.naf ves:q.yub] t.naf]] -- -``` +`` The Lisp family (particularly the Common Lisp branch) has a reputation for silly terminology, but I think `hoon` deserves the crown. All control structures are punctuation-only ASCII digraphs, and almost every name is a monosyllabic nonsense word. Still, this Lewis-Carroll-esque naming convention of making words mean what you define them to mean makes at least as much sense as the standard naming convention in mathematics, naming theorems after their discoverers! (Or at least, [after someone else](https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy).) From 18590f3794c8f10b4f92bfc73a8156477484fa7d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 16:52:11 +0300 Subject: [PATCH 523/832] 0.15.0: update notes for `s`, `imathify`, `gmathify` --- doc/features.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index 49b95060..b28d7189 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2505,9 +2505,9 @@ For convenience, we support some special cases: ### `s`, `imathify`, `gmathify`: lazy mathematical sequences with infix arithmetic -**Changed in v0.14.3.** Added convenience mode to generate cyclic infinite sequences. +**Changed in v0.14.3.** *Added convenience mode to generate cyclic infinite sequences.* -**Changed in v0.14.3.** To improve descriptiveness, and for consistency with names of other abstractions in `unpythonic`, `m` has been renamed `imathify` and `mg` has been renamed `gmathify`. The old names will continue working in v0.14.x, and will be removed in v0.15.0. This is a one-time change; it is not likely that these names will be changed ever again. +**Changed in v0.14.3.** *To improve descriptiveness, and for consistency with names of other abstractions in `unpythonic`, `m` has been renamed `imathify` and `mg` has been renamed `gmathify`. The old names work in v0.14.3, and have been removed in v0.15.0. This is a one-time change; it is not likely that these names will be changed ever again.* We provide a compact syntax to create lazy constant, cyclic, arithmetic, geometric and power sequences: `s(...)`. Numeric (`int`, `float`, `mpmath`) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. From fa920a86807eb7e53aea2d43307774e7a49971b7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 17:01:58 +0300 Subject: [PATCH 524/832] various small doc wording fixes --- README.md | 2 +- doc/features.md | 25 +++++++++++++------------ unpythonic/numutil.py | 1 + unpythonic/seq.py | 2 +- unpythonic/syntax/tests/test_lazify.py | 2 +- unpythonic/tests/test_numutil.py | 1 + unpythonic/tests/test_seq.py | 2 +- 7 files changed, 19 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 1c47997c..9325188f 100644 --- a/README.md +++ b/README.md @@ -491,7 +491,7 @@ from itertools import repeat from unpythonic import fup t = (1, 2, 3, 4, 5) -s = fup(t)[0::2] << tuple(repeat(10, 3)) +s = fup(t)[0::2] << repeat(10) assert s == (10, 2, 10, 4, 10) assert t == (1, 2, 3, 4, 5) ``` diff --git a/doc/features.md b/doc/features.md index b28d7189..cc382a75 100644 --- a/doc/features.md +++ b/doc/features.md @@ -468,8 +468,8 @@ To rebind existing dynvars, use `dyn.k = v`, or `dyn.update(k0=v0, ...)`. Rebind There is no `set` function or `<<` operator, unlike in the other `unpythonic` environments. -
-Each thread has its own dynamic scope stack. There is also a global dynamic scope for default values, shared between threads. +
Each thread has its own dynamic scope stack. There is also a global dynamic scope for default values, shared between threads. + A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, a PR is welcome!) @@ -770,7 +770,7 @@ We also provide an **immutable** box, `Some`. This can be useful to represent op The idea is that the value, when present, is placed into a `Some`, such as `Some(42)`, `Some("cat")`, `Some(myobject)`. Then, the situation where the value is absent can be represented as a bare `None`. So specifically, `Some(None)` means that a value is present and this value is `None`, whereas a bare `None` means that there is no value. -(It is like the `Some` constructor of a `Maybe` monad, but with no monadic magic. In this interpretation, the bare constant `None` plays the role of `Nothing`.) +It is like the `Some` constructor of a `Maybe` monad, but with no monadic magic. In this interpretation, the bare constant `None` plays the role of `Nothing`. #### `ThreadLocalBox` @@ -1292,7 +1292,7 @@ from unpythonic import lazy_piped, exitpipe fibos = [] def nextfibo(a, b): # multiple arguments allowed fibos.append(a) # store result by side effect - # New state, handed to next function in the pipe. + # New state, handed to the next function in the pipe. # As of v0.15.0, use `Values(...)` to represent multiple return values. # Positional args will be passed positionally, named ones by name. return Values(a=b, b=(a + b)) @@ -1313,7 +1313,7 @@ Things missing from the standard library. - `memoize`, with exception caching. - `curry`, with passthrough like in Haskell. - `fix`: detect and break infinite recursion cycles. **Added in v0.14.2.** - - **Added in v0.15.0.** `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. + - `partial` with run-time type checking, which helps a lot with fail-fast in code that uses partial application. This function type-checks arguments against type annotations, then delegates to `functools.partial`. Supports `unpythonic`'s `@generic` and `@typed` functions, too. **Added in v0.15.0.** - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - **Changed in v0.15.0.** *For the benefit of code using the `with lazify` macro, the compose functions are now marked lazy. Arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.* - Any number of positional and keyword arguments are supported, with the same rules as in the pipe system. Multiple return values, or named return values, represented as a `Values`, are automatically unpacked to the args and kwargs of the next function in the chain. @@ -1505,7 +1505,7 @@ Our `curry` can be used both as a decorator and as a regular function. As a deco Like Haskell, and [`spicy` for Racket](https://github.com/Technologicat/spicy), our `curry` supports *passthrough*; but we pass through **both positional and named arguments**. -Any args and/or kwargs that are incompatible with the target function's call signature, are *passed through* in the sense that the function is called, and then its return value is merged with the remaining args and kwargs. +Any args and/or kwargs that are incompatible with the target function's call signature, are *passed through* in the sense that the function is called with the args and kwargs compatible with its call signature, and then its return value is merged with the remaining args and kwargs. If the *first positional return value* of the result of passthrough is callable, it is (curried and) invoked on the remaining args and kwargs, after the merging. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). @@ -2257,7 +2257,7 @@ The only differences are the name of the decorator and `return` vs. `yield from` ### `fup`: Functional update; `ShadowedSequence` -**Changed in 0.15.0.** *Bug fixed: Now an infinite replacement sequence to pull items from is actually ok, as the documentation has always claimed.* +**Changed in v0.15.0.** *Bug fixed: Now an infinite replacement sequence to pull items from is actually ok, as the documentation has always claimed.* We provide three layers, in increasing order of the level of abstraction: `ShadowedSequence`, `fupdate`, and `fup`. @@ -2613,7 +2613,7 @@ Inspired by Haskell. **Added in v0.14.2**. -We provide **lispy symbols**, an **uninterned symbol generator**, and a **pythonic singleton abstraction**. These are all pickle-aware, and instantiation is thread-safe. +We provide **lispy symbols**, an **uninterned symbol generator**, and a **pythonic singleton abstraction**. These are all pickle-aware and thread-safe. #### Symbol @@ -2637,7 +2637,7 @@ The function `gensym` creates an ***uninterned symbol***, also known as *a gensy A gensym never conflicts with any named symbol; not even if one takes the UUID from a gensym and creates a named symbol using that as the name. -*The return value is the only time you'll see that symbol object; take good care of it!* +*The return value of `gensym` is the only time you will see that particular uninterned symbol object; take good care of it!* For example: @@ -2693,11 +2693,11 @@ As the result of answering these questions, `unpythonic`'s idea of a singleton s However, Python can easily retrieve a singleton instance with syntax that looks like regular object construction, by customizing [`__new__`](https://docs.python.org/3/reference/datamodel.html#object.__new__). Hence no static accessor method is needed. This in turn raises the question, what should we do with constructor arguments, as we surely would like to (in general) to allow those, and they can obviously differ between call sites. Since there is only one object instance to load state into, we could either silently update the state, or silently ignore the new proposed arguments. Good luck tracking down bugs either way. But upon closer inspection, that question depends on an unfounded assumption. What we should be asking instead is, *what should happen* if the constructor of a singleton is called again, while an instance already exists? -We believe in the principles of [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns) and [fail-fast](https://en.wikipedia.org/wiki/Fail-fast). The textbook singleton pattern conflates two concerns, possibly due to language limitations: the *management of object instances*, and the *enforcement of the at-most-one-instance-only guarantee*. If we wish to uncouple these responsibilities, then the obvious pythonic answer is that attempting to construct the singleton again while it already exists **should be considered a run-time error**. Since a singleton **type** does not support that operation, this situation should raise a `TypeError`. This makes the error explicit as early as possible, thus adhering to the fail-fast principle, hence making it difficult for bugs to hide (constructor arguments will either take effect, or the constructor call will explicitly fail). +We believe in the principles of [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns) and [fail-fast](https://en.wikipedia.org/wiki/Fail-fast). The textbook singleton pattern conflates two concerns, possibly due to language limitations: the *management of object instances*, and the *enforcement of the at-most-one-instance-only guarantee*. If we wish to uncouple these responsibilities, then the obvious pythonic answer is that attempting to construct the singleton again while it already exists **should be considered a run-time error**. Since a singleton **type** does not support that operation, this situation should raise a `TypeError`. This makes the error explicit as early as possible, thus adhering to the fail-fast principle, hence making it difficult for bugs to hide. Constructor arguments will either take effect, or the constructor call will explicitly fail. Another question arises due to Python having builtin support for object persistence, namely `pickle`. What *should* happen when a singleton is unpickled, while an instance of that singleton already exists? Arguably, by default, it should load the state from the pickle file into the existing instance, overwriting its current state. -(Scenario: during second and later runs, a program first initializes, which causes the singleton instance to be created, just like during the first run of that program. Then the program loads state from a pickle file, containing (among other data) the state the singleton instance was in when the program previously shut down. In this scenario, considering the singleton, the data in the file is more relevant than the defaults the program initialization feeds in. Hence the default should be to replace the state of the existing singleton instance with the data from the pickle file.) +This design is based on considering the following scenario. During second and later runs, a program first initializes, which causes the singleton instance to be created, just like during the first run of that program. Then the program loads state from a pickle file, containing (among other data) the state the singleton instance was in when the program previously shut down. In this scenario, considering the singleton, the data in the file is more relevant than the defaults the program initialization feeds in. Hence the default should be to replace the state of the existing singleton instance with the data from the pickle file. Our `Singleton` abstraction is the result of these pythonifications applied to the classic pattern. For more documentation and examples, see the unit tests in [`unpythonic/tests/test_singleton.py`](../unpythonic/tests/test_singleton.py). @@ -2721,7 +2721,7 @@ Most often, **don't**. `Singleton` is provided for the very rare occasion where Cases 1 and 2 have no meaningful instance data. Case 3 may or may not, depending on the specifics. If your object does, and if you want it to support `pickle`, you may want to customize [`__getnewargs__`](https://docs.python.org/3/library/pickle.html#object.__getnewargs__) (called *at pickling time*), [`__setstate__`](https://docs.python.org/3/library/pickle.html#object.__setstate__), and sometimes maybe also [`__getstate__`](https://docs.python.org/3/library/pickle.html#object.__getstate__). Note that unpickling skips `__init__`, and calls just `__new__` (with the "newargs") and then `__setstate__`. -I'm not completely sure if it's meaningful to provide a generic `Singleton` abstraction for Python, except for teaching purposes. Practical use cases may differ so much, and some of the implementation details of the specific singleton object (esp. related to pickling) may depend so closely on the implementation details of the singleton abstraction, that it may be easier to just roll your own singleton code when needed. If you're new to customizing this part of Python, the code we have here should at least demonstrate an approach for how to do this. +I am not completely sure if it is meaningful to provide a generic `Singleton` abstraction for Python, except for teaching purposes. Practical use cases may differ so much, and some of the implementation details of the specific singleton object (especially related to pickling) may depend so closely on the implementation details of the singleton abstraction, that it may be easier to just roll your own singleton code when needed. If you are new to customizing this part of Python, the code we have here should at least demonstrate how to do that. ## Control flow tools @@ -4335,6 +4335,7 @@ c = fixpoint(cos, x0=1) # Actually "Newton's" algorithm for the square root was already known to the # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) +# Concerning naming, see also https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy def sqrt_newton(n): def sqrt_iter(x): # has an attractive fixed point at sqrt(n) return (x + n / x) / 2 diff --git a/unpythonic/numutil.py b/unpythonic/numutil.py index 72df5982..3f6defef 100644 --- a/unpythonic/numutil.py +++ b/unpythonic/numutil.py @@ -109,6 +109,7 @@ def fixpoint(f, x0, tol=0): # Actually "Newton's" algorithm for the square root was already known to the # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) + # Concerning naming, see also https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy def sqrt_newton(n): def sqrt_iter(x): # has an attractive fixed point at sqrt(n) return (x + n / x) / 2 diff --git a/unpythonic/seq.py b/unpythonic/seq.py index 712ed8d4..3c393184 100644 --- a/unpythonic/seq.py +++ b/unpythonic/seq.py @@ -251,7 +251,7 @@ def append_succ(lis): def nextfibo(state): a, b = state fibos.append(a) # store result by side effect - return (b, a + b) # new state, handed to next function in the pipe + return (b, a + b) # new state, handed to the next function in the pipe p = lazy_piped1((1, 1)) # load initial state into a lazy pipe for _ in range(10): # set up pipeline p = p | nextfibo diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index f24bc3d9..1aafaf02 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -589,7 +589,7 @@ def append_succ(lis): def nextfibo(state): a, b = state fibos.append(a) # store result by side effect - return (b, a + b) # new state, handed to next function in the pipe + return (b, a + b) # new state, handed to the next function in the pipe p = lazy_piped1((1, 1)) # load initial state into a lazy pipe for _ in range(10): # set up pipeline p = p | nextfibo diff --git a/unpythonic/tests/test_numutil.py b/unpythonic/tests/test_numutil.py index 34e9d32d..984d0bc3 100644 --- a/unpythonic/tests/test_numutil.py +++ b/unpythonic/tests/test_numutil.py @@ -47,6 +47,7 @@ def runtests(): # Actually "Newton's" algorithm for the square root was already known to the # ancient Babylonians, ca. 2000 BCE. (Carl Boyer: History of mathematics) + # Concerning naming, see also https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy def sqrt_newton(n): def sqrt_iter(x): # has an attractive fixed point at sqrt(n) return (x + n / x) / 2 diff --git a/unpythonic/tests/test_seq.py b/unpythonic/tests/test_seq.py index 03ea273a..4e127188 100644 --- a/unpythonic/tests/test_seq.py +++ b/unpythonic/tests/test_seq.py @@ -118,7 +118,7 @@ def append_succ(lis): def nextfibo(state): a, b = state fibos.append(a) # store result by side effect - return (b, a + b) # new state, handed to next function in the pipe + return (b, a + b) # new state, handed to the next function in the pipe p = lazy_piped1((1, 1)) # load initial state into a lazy pipe for _ in range(10): # set up pipeline p = p | nextfibo From 3d313d2f4d1e4309c9bf4662d4f57781232f3c31 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 17:14:30 +0300 Subject: [PATCH 525/832] 0.15.0: wording/styling of sym/symbol/singleton docs --- doc/features.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/features.md b/doc/features.md index cc382a75..54ef147e 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2627,13 +2627,13 @@ assert cat is sym("cat") assert cat is not sym("dog") ``` -The constructor `sym` produces an ***interned symbol***. Whenever (in the same process) **the same name** is passed to the `sym` constructor, it gives **the same object instance**. Even unpickling a symbol that has the same name produces the same `sym` object instance as any other `sym` with that name. +The constructor `sym` produces an ***interned symbol***. Whenever, in the same process, **the same name** is passed to the `sym` constructor, it gives **the same object instance**. Even unpickling a symbol that has the same name produces the same `sym` object instance as any other `sym` with that name. Thus a `sym` behaves like a Lisp symbol. Technically speaking, it's like a zen-minimalistic [Scheme/Racket symbol](https://stackoverflow.com/questions/8846628/what-exactly-is-a-symbol-in-lisp-scheme), since Common Lisp [stuffs all sorts of additional cruft in symbols](https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node27.html). If you insist on emulating that, note a `sym` is just a Python object you could customize in the usual ways, even though its instantiation logic plays by somewhat unusual rules. #### Gensym -The function `gensym` creates an ***uninterned symbol***, also known as *a gensym*. The label given in the call to `gensym` is a short human-readable description, like the name of a named symbol, but it has no relation to object identity. Object identity is tracked by an [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier), which is automatically assigned when `gensym` creates the value. Even if `gensym` is called with the same label, the return value is a new unique symbol each time. +The function `gensym`, which is an abbreviation for *generate symbol*, creates an ***uninterned symbol***, also known as *a gensym*. The label given in the call to `gensym` is a short human-readable description, like the name of a named symbol, but it has no relation to object identity. Object identity is tracked by an [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier), which is automatically assigned when `gensym` creates the value. Even if `gensym` is called with the same label, the return value is a new unique symbol each time. A gensym never conflicts with any named symbol; not even if one takes the UUID from a gensym and creates a named symbol using that as the name. @@ -2653,7 +2653,7 @@ print(scottishfold) # gensym:cat:94287f75-02b5-4138-9174-1e422e618d59 Uninterned symbols are useful as guaranteed-unique sentinel or [nonce (sense 2, adapted to programming)](https://en.wiktionary.org/wiki/nonce#Noun) values, like the pythonic idiom `nonce = object()`, but they come with a human-readable label. -They also have a superpower: with the help of the UUID automatically assigned by `gensym`, they survive a pickle roundtrip with object identity intact. Unpickling the *same* gensym value multiple times in the same process will produce just one object instance. (If the original return value from gensym is still alive, it is that same object instance.) +They also have a superpower: with the help of the UUID automatically assigned by `gensym`, they survive a pickle roundtrip with object identity intact. Unpickling the *same* gensym value multiple times in the same process will produce just one object instance. If the original return value from gensym is still alive, it is that same object instance. The UUID is generated with the pseudo-random algorithm [`uuid.uuid4`](https://docs.python.org/3/library/uuid.html). Due to rollover of the time field, it is possible for collisions with current UUIDs (as of the early 21st century) to occur with those generated after (approximately) the year 3400. See [RFC 4122](https://tools.ietf.org/html/rfc4122). @@ -2663,9 +2663,9 @@ Our `sym` is like a Lisp/Scheme/Racket symbol, which is essentially an [interned Our `gensym` is like the [Lisp `gensym`](http://clhs.lisp.se/Body/f_gensym.htm), and the [JavaScript `Symbol`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol). -If you're familiar with `mcpyrate`'s `gensym` or MacroPy's `gen_sym`, those mean something different. Their purpose is to create, in a macro, a lexical identifier that is not already in use in the source code being compiled, whereas our `gensym` creates an uninterned symbol object for run-time use. Lisp macros use symbols to represent identifiers, hence the potential for confusion in Python, where that is not the case. (The symbols of `unpythonic` are a purely run-time abstraction.) +If you're familiar with `mcpyrate`'s `gensym` or MacroPy's `gen_sym`, those mean something different. Their purpose is to create, in a macro, a lexical identifier that is not already in use in the source code being compiled, whereas our `gensym` creates an uninterned symbol object for run-time use. Lisp macros use symbols to represent identifiers, hence the potential for confusion in Python, where that is not the case. The symbols of `unpythonic` are a purely run-time abstraction. -If your background is in C++ or Java, you may notice the symbol abstraction is a kind of a parametric [singleton](https://en.wikipedia.org/wiki/Singleton_pattern); each symbol with the same name is a singleton (as is any gensym with the same UUID). +If your background is in C++ or Java, you may notice the symbol abstraction is a kind of a parametric [singleton](https://en.wikipedia.org/wiki/Singleton_pattern); each symbol with the same name is a singleton, as is any gensym with the same UUID. #### Singleton @@ -2682,7 +2682,7 @@ class SingleXHolder(Singleton): h = SingleXHolder(17) s = pickle.dumps(h) h2 = pickle.loads(s) -assert h2 is h # it's the same instance +assert h2 is h # the same instance! ``` Often the [singleton pattern](https://en.wikipedia.org/wiki/Singleton_pattern) is discussed in the context of classic relatively low-level, static languages such as C++ or Java. [In Python](https://stackoverflow.com/questions/6760685/creating-a-singleton-in-python), some of the classical issues, such as singletons being forced to use a clunky, nonstandard object construction syntax, are moot, because the language itself offers customization hooks that can be used to smooth away such irregularities. @@ -2697,7 +2697,7 @@ We believe in the principles of [separation of concerns](https://en.wikipedia.or Another question arises due to Python having builtin support for object persistence, namely `pickle`. What *should* happen when a singleton is unpickled, while an instance of that singleton already exists? Arguably, by default, it should load the state from the pickle file into the existing instance, overwriting its current state. -This design is based on considering the following scenario. During second and later runs, a program first initializes, which causes the singleton instance to be created, just like during the first run of that program. Then the program loads state from a pickle file, containing (among other data) the state the singleton instance was in when the program previously shut down. In this scenario, considering the singleton, the data in the file is more relevant than the defaults the program initialization feeds in. Hence the default should be to replace the state of the existing singleton instance with the data from the pickle file. +This design is based on considering the following scenario. Consider a program that uses the singleton abstraction. During its second and later runs, the program first initializes, which causes the singleton instance to be created, just like during the first run of the program. Then the program loads state from a pickle file, containing (among other data) the state the singleton instance was in when the program previously shut down. Considering the singleton, the data in the file is more relevant than the defaults the program initialization step feeds in. Hence, the default should be to *replace the state of the existing singleton instance with the data from the pickle file*. Our `Singleton` abstraction is the result of these pythonifications applied to the classic pattern. For more documentation and examples, see the unit tests in [`unpythonic/tests/test_singleton.py`](../unpythonic/tests/test_singleton.py). @@ -2710,11 +2710,11 @@ Our `Singleton` abstraction is the result of these pythonifications applied to t Most often, **don't**. `Singleton` is provided for the very rare occasion where it's the appropriate abstraction. There exist **at least** three categories of use cases where singleton-like instantiation semantics are desirable: 1. **A process-wide unique marker value**, which has no functionality other than being quickly and uniquely identifiable as that marker. - - `sym` and `gensym` are the specific tools that cover this use case, depending on whether the intent is to allow that value to be independently "constructed" in several places yet always obtaining the same instance (`sym`), or if the implementation just happens to internally need a guaranteed-unique value that no value passed in from the outside could possibly clash with (`gensym`). For the latter case, sometimes a simple (and much faster) `nonce = object()` will do just as well, if you don't need the human-readable label and `pickle` support. + - `sym` and `gensym` are the specific tools that cover this use case, depending on whether the intent is to allow that value to be independently "constructed" in several places yet always obtaining the same instance (`sym`), or if the implementation just happens to internally need a guaranteed-unique value that no value passed in from the outside could possibly clash with (`gensym`). For the latter case, sometimes the simple (and much faster) pythonic idiom `nonce = object()` will do just as well, if you don't need a human-readable label, and `pickle` support. - If you need the singleton object to have extra functionality (e.g. our `nil` supports the iterator protocol), it's possible to subclass `sym` or `gsym`, but subclassing `Singleton` is also a possible solution. 2. **An empty immutable collection**. - - It can't have elements added to it after construction, so there's no point in creating more than one instance of an empty *immutable* collection of any particular type. - - Unfortunately, a class can't easily be partly `Singleton` (i.e., only when the instance is empty). So this use case is better coded manually, like `frozendict` does. Also, for this use case silently returning the existing instance is the right thing to do. + - An immutable collection instance cannot have elements added to it after construction, so there is no point in creating more than one instance of an *empty* immutable collection of any particular type. + - Unfortunately, a class cannot easily be partly `Singleton` (i.e., only when the instance is empty). So this use case is better coded manually, like `frozendict` does. Also, for this use case silently returning the existing instance is the right thing to do. 3. **A service that may have at most one instance** per process. - *But only if it is certain* that there can't arise a situation where multiple simultaneous instances of the service are needed. - The dynamic assignment controller `dyn` is an example, and it is indeed a `Singleton`. From 0332644c40c91b408e2432c47c796a28dee0fb66 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 17:37:04 +0300 Subject: [PATCH 526/832] 0.15.0: improve trampolined/jump docs --- doc/features.md | 62 +++++++++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 22 deletions(-) diff --git a/doc/features.md b/doc/features.md index 54ef147e..ad9b3b88 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2726,13 +2726,15 @@ I am not completely sure if it is meaningful to provide a generic `Singleton` ab ## Control flow tools -Tools related to control flow. +Tools related to [control flow](https://en.wikipedia.org/wiki/Control_flow). ### `trampolined`, `jump`: tail call optimization (TCO) / explicit continuations -Express algorithms elegantly without blowing the call stack - with explicit, clear syntax. +*See also the `with tco` [macro](macros.md), which applies tail call optimization **automatically**.* -*Tail recursion*: +*Tail call optimization* is a technique to treat [tail calls](https://en.wikipedia.org/wiki/Tail_call) in such a way that they do not grow the call stack. It sometimes allows expressing algorithms very elegantly. Some functional programming patterns such as functional loops are based on tail calls. + +The factorial function is a classic example of *tail recursion*: ```python from unpythonic import trampolined, jump @@ -2741,62 +2743,66 @@ from unpythonic import trampolined, jump def fact(n, acc=1): if n == 0: return acc - else: - return jump(fact, n - 1, n * acc) + return jump(fact, n - 1, n * acc) print(fact(4)) # 24 +fact(5000) # no crash ``` -Functions that use TCO **must** be `@trampolined`. Calling a trampolined function normally starts the trampoline. +Functions that use TCO **must** be `@trampolined`. The decorator wraps the original function with a [trampoline](https://en.wikipedia.org/wiki/Trampoline_(computing)#High-level_programming). Calling a trampolined function normally starts the trampoline. Inside a trampolined function, a normal call `f(a, ..., kw=v, ...)` remains a normal call. -A tail call with target `f` is denoted `return jump(f, a, ..., kw=v, ...)`. This explicitly marks that it is indeed a tail call (due to the explicit `return`). Note that `jump` is **a noun, not a verb**. The `jump(f, ...)` part just evaluates to a `jump` instance, which on its own does nothing. Returning it to the trampoline actually performs the tail call. +A tail call with target `f` is denoted `return jump(f, a, ..., kw=v, ...)`. This explicitly marks that it is indeed a tail call, due to the explicit `return`. Note that `jump` is **a noun, not a verb**. The `jump(f, ...)` part just evaluates to a `jump` instance, which on its own does nothing. Returning the `jump` instance to the trampoline actually performs the tail call. -If the jump target has a trampoline, don't worry; the trampoline implementation will automatically strip it and jump into the actual entrypoint. +If the jump target has a trampoline, the trampoline implementation will automatically strip it and jump into the actual entry point. -Trying to `jump(...)` without the `return` does nothing useful, and will **usually** print an *unclaimed jump* warning. It does this by checking a flag in the `__del__` method of `jump`; any correctly used jump instance should have been claimed by a trampoline before it gets garbage-collected. +To return a final result, just `return` it normally. Returning anything but a `jump` shuts down the trampoline, and returns the given value from the initial call (to the `@trampolined` function) that originally started that trampoline. -(Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (`KeyboardInterrupt`). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by the trampoline.) +**CAUTION**: Trying to `jump(...)` without the `return` does nothing useful, and will **usually** print an *unclaimed jump* warning. It does this by checking a flag in the `__del__` method of `jump`; any correctly used jump instance should have been claimed by a trampoline before it gets garbage-collected. It can only print a warning, not raise an exception or halt the program, due to the limitations of `__del__`. -The final result is just returned normally. This shuts down the trampoline, and returns the given value from the initial call (to a `@trampolined` function) that originally started that trampoline. +Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (`KeyboardInterrupt`). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by a trampoline. +#### Tail recursion in a `lambda` -*Tail recursion in a lambda*: +To make a tail-recursive anonymous function, use `trampolined` together with `withself`. The `self` argument is declared explicitly, but passed implicitly, just like the `self` argument of a method: ```python +from unpythonic import trampolined, jump, withself + t = trampolined(withself(lambda self, n, acc=1: acc if n == 0 else jump(self, n - 1, n * acc))) print(t(4)) # 24 ``` -Here the jump is just `jump` instead of `return jump`, since lambda does not use the `return` syntax. - -To denote tail recursion in an anonymous function, use `unpythonic.fun.withself`. The `self` argument is declared explicitly, but passed implicitly, just like the `self` argument of a method. +Here the jump is just `jump` instead of `return jump`, because `lambda` does not use the `return` syntax. +#### Mutual recursion with TCO -*Mutual recursion with TCO*: +[Mutual recursion](https://en.wikipedia.org/wiki/Mutual_recursion) is also supported. Just ask the trampoline to `jump` into the desired function: ```python +from unpythonic import trampolines,jump + @trampolined def even(n): if n == 0: return True - else: - return jump(odd, n - 1) + return jump(odd, n - 1) @trampolined def odd(n): if n == 0: return False - else: - return jump(even, n - 1) + return jump(even, n - 1) assert even(42) is True assert odd(4) is False assert even(10000) is True # no crash ``` -*Mutual recursion in `letrec` with TCO*: +#### Mutual recursion in `letrec` with TCO ```python +from unpythonic import letrec, trampolined, jump + letrec(evenp=lambda e: trampolined(lambda x: (x == 0) or jump(e.oddp, x - 1)), @@ -2807,6 +2813,18 @@ letrec(evenp=lambda e: e.evenp(10000)) ``` +For comparison, with the macro API of `letrec`, this becomes: + +```python +from unpythonic.syntax import macros, letrec +from unpythonic import trampolined, jump + +letrec[[evenp << trampolined(lambda x: + (x == 0) or jump(oddp, x - 1)), + oddp << trampolined(lambda x: + (x != 0) and jump(evenp, x - 1))] in + evenp(10000)] +``` #### Reinterpreting TCO as explicit continuations @@ -2849,7 +2867,7 @@ Clojure has [`(trampoline ...)`](https://clojuredocs.org/clojure.core/trampoline The `return jump(...)` solution is essentially the same there (the syntax is `#(...)`), but in Clojure, the trampoline must be explicitly enabled at the call site, instead of baking it into the function definition, as our decorator does. -Clojure's trampoline system is thus more explicit and simple than ours (the trampoline doesn't need to detect and strip the tail-call target's trampoline, if it has one - because with Clojure's solution, it never does), at some cost to convenience at each use site. We have chosen to emphasize use-site convenience. +Clojure's trampoline system is thus more explicit and simple than ours (the trampoline does not need to detect and strip the tail-call target's trampoline, if it has one - because with Clojure's solution, it never does), at some cost to convenience at each use site. We have chosen to emphasize use-site convenience. ### `looped`, `looped_over`: loops in FP style (with TCO) From 3c88ac2d41dae97fc8dca2cd0207390a98585c4f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 14 Jun 2021 18:28:09 +0300 Subject: [PATCH 527/832] extend fup example --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 9325188f..16d0d0de 100644 --- a/README.md +++ b/README.md @@ -494,6 +494,13 @@ t = (1, 2, 3, 4, 5) s = fup(t)[0::2] << repeat(10) assert s == (10, 2, 10, 4, 10) assert t == (1, 2, 3, 4, 5) + +from itertools import count +from unpythonic import imemoize +t = (1, 2, 3, 4, 5) +s = fup(t)[::-2] << imemoize(count(start=10))() +assert s == (12, 2, 11, 4, 10) +assert t == (1, 2, 3, 4, 5) ```
Live list slices. From 284c91433b57bdc7b721d308bdbbef128f7c4f9a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 15 Jun 2021 00:25:06 +0300 Subject: [PATCH 528/832] add TCO macro API comparison --- doc/features.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/doc/features.md b/doc/features.md index ad9b3b88..638fc0d5 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2762,6 +2762,22 @@ To return a final result, just `return` it normally. Returning anything but a `j Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (`KeyboardInterrupt`). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by a trampoline. +For comparison, with the macro API, the example becomes: + +```python +from unpythonic.syntax import macros, tco + +with tco: + def fact(n, acc=1): + if n == 0: + return acc + return fact(n - 1, n * acc) +print(fact(4)) # 24 +fact(5000) # no crash +``` + +*The `with tco` macro implicitly inserts the `@trampolined` decorator, and converts any regular call that appears in tail position into a `jump`. It also transforms lambdas in a similar way.* + #### Tail recursion in a `lambda` To make a tail-recursive anonymous function, use `trampolined` together with `withself`. The `self` argument is declared explicitly, but passed implicitly, just like the `self` argument of a method: @@ -2776,6 +2792,18 @@ print(t(4)) # 24 Here the jump is just `jump` instead of `return jump`, because `lambda` does not use the `return` syntax. +For comparison, with the macro API, this becomes: + +```python +from unpythonic.syntax import macros, tco +from unpythonic import withself + +with tco: + t = withself(lambda self, n, acc=1: + acc if n == 0 else self(n - 1, n * acc)) +print(t(4)) # 24 +``` + #### Mutual recursion with TCO [Mutual recursion](https://en.wikipedia.org/wiki/Mutual_recursion) is also supported. Just ask the trampoline to `jump` into the desired function: From f590b23d1598adbc23b9dd4ed37834cc6efad7ad Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 15 Jun 2021 00:25:22 +0300 Subject: [PATCH 529/832] 0.15.0: improve looped/looped_over docs --- doc/features.md | 158 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 111 insertions(+), 47 deletions(-) diff --git a/doc/features.md b/doc/features.md index 638fc0d5..405b88f0 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2900,17 +2900,20 @@ Clojure's trampoline system is thus more explicit and simple than ours (the tram ### `looped`, `looped_over`: loops in FP style (with TCO) -*Functional loop with automatic tail call optimization* (for calls re-invoking the loop body): +In functional programming, looping can be represented as recursion. The loop body is written as a recursive function. To loop, the function tail-calls itself, possibly with new argument values. Both `for` and `while` loops can be expressed in this way. + +As a practical detail, tail-call optimization is important, to avoid growing the call stack at each iteration of the loop. + +Here is a functional loop using `unpythonic`, with automatic tail call optimization - no macros needed: ```python -from unpythonic import looped, looped_over +from unpythonic import looped @looped def s(loop, acc=0, i=0): if i == 10: return acc - else: - return loop(acc + i, i + 1) + return loop(acc + i, i + 1) print(s) # 45 ``` @@ -2927,32 +2930,39 @@ define s displayln s ; 45 ``` -The `@looped` decorator is essentially sugar. Behaviorally equivalent code: - -```python -@trampolined -def s(acc=0, i=0): - if i == 10: - return acc - else: - return jump(s, acc + i, i + 1) -s = s() -print(s) # 45 -``` - -In `@looped`, the function name of the loop body is the name of the final result, like in `@call`. The final result of the loop is just returned normally. +In `@looped`, the function name of the loop body is the name of the final result, like in `@call`. To terminate the loop, just `return` the final result normally. This shuts down the loop and replaces the loop body definition (in the example, `s`) with the final result value. The first parameter of the loop body is the magic parameter `loop`. It is *self-ish*, representing a jump back to the loop body itself, starting a new iteration. Just like Python's `self`, `loop` can have any name; it is passed positionally. -Note that `loop` is **a noun, not a verb.** This is because the expression `loop(...)` is essentially the same as `jump(...)` to the loop body itself. However, it also inserts the magic parameter `loop`, which can only be set up via this mechanism. +Note that `loop` is **a noun, not a verb.** This is because the expression `loop(...)` is essentially the same as `jump(...)` to the loop body itself. However, it also arranges things so that the trampolined call inserts the magic parameter `loop`, which can only be set up via this mechanism. Additional arguments can be given to `loop(...)`. When the loop body is called, any additional positional arguments are appended to the implicit ones, and can be anything. Additional arguments can also be passed by name. The initial values of any additional arguments **must** be declared as defaults in the formal parameter list of the loop body. The loop is automatically started by `@looped`, by calling the body with the magic `loop` as the only argument. -Any loop variables such as `i` in the above example are **in scope only in the loop body**; there is no `i` in the surrounding scope. Moreover, it's a fresh `i` at each iteration; nothing is mutated by the looping mechanism. (But be careful if you use a mutable object instance as a loop variable. The loop body is just a function call like any other, so the usual rules apply.) +Any loop variables such as `i` in the above example are **in scope only in the loop body**; there is no `i` in the surrounding scope. Moreover, it is a fresh `i` at each iteration; nothing is mutated by the looping mechanism. + +**Be careful** if you use a mutable object instance as a loop variable: the loop body is just a function call like any other, so the usual rules apply. + +For another example of functional looping, here is a typical `while True` loop in FP style: + +```python +from unpythonic import looped + +@looped +def _(loop): + print("Enter your name (or 'q' to quit): ", end='') + s = input() + if s.lower() == 'q': + return # ...the implicit None. In a "while True:", "break" here. + else: + print(f"Hello, {s}!") + return loop() +``` -FP loops don't have to be pure: +Functional loops do not have to be pure. Here is a functional loop with a side effect: ```python +from unpythonic import looped + out = [] @looped def _(loop, i=0): @@ -2963,29 +2973,37 @@ def _(loop, i=0): assert out == [0, 1, 2, 3] ``` -Keep in mind, though, that this pure-Python FP looping mechanism is slow, so it may make sense to use it only when "the FP-ness" (no mutation, scoping) is important. +**CAUTION**: This pure-Python FP looping mechanism is slow, so it may make sense to use it only when "the FP-ness" (no mutation, scoping) is important. + +#### Relation to the TCO system -Also be aware that `@looped` is specifically neither a `for` loop nor a `while` loop; instead, it is a general looping mechanism that can express both kinds of loops. +The `@looped` decorator is essentially sugar. If you read the section further above on TCO, you may have guessed how it is implemented: the `loop` function is actually a jump record in disguise, and `@looped` installs a trampoline. -*Typical `while True` loop in FP style*: +Indeed, the following code is behaviorally equivalent to the first example: ```python -@looped -def _(loop): - print("Enter your name (or 'q' to quit): ", end='') - s = input() - if s.lower() == 'q': - return # ...the implicit None. In a "while True:", "break" here. - else: - print(f"Hello, {s}!") - return loop() +from unpythonic import trampolined, jump + +@trampolined +def s(acc=0, i=0): + if i == 10: + return acc + return jump(s, acc + i, i + 1) +s = s() +print(s) # 45 ``` +However, the actual implementation of `@looped` slightly differs from what would be implied by this straightforward translation, because the feature uses no macros. + #### FP loop over an iterable -In Python, loops often run directly over the elements of an iterable, which markedly improves readability compared to dealing with indices. Enter `@looped_over`: +In Python, loops often run directly over the elements of an iterable, which markedly improves readability compared to dealing with indices. + +For this use case, we provide `@looped_over`: ```python +from unpythonic import looped_over + @looped_over(range(10), acc=0) def s(loop, x, acc): return loop(acc + x) @@ -2995,27 +3013,33 @@ assert s == 45 The `@looped_over` decorator is essentially sugar. Behaviorally equivalent code: ```python +from unpythonic import call, looped + @call def s(iterable=range(10)): it = iter(iterable) @looped - def _tmp(loop, acc=0): + def tmp(loop, acc=0): try: x = next(it) - return loop(acc + x) + return loop(acc + x) # <-- the loop body except StopIteration: return acc - return _tmp + return tmp assert s == 45 ``` -In `@looped_over`, the loop body takes three magic positional parameters. The first parameter `loop` works like in `@looped`. The second parameter `x` is the current element. The third parameter `acc` is initialized to the `acc` value given to `@looped_over`, and then (functionally) updated at each iteration, taking as the new value the first positional argument given to `loop(...)`, if any positional arguments were given. Otherwise `acc` retains its last value. +In `@looped_over`, the loop body takes **three** magic positional parameters. The first parameter `loop` is similar to that in `@looped`. The second parameter `x` is the current element. The third parameter `acc` is initialized to the `acc` value given to `@looped_over`, and then (functionally) updated at each iteration. -If `acc` is a mutable object, mutating it is allowed. For example, if `acc` is a list, it is perfectly fine to `acc.append(...)` and then just `loop()` with no arguments, allowing `acc` to retain its last value. To be exact, keeping the last value means *the binding of the name `acc` does not change*, so when the next iteration starts, the name `acc` still points to the same object that was mutated. This strategy can be used to pythonically construct a list in an FP loop. +The new value of `acc` is the first positional argument given to `loop(...)`, if any positional arguments were given. Otherwise `acc` retains its last value. + +If `acc` is a mutable object, mutating it **is allowed**. For example, if `acc` is a list, it is perfectly fine to `acc.append(...)` and then just `loop()` with no arguments, allowing `acc` to retain its last value. To be exact, keeping the last value means *the binding of the name `acc` does not change*, so when the next iteration starts, the name `acc` still points to the same object that was mutated. This strategy can be used to pythonically construct a list in an FP loop. Additional arguments can be given to `loop(...)`. The same notes as above apply. For example, here we have the additional parameters `fruit` and `number`. The first one is passed positionally, and the second one by name: ```python +from unpythonic import looped_over + @looped_over(range(10), acc=0) def s(loop, x, acc, fruit="pear", number=23): print(fruit, number) @@ -3025,13 +3049,15 @@ def s(loop, x, acc, fruit="pear", number=23): assert s == 45 ``` -The loop body is called once for each element in the iterable. When the iterable runs out of elements, the last `acc` value that was given to `loop(...)` becomes the return value of the loop. If the iterable is empty, the body never runs; then the return value of the loop is the initial value of `acc`. +The loop body is called once for each element in the iterable. When the iterable runs out of elements, the final value of `acc` becomes the return value of the loop. If the iterable is empty, the body never runs; then the return value of the loop is the initial value of `acc`. -To terminate the loop early, just `return` your final result normally, like in `@looped`. (It can be anything, does not need to be `acc`.) +To terminate the loop early, just `return` your final result normally, like in `@looped`. It can be anything, it does not need to be `acc`. Multiple input iterables work somewhat like in Python's `for`, except any sequence unpacking must be performed inside the body: ```python +from unpythonic import looped_over + @looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=()) def p(loop, item, acc): numb, lett = item @@ -3050,6 +3076,8 @@ This is because while *tuple parameter unpacking* was supported in Python 2.x, i FP loops can be nested (also those over iterables): ```python +from unpythonic import looped_over + @looped_over(range(1, 4), acc=()) def outer_result(outer_loop, y, outer_acc): @looped_over(range(1, 3), acc=()) @@ -3071,6 +3099,8 @@ As [the reference warns (note 6)](https://docs.python.org/3/library/stdtypes.htm Mutable sequence (Python `list`): ```python +from unpythonic import looped_over + @looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=[]) def p(loop, item, acc): numb, lett = item @@ -3083,7 +3113,7 @@ assert p == ['1a', '2b', '3c'] Linked list: ```python -from unpythonic import cons, nil, ll +from unpythonic import looped_over, cons, nil, ll, lreverse @lreverse @looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=nil) @@ -3099,6 +3129,8 @@ Note the unpythonic use of the `lreverse` function as a decorator. `@looped_over To get the output as a tuple, we can add `tuple` to the decorator chain: ```python +from unpythonic import looped_over, cons, nil, ll, lreverse + @tuple @lreverse @looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=nil) @@ -3119,9 +3151,11 @@ If you want to exit the function *containing* the loop from inside the loop, see #### `continue` -The main way to *continue* an FP loop is, at any time, to `loop(...)` with the appropriate arguments that will make it proceed to the next iteration. Or package the appropriate `loop(...)` expression into your own function `cont`, and then use `cont(...)`: +The main way to *continue* an FP loop is, at any time, to `loop(...)` with the appropriate arguments that will make the loop proceed to the next iteration. Or package the appropriate `loop(...)` expression into your own function `cont`, and then use `cont(...)`: ```python +from unpythonic import looped + @looped def s(loop, acc=0, i=0): cont = lambda newacc=acc: loop(newacc, i + 1) # always increase i; by default keep current value of acc @@ -3140,9 +3174,9 @@ This approach separates the computations of the new values for the iteration cou See `@breakably_looped` (offering `brk`) and `@breakably_looped_over` (offering `brk` and `cnt`). -The point of `brk(value)` over just `return value` is that `brk` is first-class, so it can be passed on to functions called by the loop body (so that those functions then have the power to directly terminate the loop). +The point of `brk(value)` over just `return value` is that `brk` is first-class, so it can be passed on to functions called by the loop body - so that those functions then have the power to directly terminate the loop. -In `@looped`, a library-provided `cnt` wouldn't make sense, since all parameters except `loop` are user-defined. *The client code itself defines what it means to proceed to the "next" iteration*. Really the only way in a construct with this degree of flexibility is for the client code to fill in all the arguments itself. +In `@looped`, a library-provided `cnt` would not make sense, since all parameters except `loop` are user-defined. *The client code itself defines what it means to proceed to the "next" iteration*. Really the only way in a construct with this degree of flexibility is for the client code to fill in all the arguments itself. Because `@looped_over` is a more specific abstraction, there the concept of *continue* is much more clear-cut. We define `cnt` to mean *proceed to take the next element from the iterable, keeping the current value of `acc`*. Essentially `cnt` is a partially applied `loop(...)` with the first positional argument set to the current value of `acc`. @@ -3151,16 +3185,20 @@ Because `@looped_over` is a more specific abstraction, there the concept of *con Just call the `looped()` decorator manually: ```python +from unpythonic import looped + s = looped(lambda loop, acc=0, i=0: loop(acc + i, i + 1) if i < 10 else acc) print(s) ``` -It's not just a decorator; in Lisps, a construct like this would likely be named `call/looped`. +It's not just a decorator; in the Scheme family of Lisps, a construct like this would likely be named `call/looped`. We can also use `let` to make local definitions: ```python +from unpythonic import looped, let + s = looped(lambda loop, acc=0, i=0: let(cont=lambda newacc=acc: loop(newacc, i + 1), @@ -3172,6 +3210,8 @@ print(s) The `looped_over()` decorator also works, if we just keep in mind that parameterized decorators in Python are actually decorator factories: ```python +from unpythonic import looped_over + r10 = looped_over(range(10), acc=0) s = r10(lambda loop, x, acc: loop(acc + x)) @@ -3180,15 +3220,39 @@ assert s == 45 If you **really** need to make that into an expression, bind `r10` using `let` (if you use `letrec`, keeping in mind it is a callable), or to make your code unreadable, just inline it. -With `curry`, this is also a possible solution: +With `curry`, using its passthrough feature, this is also a possible solution: ```python +from unpythonic import curry, looped_over + s = curry(looped_over, range(10), 0, lambda loop, x, acc: loop(acc + x)) assert s == 45 ``` +As of v0.15.0, `curry` handles also named arguments, so we can make explicit what the `0` means: + +```python +from unpythonic import curry, looped_over + +s = curry(looped_over, range(10), acc=0, + body=(lambda loop, x, acc: + loop(acc + x))) +assert s == 45 +``` + +but because, due to syntactic limitations of Python, no positional arguments can be given *after* a named argument, you then have to know - in order to be able to provide the loop body - that the decorator returned by the factory `looped_over` calls it `body`. + +You can of course obtain such information by inspection (here shown in IPython running Python 3.8): + +```python +In [2]: looped_over(range(10), acc=0) +Out[2]: .run(body)> +``` + +or by looking at [the source code](../unpythonic/fploop.py). + ### `gtrampolined`: generators with TCO In `unpythonic`, a generator can tail-chain into another generator. This is like invoking `itertools.chain`, but as a tail call from inside the generator - so the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. From 84c30fded0d6eb0a3766768e4f2885c0742832ae Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 15 Jun 2021 00:31:31 +0300 Subject: [PATCH 530/832] 0.15.0: update gtrampolined doc --- doc/features.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 405b88f0..0ff5a7f7 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3253,9 +3253,10 @@ Out[2]: .run(body)> or by looking at [the source code](../unpythonic/fploop.py). + ### `gtrampolined`: generators with TCO -In `unpythonic`, a generator can tail-chain into another generator. This is like invoking `itertools.chain`, but as a tail call from inside the generator - so the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. +In `unpythonic`, a generator can tail-chain into another generator. This is like invoking `itertools.chain`, but as a tail call from inside the generator - so that the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. Python provides a convenient hook to build things like this, in the guise of `return`: @@ -3298,15 +3299,15 @@ def fibos(): # see numerics.py print(tuple(take(10, fibos()))) # --> (1, 1, 2), only 3 terms?! ``` -This sequence (technically iterable, but in the mathematical sense) is recursively defined, and the `return` shuts down the generator before it can yield more terms into `scanl`. With `yield from` instead of `return` the second example works (but since it is recursive, it eventually blows the call stack). +This sequence (technically iterable, but in the mathematical sense) is recursively defined, and the `return` shuts down the generator before it can yield more terms into `scanl`. With `yield from` instead of `return` the second example works - but since it is recursive, it eventually blows the call stack. This particular example can be converted into a linear process with a different higher-order function, no TCO needed: ```python -from unpythonic import unfold, take, last +from unpythonic import unfold, take, last, Values def fibos(): def nextfibo(a, b): - return a, b, a + b # value, *newstates + return Values(a, a=b, b=a + b) return unfold(nextfibo, 1, 1) assert tuple(take(10, fibos())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) last(take(10000, fibos())) # no crash From d02857fb5cb60ae0f35870de429ffeee645a33e9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 15 Jun 2021 00:31:39 +0300 Subject: [PATCH 531/832] update deprecation notice --- doc/features.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 0ff5a7f7..8c5b0193 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3316,7 +3316,9 @@ last(take(10000, fibos())) # no crash ### `catch`, `throw`: escape continuations (ec) -**Changed in v0.14.2.** *These constructs were previously named `setescape`, `escape`. The names have been changed to match the standard naming for this feature in several Lisps. Starting in 0.14.2, using the old names emits a `FutureWarning`, and the old names will be removed in 0.15.0.* +**Changed in v0.15.0.** *The deprecated names have been removed.* + +**Changed in v0.14.2.** *These constructs were previously named `setescape`, `escape`. The names have been changed to match the standard naming for this feature in several Lisps. Starting in 0.14.2, using the old names emits a `FutureWarning`.* Escape continuations can be used as a *multi-return*: From 5b0b61ba3ed8ff6b5894e73b67dd6512a30f4238 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 15 Jun 2021 00:34:45 +0300 Subject: [PATCH 532/832] deprecation notice wording --- doc/features.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 8c5b0193..6ce6dfd1 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1170,7 +1170,9 @@ assert y == 17 *The variants `piped` and `lazy_piped` automatically pack the initial arguments into a `Values`.* -**Changed in v0.14.2**. *Both `getvalue` and `runpipe`, used in the shell-like syntax, are now known by the single unified name `exitpipe`. This is just a rename, with no functionality changes. The old names are deprecated in 0.14.2 and 0.14.3, and have been removed in 0.15.0.* +*The deprecated names `getvalue` and `runpipe` have been removed.* + +**Changed in v0.14.2**. *Both `getvalue` and `runpipe`, used in the shell-like syntax, are now known by the single unified name `exitpipe`. This is just a rename, with no functionality changes. The old names are now deprecated.* Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/), but no macros. A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It is just function composition, but with an emphasis on data flow, which helps improve readability. @@ -2505,9 +2507,11 @@ For convenience, we support some special cases: ### `s`, `imathify`, `gmathify`: lazy mathematical sequences with infix arithmetic -**Changed in v0.14.3.** *Added convenience mode to generate cyclic infinite sequences.* +**Changed in v0.15.0.** *The deprecated names have been removed.* -**Changed in v0.14.3.** *To improve descriptiveness, and for consistency with names of other abstractions in `unpythonic`, `m` has been renamed `imathify` and `mg` has been renamed `gmathify`. The old names work in v0.14.3, and have been removed in v0.15.0. This is a one-time change; it is not likely that these names will be changed ever again.* +**Changed in v0.14.3.** *To improve descriptiveness, and for consistency with names of other abstractions in `unpythonic`, `m` has been renamed `imathify` and `mg` has been renamed `gmathify`. This is a one-time change; it is not likely that these names will be changed ever again. The old names are now deprecated.* + +**Changed in v0.14.3.** *Added convenience mode to generate cyclic infinite sequences.* We provide a compact syntax to create lazy constant, cyclic, arithmetic, geometric and power sequences: `s(...)`. Numeric (`int`, `float`, `mpmath`) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. @@ -3318,7 +3322,7 @@ last(take(10000, fibos())) # no crash **Changed in v0.15.0.** *The deprecated names have been removed.* -**Changed in v0.14.2.** *These constructs were previously named `setescape`, `escape`. The names have been changed to match the standard naming for this feature in several Lisps. Starting in 0.14.2, using the old names emits a `FutureWarning`.* +**Changed in v0.14.2.** *These constructs were previously named `setescape`, `escape`. The names have been changed to match the standard naming for this feature in several Lisps. The old names are now deprecated.* Escape continuations can be used as a *multi-return*: From 2f112413777fe7938d5bb097d1923ea7222d4340 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 15 Jun 2021 00:40:01 +0300 Subject: [PATCH 533/832] document (as a test): fupdate cannot read a general iterable backwards (The iterable must be imemoized to be able to do that; that functionality is already covered by the tests.) --- unpythonic/tests/test_fup.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/unpythonic/tests/test_fup.py b/unpythonic/tests/test_fup.py index 827bb9ce..ee77832f 100644 --- a/unpythonic/tests/test_fup.py +++ b/unpythonic/tests/test_fup.py @@ -127,6 +127,10 @@ def runtests(): # cannot specify both indices and bindings test_raises[ValueError, fupdate(tup, slice(1, None, 2), (10,), somename="some value")] + # not memoized, cannot read a general iterable backwards + tup = (1, 2, 3, 4, 5) + test_raises[IndexError, fupdate(tup, slice(None, None, -1), count(start=10))] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From cd16663e0775b826fa995e92a7d810679d8d96ae Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 10:46:25 +0300 Subject: [PATCH 534/832] add tests for subscripting a memoized generator --- unpythonic/tests/test_gmemo.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/unpythonic/tests/test_gmemo.py b/unpythonic/tests/test_gmemo.py index a0e05627..f72ea945 100644 --- a/unpythonic/tests/test_gmemo.py +++ b/unpythonic/tests/test_gmemo.py @@ -92,6 +92,23 @@ def gen(): fail["Should have raised at the second next() call."] # pragma: no cover test[total_evaluations == 2] + with testset("subscripting to get already computed items"): + @gmemoize + def gen(): + yield from range(5) + g3 = gen() + test[len(g3) == 0] + next(g3) + test[len(g3) == 1] + next(g3) + test[len(g3) == 2] + next(g3) + test[len(g3) == 3] + test[g3[0] == 0] + test[g3[1] == 1] + test[g3[2] == 2] + test_raises[IndexError, g3[3]] + with testset("memoizing a sequence partially"): # To do this, build a chain of generators, then memoize only the last one: evaluations = Counter() From af8da7aeb6f7edf60602733e8a49edfcc2ec48b7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 10:48:53 +0300 Subject: [PATCH 535/832] imemoize: improve clarity and stack traces Use a named function instead of a lambda. This replaces one long line requiring a one-line comment (so a total of two lines) with four easily readable lines. --- unpythonic/gmemo.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/unpythonic/gmemo.py b/unpythonic/gmemo.py index 6e9194e2..4ac9a70c 100644 --- a/unpythonic/gmemo.py +++ b/unpythonic/gmemo.py @@ -172,8 +172,10 @@ def imemoize(iterable): If you need to take arguments to create the iterable, see ``fimemoize``. """ - # The lambda is the gfunc; decorate it with gmemoize and return that. - return gmemoize(lambda: (yield from iterable)) + @gmemoize + def iterable_as_gfunc(): + yield from iterable + return iterable_as_gfunc @register_decorator(priority=10) def fimemoize(ifactory): From 7fa7c98abd55f8189d7ad9fa5041966e22544462 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 11:10:20 +0300 Subject: [PATCH 536/832] add slicing support to memo subscripting in memoized generators --- unpythonic/gmemo.py | 18 +++++++++++++++--- unpythonic/tests/test_gmemo.py | 24 ++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/unpythonic/gmemo.py b/unpythonic/gmemo.py index 4ac9a70c..ecd14e65 100644 --- a/unpythonic/gmemo.py +++ b/unpythonic/gmemo.py @@ -132,12 +132,24 @@ def __next__(self): if kind is _fail: raise value return value - # Support the `collections.abc.Sequence` API for already-computed items + # Support a subset of the `collections.abc.Sequence` API for already-computed items def __len__(self): return len(self.memo) def __getitem__(self, k): - if k >= len(self.memo): - raise IndexError(f"Attempted to access index {k} of memoized generator; only {len(self.memo)} items available (at least so far)") + if not isinstance(k, (int, slice)): + raise TypeError(f"Expected an int or slice index, got {type(k)} with value {repr(k)}") + length = len(self.memo) + if isinstance(k, slice): + # For slices where at least one item raises an exception, we raise the + # exception that is encountered first when walking the slice. + lst = [] + for kind, value in self.memo[k]: + if kind is _fail: + raise value + lst.append(value) + return lst + if k >= length or k < -length: + raise IndexError(f"memoized generator index out of range; got {k}, with {len(self.memo)} items currently available") kind, value = self.memo[k] if kind is _fail: raise value diff --git a/unpythonic/tests/test_gmemo.py b/unpythonic/tests/test_gmemo.py index f72ea945..d3539044 100644 --- a/unpythonic/tests/test_gmemo.py +++ b/unpythonic/tests/test_gmemo.py @@ -97,6 +97,9 @@ def gen(): def gen(): yield from range(5) g3 = gen() + + # Any item that has entered the memo can be retrieved by subscripting. + # len() is the current length of the memo. test[len(g3) == 0] next(g3) test[len(g3) == 1] @@ -107,8 +110,29 @@ def gen(): test[g3[0] == 0] test[g3[1] == 1] test[g3[2] == 2] + + # Items not yet memoized cannot be retrieved from the memo. test_raises[IndexError, g3[3]] + # Negative indices work too, counting from the current end of the memo. + test[g3[-1] == 2] + test[g3[-2] == 1] + test[g3[-3] == 0] + + # Counting back past the start is an error, just like in `list`. + test_raises[IndexError, g3[-4]] + + # Slicing is supported. + test[g3[0:3] == [0, 1, 2]] + test[g3[0:2] == [0, 1]] + test[g3[::-1] == [2, 1, 0]] + test[g3[0::2] == [0, 2]] + test[g3[2::-2] == [2, 0]] + + # Out-of-range slices produce the empty list, like in `list`. + test[g3[3:] == []] + test[g3[-4::-1] == []] + with testset("memoizing a sequence partially"): # To do this, build a chain of generators, then memoize only the last one: evaluations = Counter() From 390fc74a04510502592cb152831b6772fb074a81 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 11:19:27 +0300 Subject: [PATCH 537/832] update changelog --- CHANGELOG.md | 2 +- doc/features.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30309e5a..d945abef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,7 +119,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Positional passthrough works as before. Named passthrough added. - Any remaining arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result (if any), and then outward on the curry context stack as a `Values`. Since `curry` in this role is essentially a function-composition utility, the receiving curried function instance unpacks the `Values` into args and kwargs. - If any extra arguments (positional or named) remain when the top-level curry context exits, then by default, `TypeError` is raised. To override, use `with dyn.let(curry_context=["whatever"])`, just like before. Then you'll get a `Values` object. - - The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Note that they do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose. + - The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Asking for the `len` returns the current length of the memo. For subscripting, both a single `int` index and a slice are accepted. Note that memoized generators do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose. - `fup`/`fupdate`/`ShadowedSequence` can now walk the start of a memoized infinite replacement backwards. (Use `imemoize` on the original iterable, instantiate the generator, and use that generator instance as the replacement.) - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. diff --git a/doc/features.md b/doc/features.md index 6ce6dfd1..bdae17e8 100644 --- a/doc/features.md +++ b/doc/features.md @@ -2163,7 +2163,7 @@ Inspired by Python itself. ### `gmemoize`, `imemoize`, `fimemoize`: memoize generators -**Changed in v0.15.0.** *The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Note that they do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose.* +**Changed in v0.15.0.** *The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Asking for the `len` returns the current length of the memo. For subscripting, both a single `int` index and a slice are accepted. Note that memoized generators do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose.* Make generator functions (gfunc, i.e. a generator definition) which create memoized generators, similar to how streams behave in Racket. From 89068f3298e1f76a177dc123f1921685a606fd22 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 13:22:28 +0300 Subject: [PATCH 538/832] tests: "thread-safety", not "multithreading" is what is being tested --- unpythonic/tests/test_conditions.py | 2 +- unpythonic/tests/test_dynassign.py | 4 ++-- unpythonic/tests/test_fix.py | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/unpythonic/tests/test_conditions.py b/unpythonic/tests/test_conditions.py index e41bf1b4..3f7a529e 100644 --- a/unpythonic/tests/test_conditions.py +++ b/unpythonic/tests/test_conditions.py @@ -557,7 +557,7 @@ def lowlevel3(): cancel_and_delegate() # Multithreading. Threads behave independently. - with testset("multithreading"): + with testset("thread-safety"): def multithreading(): comm = Queue() def lowlevel4(tag): diff --git a/unpythonic/tests/test_dynassign.py b/unpythonic/tests/test_dynassign.py index d3457b2a..2a5b7049 100644 --- a/unpythonic/tests/test_dynassign.py +++ b/unpythonic/tests/test_dynassign.py @@ -38,7 +38,7 @@ def basictests(): test_raises[AttributeError, dyn.b] # no longer exists - with testset("multithreading"): + with testset("thread-safety"): comm = Queue() def threadtest(q): try: @@ -112,7 +112,7 @@ def threadtest(q): test[noimplicits(dyn.items()) == (("a", 10), ("b", 20))] test[noimplicits(dyn.items()) == ()] - with testset("mass update with multithreading"): + with testset("mass update, thread-safety"): comm = Queue() def worker(): # test[] itself is thread-safe, but the worker threads don't have a diff --git a/unpythonic/tests/test_fix.py b/unpythonic/tests/test_fix.py index 0a93b18d..63edf2b8 100644 --- a/unpythonic/tests/test_fix.py +++ b/unpythonic/tests/test_fix.py @@ -105,7 +105,7 @@ def iterate1_rec(f, x): f, c = cosser2(1) # f ends up in the return value because it's in the args of iterate1_rec. test[the[c] == the[cos(c)]] - with testset("multithreading"): + with testset("thread-safety"): def threadtest(): a_calls = [] @fix() From b90744e0c9d6289ba6be24511f67dc7c660806b0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 13:23:01 +0300 Subject: [PATCH 539/832] naming: `que` instead of `q` since `q` is also mcpyrate's quasiquote Just for disambiguation for humans. --- unpythonic/tests/test_dynassign.py | 6 +++--- unpythonic/tests/test_fix.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/unpythonic/tests/test_dynassign.py b/unpythonic/tests/test_dynassign.py index 2a5b7049..d19d8ec9 100644 --- a/unpythonic/tests/test_dynassign.py +++ b/unpythonic/tests/test_dynassign.py @@ -40,12 +40,12 @@ def basictests(): with testset("thread-safety"): comm = Queue() - def threadtest(q): + def threadtest(que): try: dyn.c # just access dyn.c except AttributeError as err: - q.put(err) - q.put(None) + que.put(err) + que.put(None) with dyn.let(c=42): t1 = threading.Thread(target=threadtest, args=(comm,), kwargs={}) diff --git a/unpythonic/tests/test_fix.py b/unpythonic/tests/test_fix.py index 63edf2b8..bc17d873 100644 --- a/unpythonic/tests/test_fix.py +++ b/unpythonic/tests/test_fix.py @@ -119,9 +119,9 @@ def b(tid, k): return a(tid, (k + 1) % 3) comm = Queue() - def worker(q): + def worker(que): r = a(id(threading.current_thread()), 0) - q.put(r is NoReturn) + que.put(r is NoReturn) n = 1000 threads = [threading.Thread(target=worker, args=(comm,), kwargs={}) for _ in range(n)] From a40769998deb1d73d5742731ec9d199021192e78 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 13:24:05 +0300 Subject: [PATCH 540/832] add comment --- unpythonic/fun.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index f1136272..f7525a56 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -322,6 +322,8 @@ def curried(*args, **kwargs): # In order to decide what to do when the curried function is called, we must first compute # the parameter bindings. All of `f`'s parameters must be bound (whether by position or by # name) before calling `f`. + # + # The parameter binding analysis result is needed for passthrough. try: action, analysis = _analyze_parameter_bindings(f, args, kwargs) except ValueError as err: # inspection failed in inspect.signature()? From eb52f0958c9864506bc8428e6e849378703a4499 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 13:24:15 +0300 Subject: [PATCH 541/832] naming, docstring --- unpythonic/fun.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index f7525a56..26349236 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -325,7 +325,7 @@ def curried(*args, **kwargs): # # The parameter binding analysis result is needed for passthrough. try: - action, analysis = _analyze_parameter_bindings(f, args, kwargs) + action, analysis = _decide_curry_action(f, args, kwargs) except ValueError as err: # inspection failed in inspect.signature()? msg = err.args[0] if "no signature found" in msg: @@ -453,11 +453,18 @@ def _currycall(f, *args, **kwargs): _Analysis = namedtuple("_Analysis", ["bound_arguments", "unbound_parameters", "extra_args", "extra_kwargs"]) -# Internal helper for `curry`. -# # For performance, it is important to have this function defined once at the top level # of the module, instead of defining it as a closure each time `curry` is called. -def _analyze_parameter_bindings(f, args, kwargs): +def _decide_curry_action(f, args, kwargs): + """ Internal helper for `curry`. + + The `args` and `kwargs` are those added at this step of currying. + + We detect if `f` is a `functools.partial` object, and automatically extract + any previously supplied `args` and `kwargs` for analysis. + + Return value is `(action, analysis)`. See source code for details. + """ # `functools.partial()` doesn't remove an already-set kwarg from the signature (as seen by # `inspect.signature`), but `functools.partial` objects have a `keywords` attribute, which # contains what we want. From f1551f467145759daccf1531c9fcbb825c25cace Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 13:37:23 +0300 Subject: [PATCH 542/832] memoize: make thread-safe --- CHANGELOG.md | 2 ++ unpythonic/fun.py | 32 +++++++++++++++++++++++++------- unpythonic/tests/test_fun.py | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d945abef..3caae4a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -204,6 +204,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Fix bug: `fup`/`fupdate`/`ShadowedSequence` now actually accept an infinite-length iterable as a replacement sequence (under the obvious usage limitations), as the documentation has always claimed. +- Fix bug: `memoize` is now thread-safe. + --- diff --git a/unpythonic/fun.py b/unpythonic/fun.py index 26349236..a8788f63 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -20,6 +20,7 @@ from collections import namedtuple from functools import wraps, partial as functools_partial from inspect import signature +from threading import RLock from typing import get_type_hints from .arity import (_resolve_bindings, tuplify_bindings, _bind) @@ -61,18 +62,35 @@ def memoize(f): **CAUTION**: ``f`` must be pure (no side effects, no internal state preserved between invocations) for this to make any sense. + + Beginning with v0.15.0, `memoize` is thread-safe even when the same memoized + function instance is called concurrently from multiple threads. Exactly one + thread will compute the result. If `f` is recursive, the thread that acquired + the lock is the one that is allowed to recurse into the memoized `f`. """ + # One lock per use site of `memoize`. We use an `RLock` to allow recursive calls + # to the memoized `f` in the thread that acquired the lock. + lock = RLock() memo = {} @wraps(f) def memoized(*args, **kwargs): k = tuplify_bindings(_resolve_bindings(f, args, kwargs, _partial=False)) - if k not in memo: - try: - result = (_success, maybe_force_args(f, *args, **kwargs)) - except BaseException as err: - result = (_fail, err) - memo[k] = result # should yell separately if k is not a valid key - kind, value = memo[k] + try: # EAFP to eliminate TOCTTOU. + kind, value = memo[k] + except KeyError: + # But we still need to be careful to avoid race conditions. + with lock: + if k not in memo: + # We were the first thread to acquire the lock. + try: + result = (_success, maybe_force_args(f, *args, **kwargs)) + except BaseException as err: + result = (_fail, err) + memo[k] = result # should yell separately if k is not a valid key + else: + # Some other thread acquired the lock before us. + pass + kind, value = memo[k] if kind is _fail: raise value return value diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 4eaa7650..3b8f38db 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -5,6 +5,9 @@ from collections import Counter import sys +from queue import Queue +import threading +from time import sleep from ..dispatch import generic from ..fun import (memoize, partial, curry, apply, @@ -16,6 +19,8 @@ to1st, to2nd, tokth, tolast, to, withself) from ..funutil import Values +from ..it import allsame +from ..misc import slurp from ..dynassign import dyn @@ -135,6 +140,36 @@ def t(): fail["memoize should not prevent exception propagation."] # pragma: no cover test[evaluations == 1] + with testset("@memoize thread-safety"): + def threadtest(): + @memoize + def f(x): + # Sleep a "long" time to make actual concurrent operation more likely. + sleep(0.001) + + # The trick here is that because only one thread will acquire the lock + # for the memo, then for the same `x`, all the results should be the same. + return (id(threading.current_thread()), x) + + comm = Queue() + def worker(que): + # The value of `x` doesn't matter, as long as it's the same in all workers. + r = f(42) + que.put(r) + + n = 1000 + threads = [threading.Thread(target=worker, args=(comm,), kwargs={}) for _ in range(n)] + for t in threads: + t.start() + for t in threads: + t.join() + + # Test that all threads finished, and that the results from each thread are the same. + results = slurp(comm) + test[the[len(results)] == the[n]] + test[allsame(results)] + threadtest() + with testset("partial (type-checking wrapper)"): def nottypedfunc(x): return "ok" From da2c8a07bbef0648bd49afe0046a424df390aabf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:13:25 +0300 Subject: [PATCH 543/832] add missing TOC links --- doc/features.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/features.md b/doc/features.md index bdae17e8..d36c8239 100644 --- a/doc/features.md +++ b/doc/features.md @@ -69,12 +69,27 @@ The exception are the features marked **[M]**, which are primarily intended as a [**Control flow tools**](#control-flow-tools) - [`trampolined`, `jump`: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) + - [Tail recursion in a `lambda`](#tail-recursion-in-a-lambda) + - [Mutual recursion with TCO](#mutual-recursion-with-tco) + - [Mutual recursion in `letrec` with TCO](#mutual-recursion-in-letrec-with-tco) + - [Reinterpreting TCO as explicit continuations](#reinterpreting-tco-as-explicit-continuations) - [`looped`, `looped_over`: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) + - [Relation to the TCO system](#relation-to-the-tco-system) + - [FP loop over an iterable](#fp-loop-over-an-iterable): the `looped_over` parametric decorator + - [Accumulator type and runtime cost](#accumulator-type-and-runtime-cost) + - [`break`](#break) + - [`continue`](#continue) + - [Prepackaged `break` and `continue`](#prepackaged-break-and-continue) + - [FP loops using a lambda as body](#fp-loops-using-a-lambda-as-body) - [`gtrampolined`: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like `itertools.chain`, but from inside a generator. - [`catch`, `throw`: escape continuations (ec)](#catch-throw-escape-continuations-ec) (as in [Lisp's `catch`/`throw`](http://www.gigamonkeys.com/book/the-special-operators.html), unlike C++ or Java) - [`call_ec`: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's `call/ec`. - [`forall`: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. - [`handlers`, `restarts`: conditions and restarts](#handlers-restarts-conditions-and-restarts), a.k.a. **resumable exceptions**. + - [Fundamental signaling protocol](#fundamental-signaling-protocol) + - [API summary](#api-summary) + - [High-level signaling protocols](#high-level-signaling-protocols) + - [Conditions vs. exceptions](#conditions-vs-exceptions) - [`generic`, `typed`, `isoftype`: multiple dispatch](#generic-typed-isoftype-multiple-dispatch): create generic functions with type annotation syntax; also some friendly utilities. [**Exception tools**](#exception-tools) From c1d7da2de97f168f9820586bd4ad5c22c5f92f24 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:14:14 +0300 Subject: [PATCH 544/832] improve ec docs --- doc/features.md | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/doc/features.md b/doc/features.md index d36c8239..03dcf2af 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3339,7 +3339,9 @@ last(take(10000, fibos())) # no crash **Changed in v0.14.2.** *These constructs were previously named `setescape`, `escape`. The names have been changed to match the standard naming for this feature in several Lisps. The old names are now deprecated.* -Escape continuations can be used as a *multi-return*: +In a nutshell, an *escape continuation*, often abbreviated *ec*, transfers control outward on the call stack. Escape continuations are a generalization of `continue`, `break` and `return`. Those three constructs are essentially second-class ecs with a hard-coded escape point (respectively: end of iteration of loop; end of loop; end of function). A general escape continuation mechanism allows setting an escape point explicitly. + +For example, escape continuations can be used as a *multi-return*: ```python from unpythonic import catch, throw @@ -3354,13 +3356,11 @@ def f(): assert f() == "hello from g" ``` -**CAUTION**: The implementation is based on exceptions, so catch-all `except:` statements will intercept also throws, breaking the escape mechanism. As you already know, be specific in which exception types you catch in an `except` clause! - In Lisp terms, `@catch` essentially captures the escape continuation (ec) of the function decorated with it. The nearest (dynamically) surrounding ec can then be invoked by `throw(value)`. When the `throw` is performed, the function decorated with `@catch` immediately terminates, returning `value`. -In Python terms, a throw means just raising a specific type of exception; the usual rules concerning `try/except/else/finally` and `with` blocks apply. It is a function call, so it works also in lambdas. +In Python terms, a throw (in the escape continuation sense) means just raising a specific type of exception; the usual rules concerning `try/except/else/finally` and `with` blocks apply. The `throw` is a function call, so it works also in lambdas. -Escaping the function surrounding an FP loop, from inside the loop: +For another example, here we return from the function surrounding an FP loop, from inside the loop: ```python @catch() @@ -3394,24 +3394,28 @@ def foo(): assert foo() == 15 ``` -For details on tagging, especially how untagged and tagged throw and catch points interact, and how to make one-to-one connections, see the docstring for `@catch`. +For details on tagging, especially how untagged and tagged throw and catch points interact, and how to make one-to-one connections, see the docstring for `@catch`. See also `call_ec` (below), which is a compact syntax to make a one-to-one connection. + +**CAUTION**: The implementation is based on exceptions, so catch-all `except:` statements will intercept also throws, breaking the escape mechanism. As you already know, be specific in which exception types you catch in an `except` clause! **Etymology** -This feature is known as `catch`/`throw` in several Lisps, e.g. in Emacs Lisp and in Common Lisp (as well as some of its ancestors). This terminology is independent of the use of `throw`/`catch` in C++/Java for the exception handling mechanism. Common Lisp also provides a lexically scoped variant (`BLOCK`/`RETURN-FROM`) that is more idiomatic [according to Seibel](http://www.gigamonkeys.com/book/the-special-operators.html). +This feature is known as `catch`/`throw` in several Lisps, e.g. in Emacs Lisp and in Common Lisp (as well as some of its ancestors). This terminology is independent of the use of `throw`/`catch` in C++/Java for the exception handling mechanism. + +Common Lisp also provides a lexically scoped variant (`BLOCK`/`RETURN-FROM`) that is more idiomatic ([according to Seibel](http://www.gigamonkeys.com/book/the-special-operators.html)), but we currently provide only this dynamic variant. #### `call_ec`: first-class escape continuations -We provide `call/ec` (a.k.a. `call-with-escape-continuation`), in Python spelled as `call_ec`. It's a decorator that, like `@call`, immediately runs the function and replaces the def'd name with the return value. The twist is that it internally sets up a catch point, and hands a **first-class escape continuation** to the callee. +We provide the function `call/ec` (a.k.a. [`call-with-escape-continuation`](https://docs.racket-lang.org/reference/cont.html#(def._((quote._~23~25kernel)._call-with-escape-continuation)))), in Python spelled as `call_ec`. It's a decorator that, like `@call`, immediately runs the function and replaces the def'd name with the return value. The twist is that it internally sets up a catch point, and hands a **first-class escape continuation** to the callee. -The function to be decorated **must** take one positional argument, the ec instance. +The function to be decorated **must** take one positional argument, the ec instance. The parameter is conventionally named `ec`. -The ec instance itself is another function, which takes one positional argument: the value to send to the catch point. The ec instance and the catch point are connected one-to-one. No other `@catch` point will catch the ec instance, and the catch point catches only this particular ec instance and nothing else. +The ec instance itself is another function, which takes one positional argument: the value to send to the catch point. That value can also be a `Values` object if you want to escape with multiple-return-values or named return values; the ec will send any argument given to it. -Any particular ec instance is only valid inside the dynamic extent of the `call_ec` invocation that created it. Attempting to call the ec later raises `RuntimeError`. +The ec instance and the catch point are connected one-to-one. No other `@catch` point will catch the ec instance, and the catch point catches only the ec instances created by this invocation of `call_ec`, and nothing else. -This builds on `@catch` and `throw`, so the caution about catch-all `except:` statements applies here, too. +Any particular ec instance is only valid inside the dynamic extent of the `call_ec` invocation that created it. Attempting to call the ec later raises `RuntimeError`. ```python from unpythonic import call_ec @@ -3438,7 +3442,7 @@ def result(ec): assert result == 42 ``` -The ec doesn't have to be called from the lexical scope of the call_ec'd function, as long as the call occurs within the dynamic extent of the `call_ec`. It's essentially a *return from me* for the original function: +The ec does not have to be called from the lexical scope of the `call_ec`'d function, as long as the call occurs *within the dynamic extent* of the `call_ec`. It's essentially a *return from me* for the original function: ```python def f(ec): @@ -3466,7 +3470,7 @@ Normally `begin()` would return the last value, but the ec overrides that; it is But wait, doesn't Python evaluate all the arguments of `begin(...)` before the `begin` itself has a chance to run? Why doesn't the example print also *never reached*? This is because escapes are implemented using exceptions. Evaluating the ec call raises an exception, preventing any further elements from being evaluated. -This usage is valid with named functions, too - `call_ec` is not only a decorator: +This usage is valid with named functions, too, so strictly speaking, `call_ec` is not only a decorator: ```python def f(ec): @@ -3480,6 +3484,10 @@ result = call_ec(f) assert result == 42 ``` +*If you use the macro API of `unpythonic`, be aware that the macros cannot analyze this last example properly, because there is no lexical clue that `f` will actually be called using `call_ec`. To be safe in situations like this, name your ec parameter `ec`; then it will be recognized as an escape continuation. Also `brk` (defined by `@looped_over`) and `throw` are recognized by name.* + +**CAUTION**: The `call_ec` mechanism builds on `@catch` and `throw`, so the caution about catch-all `except:` statements applies here, too. + ### `forall`: nondeterministic evaluation From 9541e68b514424a07d322d42b7f314b2b7439f13 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:14:31 +0300 Subject: [PATCH 545/832] improve forall docs --- doc/features.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/features.md b/doc/features.md index 03dcf2af..b544da0f 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3493,9 +3493,9 @@ assert result == 42 We provide a simple variant of nondeterministic evaluation. This is essentially a toy that has no more power than list comprehensions or nested for loops. See also the easy-to-use [macro](macros.md) version with natural syntax and a clean implementation. -An important feature of McCarthy's [`amb` operator](https://rosettacode.org/wiki/Amb) is its nonlocality - being able to jump back to a choice point, even after the dynamic extent of the function where that choice point resides. If that sounds a lot like `call/cc`, that's because that's how `amb` is usually implemented. See examples [in Ruby](http://www.randomhacks.net/2005/10/11/amb-operator/) and [in Racket](http://www.cs.toronto.edu/~david/courses/csc324_w15/extra/choice.html). +An important feature of McCarthy's [`amb` operator](https://rosettacode.org/wiki/Amb) is its nonlocality - being able to jump back to a choice point, even after the dynamic extent of the function where that choice point resides. If that sounds a lot like `call/cc`, that is because that's how `amb` is usually implemented. See examples [in Ruby](http://www.randomhacks.net/2005/10/11/amb-operator/) and [in Racket](http://www.cs.toronto.edu/~david/courses/csc324_w15/extra/choice.html). -Python can't do that, short of transforming the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), while applying TCO everywhere to prevent stack overflow. **If that's what you want**, see `continuations` in [the macros](macros.md). +Python cannot do that, short of transforming the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), while applying TCO everywhere to prevent stack overflow. **If that is what you want**, see `continuations` in [the macros](macros.md). This `forall` is essentially a tuple comprehension that: @@ -3505,7 +3505,7 @@ This `forall` is essentially a tuple comprehension that: The `unpythonic.amb` module defines four operators: - - `forall` is the control structure, which marks a section with nondeterministic evaluation. + - `forall` is the control structure, which marks a section that uses nondeterministic evaluation. - `choice` binds a name: `choice(x=range(3))` essentially means `for e.x in range(3):`. - `insist` is a filter, which allows the remaining lines to run if the condition evaluates to truthy. - `deny` is `insist not`; it allows the remaining lines to run if the condition evaluates to falsey. @@ -3544,13 +3544,13 @@ assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), Beware: ```python -out = forall(range(2), # do the rest twice! +out = forall(range(2), # evaluate remaining items twice! choice(x=range(1, 4)), lambda e: e.x) assert out == (1, 2, 3, 1, 2, 3) ``` -The initial `range(2)` causes the remaining lines to run twice - because it yields two output values - regardless of whether we bind the result to a variable or not. In effect, each line, if it returns more than one output, introduces a new nested loop at that point. +The initial `range(2)` causes the remaining items to run twice - because it yields two output values - regardless of whether we bind the result to a variable or not. In effect, each line, if it returns more than one output, introduces a new nested loop at that point. For more, see the docstring of `forall`. From c92ca59ba65e9d8c91a58950e1d07e5a4eb11eca Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:15:26 +0300 Subject: [PATCH 546/832] improve condition system docs --- doc/features.md | 52 ++++++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 24 deletions(-) diff --git a/doc/features.md b/doc/features.md index b544da0f..c1323191 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3567,29 +3567,29 @@ The implementation is based on the List monad, and a bastardized variant of do-n ### `handlers`, `restarts`: conditions and restarts -**Added in v0.14.2**. +**Changed in v0.15.0.** *Functions `resignal_in` and `resignal` added; these perform the same job for conditions as `reraise_in` and `reraise` do for exceptions, that is, they allow you to map library exception types to semantically appropriate application exception types, with minimum boilerplate.* -**Changed in v0.14.3**. *Conditions can now inherit from `BaseException`, not only from `Exception.` `with handlers` catches also derived types, e.g. a handler for `Exception` now catches a signaled `ValueError`.* +*Upon an unhandled signal, `signal` now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report.* -*When an unhandled `error` or `cerror` occurs, the original unhandled error is now available in the `__cause__` attribute of the `ControlError` exception that is raised in this situation.* +*The error-handling protocol that was used to send a signal is now available for inspection in the `__protocol__` attribute of the condition instance. It is the callable that sent the signal, such as `signal`, `error`, `cerror` or `warn`. It is the responsibility of each error-handling protocol (except the fundamental `signal` itself) to pass its own function to `signal` as the `protocol` argument; if not given, `protocol` defaults to `signal`. The protocol information is used by the `resignal` mechanism.* -*Signaling a class, as in `signal(SomeExceptionClass)`, now implicitly creates an instance with no arguments, just like the `raise` statement does. On Python 3.7+, `signal` now automatically equips the condition instance with a traceback, just like the `raise` statement does for an exception.* +**Changed in v0.14.3**. *Conditions can now inherit from `BaseException`, not only from `Exception.` Just like the `except` statement, `with handlers` catches also derived types, e.g. a handler for `Exception` now catches a signaled `ValueError`.* -**Changed in v0.15.0.** *Functions `resignal_in` and `resignal` added; these perform the same job for conditions as `reraise_in` and `reraise` do for exceptions, that is, they allow you to map library exception types to semantically appropriate application exception types, with minimum boilerplate.* +*When an unhandled `error` or `cerror` occurs, the original unhandled error is now available in the `__cause__` attribute of the `ControlError` exception that is raised in this situation.* -*Upon an unhandled signal, `signal` now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report.* +*Signaling a class, as in `signal(SomeExceptionClass)`, now implicitly creates an instance with no arguments, just like the `raise` statement does. On Python 3.7+, `signal` now automatically equips the condition instance with a traceback, just like the `raise` statement does for an exception.* -*The error-handling protocol that was used to send a signal is now available for inspection in the `__protocol__` attribute of the condition instance. It is the callable that sent the signal, such as `signal`, `error`, `cerror` or `warn`. It is the responsibility of each error-handling protocol (except the fundamental `signal` itself) to pass its own function to `signal` as the `protocol` argument; if not given, `protocol` defaults to `signal`. The protocol information is used by the `resignal` mechanism.* +**Added in v0.14.2**. One of the killer features of Common Lisp are *conditions*, which are essentially **resumable exceptions**. -Following Peter Seibel ([Practical Common Lisp, chapter 19](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), we define *errors* as the consequences of [Murphy's Law](https://en.wikipedia.org/wiki/Murphy%27s_law), i.e. situations where circumstances cause interaction between the program and the outside world to fail. An error is no bug, but failing to handle an error certainly is. +Following Peter Seibel ([Practical Common Lisp, chapter 19](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), we define *errors* as the consequences of [Murphy's Law](https://en.wikipedia.org/wiki/Murphy%27s_law), i.e. situations where circumstances cause interaction between the program and the outside world to fail. An error is not a bug, but failing to handle an error certainly is. An exception system splits error-recovery responsibilities into two parts. In Python terms, we speak of *raising* and then *handling* an exception. In comparison, a condition system splits error-recovery responsibilities into **three parts**: *signaling*, *handling* and *restarting*. -The result is improved modularity. Consider [separation of mechanism and policy](https://en.wikipedia.org/wiki/Separation_of_mechanism_and_policy). We place the actual error-recovery code (the mechanism) in *restarts*, at the inner level (of the call stack) - which has access to all the low-level technical details that are needed to actually perform the recovery. We can provide *several different* canned recovery strategies, which implement any appropriate ways to recover, in the context of each low- or middle-level function. We defer the decision of which one to use (the policy), *to an outer level*. The outer level knows about the big picture - *why* the inner levels are running in this particular case, i.e. what we are trying to accomplish and how. Hence, it is in the ideal position to choose which error-recovery strategy should be used *in its high-level context*. +The result is improved modularity and better [separation of mechanism and policy](https://en.wikipedia.org/wiki/Separation_of_mechanism_and_policy). The actual error-recovery code (the **mechanism**) lives in *restarts*, at the inner level (of the call stack) - which has access to all the low-level technical details that are needed to actually perform an error recovery. It is possible to provide *several different* canned recovery strategies, which implement any appropriate ways to recover, in the context of each low- or middle-level function. The decision of which strategy to use (the **policy**) in any particular situation is deferred *to an outer level* (of the call stack). The outer level knows the big picture - *why* the inner levels are running in this particular case, i.e., what we are trying to accomplish and how. Hence, it is the appropriate place to choose which error-recovery strategy should be used *in its high-level context*. -Practical Common Lisp explains conditions in the context of a log file parser. In contrast, let us explain them with some Theoretical Python: +Seibel's *Practical Common Lisp* explains conditions in the context of a log file parser. In contrast, let us explain them with some *Theoretical Python*: ```python from unpythonic import restarts, handlers, signal, invoke, unbox @@ -3632,19 +3632,21 @@ high3() #### Fundamental signaling protocol -Generally a condition system operates as follows. A *signal* is sent (outward on the call stack) from the actual location where the error was detected. A *handler* at any outer level may then respond to it, and execution resumes from the *restart* that is *invoked* by the handler. +Generally a conditions-and-restarts system operates as follows. A *signal* is sent, outward on the call stack, from the actual location where an error was detected. A *handler* at any outer level (of the call stack) may then respond to it, and execution resumes from the *restart* that is *invoked* by the handler. -The sequence of catching a signal and invoking a restart is termed *handling* the signal. Handlers are searched in order from innermost to outermost on the call stack. (Strictly speaking, the handlers live on a separate stack; we consider those handlers whose dynamic extent the point of execution is in, at the point of time when the signal is sent.) +The sequence of catching a signal and invoking a restart is termed *handling* the signal. Handlers are searched in order from innermost to outermost on the call stack. Strictly speaking, though, the handlers live on a separate stack; we consider those handler bindings whose dynamic extent the point of execution is in, at the point of time when the signal is sent. In general, it is allowed for a handler to fall through (return normally); then the next outer handler for the same signal type gets control. This allows the programmer to chain handlers to obtain their side effects, such as logging. This is referred to as *canceling*, since as a result, the signal remains unhandled. -Viewed with respect to the call stack, the restarts live between the (outer) level of the handler, and the (inner) level where the signal was sent from. The main difference to the exception model is that unlike raising an exception, **sending a signal does not unwind the call stack**. Although the handlers live further out on the call stack, the stack does not unwind that far. The handlers are just consulted for what to do. The call stack unwinds only when a restart is being invoked. Then, only the part of the call stack between the location that sent the signal, and the invoked restart, is unwound. +Viewed with respect to the call stack, the restarts live between the (outer) level of the handler, and the (inner) level where the signal was sent from. The main difference to the exception model is that unlike raising an exception, **sending a signal does not unwind the call stack**. (Let that sink in for a moment.) + +Although the handlers live further out on the call stack, the stack does not unwind that far. The handlers are just consulted for what to do. **The call stack unwinds only when a restart is invoked.** Then, only the part of the call stack between the location that sent the signal, and the invoked restart, is unwound. -Restarts, despite the name, are a mildly behaved, structured control construct. The block of code that encountered the error is actually not arbitrarily resumed; instead, the restart code runs instead of the rest of the block, and the return value of the restart replaces the normal return value. (But see `cerror`.) +Restarts, despite the name, are a mildly behaved, structured control construct. The block of code that encountered the error is actually not arbitrarily resumed; instead, the code of the invoked restart runs instead of the rest of the block, and the return value of the restart replaces the normal return value. (But see `cerror`.) #### API summary -Restarts are set up using the `with restarts` context manager (Common Lisp: `RESTART-CASE`). Restarts are defined by giving named arguments to the `restarts` form; the argument name sets the restart name. The restart name is distinct from the name (if any) of the function that is used as the restart. A restart can only be invoked from within the dynamic extent of its `with restarts` (the same rule is effect also in Common Lisp). A restart may take any args and kwargs; any that it expects must be provided when it is invoked. +Restarts are set up using the `with restarts` context manager (Common Lisp: `RESTART-CASE`). Restarts are defined by passing named arguments to the `restarts` form; the argument name sets the *restart name*. The restart name is distinct from the name (if any) of the function that is used as the restart. A restart can only be invoked from within the dynamic extent of its `with restarts` (the same rule is effect also in Common Lisp). A restart may take any args and kwargs; any that it expects must be provided when it is invoked. *Note difference to the API of [python-cl-conditions](https://github.com/svetlyak40wt/python-cl-conditions/), which requires functions used as restarts to be named, and uses the function name as the restart name.* @@ -3654,13 +3656,13 @@ Signals are sent using `signal` (Common Lisp: `SIGNAL`). Any exception or warnin Handlers are established using the `with handlers` context manager (Common Lisp: `HANDLER-BIND`). Handlers are bound to exception types, or tuples of types, just like regular exception handlers in Python. The `handlers` form takes as its arguments any number of `(exc_spec, handler)` pairs. Here `exc_spec` specifies the exception types to catch (when sent via `signal`), and `handler` is a callable. When catching a signal, in case of multiple matches in the same `with handlers` form, the handler that appears earlier in the argument list wins. -A handler catches signals of the types it is bound to. The code in the handler may invoke a restart by calling `invoke` (Common Lisp: `INVOKE-RESTART`), with the desired restart name as a string. In case of duplicate names, the most recently established restart (that is still in scope) with the given name wins. Any extra args and kwargs are passed through to the restart. The `invoke` function always transfers control, never returns normally. +A handler catches signals of the types it is bound to, and their subtypes. The code in the handler may invoke a restart by calling `invoke` (Common Lisp: `INVOKE-RESTART`), with the desired restart name as a string. In case of duplicate names, the most recently established restart (that is still in scope) with the given name wins. Any extra args and kwargs are passed through to the restart. The `invoke` function always transfers control, it never returns normally. -A handler **may** take one optional positional argument, the exception instance being signaled. Roughly, API-wise signal handlers are similar to exception handlers (`except` clauses). A handler that accepts an argument is like an `except ... as ...`, whereas one that does not is like `except ...`. **The main difference** to an exception handler is that a **signal handler should not try to recover from the error itself**; instead, **it should just choose** which strategy the lower-level code should use to recover from the error. Usually, the only thing a signal handler needs to do, is to invoke a particular restart. +A handler **may** take one optional positional argument, the exception instance being signaled. Roughly, API-wise signal handlers are similar to exception handlers (`except` clauses). A handler that accepts an argument is like an `except ... as ...`, whereas one that does not is like `except ...`. **The main difference** to an exception handler is that a **signal handler should not try to recover from the error itself**; instead, **it should just choose** which strategy the lower-level code should use to recover from the error. Usually, the only thing a signal handler needs to do is to invoke a particular restart. To create a simple handler that does not take an argument, and just invokes a pre-specified restart, see `invoker`. If you instead want to create a function that you can call from a handler, in order to invoke a particular restart immediately (so to define a shorthand notation similar to `use_value`), use `functools.partial(invoke, "my_restart_name")`. -Following Common Lisp terminology, *a named function that invokes a specific restart* - whether it is intended to act as a handler or to be called from one - is termed a *restart function*. (This is somewhat confusing, as a *restart function* is not a function that implements a restart, but a function that *invokes* a specific one.) The `use_value` function mentioned above is an example. +Following Common Lisp terminology, *a named function that invokes a specific restart* - whether it is intended to act as a handler or to be called from one - is termed a *restart function*. This is somewhat confusing, as a *restart function* is not a function that implements a restart, but a function that *invokes* a specific one. The `use_value` function mentioned above is an example. For a detailed API reference, see the module `unpythonic.conditions`. @@ -3668,7 +3670,7 @@ For a detailed API reference, see the module `unpythonic.conditions`. We actually provide four signaling protocols: `signal` (i.e. the fundamental protocol), and three that build additional behavior on top of it: `error`, `cerror` and `warn`. Each of the three is modeled after its Common Lisp equivalent. -If no handler *handles* the signal, the `signal(...)` protocol just returns normally. In effect, with respect to control flow, unhandled signals are ignored by this protocol. (But any side effects of handlers that caught the signal but did not invoke a restart, still take place.) +If no handler *handles* the signal, the `signal(...)` protocol just returns normally. In effect, with respect to control flow, unhandled signals are ignored by this protocol. However, any side effects of handlers that caught the signal but did not invoke a restart, still take place. The `error(...)` protocol first delegates to `signal`, and if the signal was not handled by any handler, then **raises** `ControlError` as a regular exception. (Note the Common Lisp `ERROR` function would at this point drop you into the debugger.) The implementation of `error` itself is the only place in the condition system that *raises* an exception for the end user; everything else (including any error situations) uses the signaling mechanism. @@ -3678,17 +3680,19 @@ Finally, there is the `warn(...)` protocol, which is just a lispy interface to P The combination of `warn` and `muffle` (as well as `cerror` when a handler invokes its `proceed` restart) behaves somewhat like [`contextlib.suppress`](https://docs.python.org/3/library/contextlib.html#contextlib.suppress), except that execution continues normally from the next statement in the caller of `warn` (respectively `cerror`) instead of unwinding to the handler. -If the standard protocols don't cover what you need, you can also build your own high-level protocols on top of `signal`. See the source code of `error`, `cerror` and `warn` for examples (it's just a few lines in each case). +If the standard protocols do not cover what you need, you can also build your own high-level protocols on top of `signal`. See the source code of `error`, `cerror` and `warn` for examples (it's just a few lines in each case). ##### Notes The name `cerror` stands for *correctable error*, see e.g. [CERROR in the CL HyperSpec](http://clhs.lisp.se/Body/f_cerror.htm). What we call `proceed`, Common Lisp calls `CONTINUE`; the name is different because in Python the function naming convention is lowercase, and `continue` is a reserved word. -If you really want to emulate `ON ERROR RESUME NEXT`, just use `Exception` as the condition type for your handler, and all `cerror` calls within the block will return normally, provided that no other handler handles those conditions first. +If you really want to emulate `ON ERROR RESUME NEXT`, just use `Exception` as the condition type for your handler, and all `cerror` calls within the block will return normally, provided that no other handler (that appears in an inner position on the call stack) handles those conditions first. #### Conditions vs. exceptions -Using the condition system essentially requires eschewing exceptions, using only restarts and handlers instead. A regular `raise` will fly past a `with handlers` form uncaught. The form just maintains a stack of functions; it does not establish an *exception* handler. Similarly, a `try`/`except` cannot catch a signal, because no exception is raised yet at handler lookup time. Delaying the stack unwind, to achieve the three-way split of responsibilities, is the whole point of the condition system. Which of the two systems to use is a design decision that must be made consistently on a per-project basis. +Using the condition system essentially requires eschewing exceptions, using only restarts and handlers instead. A regular `raise` will fly past a `with handlers` form uncaught. The form just maintains a stack of functions; it does not establish an *exception* handler. Similarly, a `try`/`except` cannot catch a signal, because no exception is raised yet at handler lookup time. Delaying the stack unwind, to achieve the three-way split of responsibilities, is the whole point of the condition system. + +Which of the two systems to use is a design decision that must be made consistently on a per-project basis. Even better would be to make it globally on a per-language basis. Python's standard library, as well as all existing libraries, use exceptions instead of conditions, so to obtain a truly seamless conditions-and-restarts user experience, one would have to wrap (or rewrite) at least all of the standard library, plus any other libraries a project needs, to be protected from sudden, unexpected unwinds of the call stack. (The nature of both conditions and exceptions is that, in principle, they may be triggered anywhere.) Be aware that error-recovery code in a Lisp-style signal handler is of a very different nature compared to error-recovery code in an exception handler. A signal handler usually only chooses a restart and invokes it; as was explained above, the code that actually performs the error recovery (i.e. the *restart*) lives further in on the call stack, and still has available (in its local variables) the state that is needed to perform the recovery. An exception handler, on the other hand, must respond by directly performing error recovery right where it is, without any help from inner levels - because the stack has already unwound when the exception handler gets control. @@ -3702,13 +3706,13 @@ If this `ControlError` signal is not handled, a `ControlError` will then be **ra #### Historical note -Conditions are one of the killer features of Common Lisp, so if you're new to conditions, [Peter Seibel: Practical Common Lisp, chapter 19](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) is a good place to learn about them. There's also a relevant [discussion on Lambda the Ultimate](http://lambda-the-ultimate.org/node/1544). +Conditions are one of the killer features of Common Lisp, so if you are new to conditions, [Peter Seibel: Practical Common Lisp, chapter 19](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) is a good place to learn about them. There is also a relevant [discussion on Lambda the Ultimate](http://lambda-the-ultimate.org/node/1544). For Python, conditions were first implemented in [python-cl-conditions](https://github.com/svetlyak40wt/python-cl-conditions/) by Alexander Artemenko (2016). What we provide here is essentially a rewrite, based on studying that implementation. The main reasons for the rewrite are to give the condition system an API consistent with the style of `unpythonic`, to drop any and all historical baggage without needing to consider backward compatibility, and to allow interaction with (and customization taking into account) the other parts of `unpythonic`. -The core idea can be expressed in fewer than 100 lines of Python; ours is (as of v0.14.2) 151 lines, not counting docstrings, comments, or blank lines. The main reason our module is over 700 lines are the docstrings. +The core idea can be expressed in fewer than 100 lines of Python; ours is (as of v0.15.0) 199 lines, not counting docstrings, comments, or blank lines. The main reason our module is over 900 lines are the docstrings. ### `generic`, `typed`, `isoftype`: multiple dispatch From 063f210270c4c6a17c645cc6441758d383f78405 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:15:39 +0300 Subject: [PATCH 547/832] error message wording --- unpythonic/conditions.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 56907434..9dec030d 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -218,7 +218,7 @@ def canonize(exc, err_reason): return exc() # instantiate with no args, like `raise` does except TypeError: # "issubclass() arg 1 must be a class" pass - error(ControlError(f"Only exceptions and subclasses of Exception can {err_reason}; got {type(condition)} with value {repr(condition)}.")) + error(ControlError(f"Only instances (derived too) and subclasses of BaseException can {err_reason}; got {type(condition)} with value {repr(condition)}.")) condition = canonize(condition, "be signaled") cause = canonize(cause, "act as the cause of another signal") From ed3b5b804ef898022f686e8595964b9a4b7404ef Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:15:49 +0300 Subject: [PATCH 548/832] fix docstring --- unpythonic/conditions.py | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 9dec030d..b17af772 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -869,17 +869,23 @@ def _resignal_handler(mapping, condition): `mapping`: dict-like, `{LibraryExc0: ApplicationExc0, ...}` - Each `LibraryExc` must be a signal type. + Each `LibraryExc` must be an exception type or a tuple of + exception types. It will be matched using `isinstance`. - Each `ApplicationExc` can be a condition type or an instance. - If an instance, then that exact instance is signaled as the - converted condition. + Each `ApplicationExc` can be an exception type or an exception + instance. If an instance, then that exact instance is signaled + as the converted signal. - `libraryexc`: the signal instance to convert. It is - automatically chained into `ApplicationExc`. + `condition`: the exception instance that was signaled, and is to + be converted (if it matches an entry in `mapping`). + When converted, it is automatically chained into + an `ApplicationExc` signal. - This function never returns normally. If no key in the mapping - matches, this delegates to the next outer handler. + Conversions in `mapping` are tried in the order specified; hence, + just like in `with handlers`, place more specific types first. + + If no key in the mapping matches, this delegates to the next outer + signal handler. """ for LibraryExc, ApplicationExc in mapping.items(): if isinstance(condition, LibraryExc): From 51e5e44ff3a2a0586d79908d660426bef29e07a6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:17:41 +0300 Subject: [PATCH 549/832] mention in changelog and doc that `memoize` is now thread-safe --- CHANGELOG.md | 2 +- doc/features.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3caae4a6..1db2931a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -204,7 +204,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Fix bug: `fup`/`fupdate`/`ShadowedSequence` now actually accept an infinite-length iterable as a replacement sequence (under the obvious usage limitations), as the documentation has always claimed. -- Fix bug: `memoize` is now thread-safe. +- Fix bug: `memoize` is now thread-safe. Even when the same memoized function instance is called concurrently from multiple threads. Exactly one thread will compute the result. If `f` is recursive, the thread that acquired the lock is the one that is allowed to recurse into the memoized `f`. --- diff --git a/doc/features.md b/doc/features.md index c1323191..4cc98ea6 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1405,6 +1405,8 @@ assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) #### `memoize` +**Changed in v0.15.0.** *Fix bug: `memoize` is now thread-safe. Even when the same memoized function instance is called concurrently from multiple threads. Exactly one thread will compute the result. If `f` is recursive, the thread that acquired the lock is the one that is allowed to recurse into the memoized `f`.* + [*Memoization*](https://en.wikipedia.org/wiki/Memoization) is a functional programming technique, meant to be used with [pure functions](https://en.wikipedia.org/wiki/Pure_function). It caches the return value, so that *for each unique set of arguments*, the original function will be evaluated only once. All arguments must be hashable. Our `memoize` caches also exceptions, à la the [Mischief package in Racket](https://docs.racket-lang.org/mischief/memoize.html). If the memoized function is called again with arguments with which it raised an exception the first time, **that same exception instance** is raised again. From 71705b855b39babc685777860c6c9662508669a9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Jun 2021 20:20:34 +0300 Subject: [PATCH 550/832] add holy traits link to README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 16d0d0de..db7cedb0 100644 --- a/README.md +++ b/README.md @@ -246,7 +246,7 @@ def my_range(start: int, step: int, stop: int): This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. -*Holy traits* are also a possibility: +[*Holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) are also a possibility: ```python import typing From 5e7e398c947b20362ef2fc59649e1f0045393928 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:01:28 +0300 Subject: [PATCH 551/832] styling, wording --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index db7cedb0..07b5ad64 100644 --- a/README.md +++ b/README.md @@ -244,7 +244,7 @@ def my_range(start: int, step: int, stop: int): return start, step, stop ``` -This is a purely run-time implementation, so it doesn't give performance benefits, but it can make code more readable, and easily allows adding support for new input types to an existing function without monkey-patching the original. +This is a purely run-time implementation, so it does **not** give performance benefits, but it can make code more readable, and makes it modular to add support for new input types (or different call signatures) to an existing function later. [*Holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) are also a possibility: From 660cb32bc3ee5152e19fc3a2f715952acbbebc82 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:02:00 +0300 Subject: [PATCH 552/832] add link to descriptor docs --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 4cc98ea6..5ce3c1cc 100644 --- a/doc/features.md +++ b/doc/features.md @@ -855,7 +855,7 @@ A `Shim` is an *attribute access proxy*. The shim holds a `box` (or a `ThreadLoc For example, `Shim` can combo with `ThreadLocalBox` to redirect standard output only in particular threads. Place the stream object in a `ThreadLocalBox`, shim that box, then replace `sys.stdout` with the shim. See the source code of `unpythonic.net.server` for an example that actually does (and cleanly undoes) this. -Since deep down, attribute access is the whole point of objects, `Shim` is essentially a transparent object proxy. (For example, a method call is an attribute read (via a descriptor), followed by a function call.) +Since deep down, attribute access is the whole point of objects, `Shim` is essentially a transparent object proxy. (For example, a method call is an attribute read (via a [descriptor](https://docs.python.org/3/howto/descriptor.html)), followed by a function call.) ```python from unpythonic import Shim, box, unbox From 84743b3c14ac29d0fe4954410650d215d8a46223 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:02:16 +0300 Subject: [PATCH 553/832] improve error message in example --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 5ce3c1cc..1614000f 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1463,7 +1463,7 @@ There are some **important differences** to the nearest equivalents in the stand return thrice_int(x) elif isinstance(x, float): return thrice_float(x) - raise TypeError(type(x)) + raise TypeError(f"unsupported argument: {type(x)} with value {repr(x)}") @memoize def thrice_int(x): From 54e3fce51ab133726660a63a25d45e8900958ff3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:12:26 +0300 Subject: [PATCH 554/832] 0.15.0: improve multiple-dispatch docs --- doc/features.md | 207 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 180 insertions(+), 27 deletions(-) diff --git a/doc/features.md b/doc/features.md index 1614000f..11b94f10 100644 --- a/doc/features.md +++ b/doc/features.md @@ -91,6 +91,10 @@ The exception are the features marked **[M]**, which are primarily intended as a - [High-level signaling protocols](#high-level-signaling-protocols) - [Conditions vs. exceptions](#conditions-vs-exceptions) - [`generic`, `typed`, `isoftype`: multiple dispatch](#generic-typed-isoftype-multiple-dispatch): create generic functions with type annotation syntax; also some friendly utilities. + - [`generic`: multiple dispatch with type annotation syntax](#generic-multiple-dispatch-with-type-annotation-syntax) + - [`augment`: add a new multimethod to an existing generic function](#augment-add-a-new-multimethod-to-an-existing-generic-function) + - [`typed`: add run-time type checks with type annotation syntax](#typed-add-run-time-type-checks-with-type-annotation-syntax) + - [`isoftype`: the big sister of `isinstance`](#isoftype-the-big-sister-of-isinstance) [**Exception tools**](#exception-tools) - [`raisef`, `tryf`: `raise` and `try` as functions](#raisef-tryf-raise-and-try-as-functions), useful inside a lambda. @@ -3719,31 +3723,29 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of ### `generic`, `typed`, `isoftype`: multiple dispatch -**Added in v0.14.2**. - -**Changed in v0.14.3**. *The multiple-dispatch decorator `@generic` no longer takes a master definition. Multimethods are registered directly with `@generic`; the first method definition implicitly creates the generic function.* - -**Changed in v0.14.3**. *The `@generic` and `@typed` decorators can now decorate also instance methods, class methods and static methods (beside regular functions, as previously in 0.14.2).* - **Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Starting with this release, they receive the same semantic-versioning guarantees as the rest of `unpythonic`.* -*Added the `@augment` parametric decorator that can register a new multimethod on an existing generic function originally defined in another lexical scope. Be careful of [type piracy](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy) when you use it.* +*Added the `@augment` parametric decorator that can register a new multimethod on an existing generic function originally defined in another lexical scope.* -*Added the function `methods`, which displays a list of multimethods of a generic function.* +*Added the function `methods`, which displays a list of multimethods of a generic function. This is especially useful in the REPL.* *Docstrings of the multimethods are now automatically concatenated to make up the docstring of the generic function, so you can document each multimethod separately.* -*`curry` now supports `@generic`. In the case where the **number** of positional arguments supplied so far matches at least one multimethod, but there is no match for the given combination of argument **types**, `curry` waits for more arguments (returning the curried function).* +*`curry` now supports `@generic`. In the case where the **number** of positional arguments supplied so far matches at least one multimethod, but there is no match for the given combination of argument **types**, `curry` waits for more arguments (returning the curried function). See the manual section on `curry` for details.* *It is now possible to dispatch also on a homogeneous type of contents collected by a `**kwargs` parameter. In the type signature, use `typing.Dict[str, mytype]`. Note that in this use, the key type is always `str`.* -The `generic` decorator allows creating multiple-dispatch generic functions with type annotation syntax. We also provide some friendly utilities: `augment` adds a new multimethod to an existing generic function, `typed` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type checking code), and `isoftype` (which powers the first three) is the big sister of `isinstance`, with support for many (but unfortunately not all) features of the `typing` standard library module. +**Changed in v0.14.3**. *The multiple-dispatch decorator `@generic` no longer takes a master definition. Multimethods are registered directly with `@generic`; the first multimethod definition implicitly creates the generic function.* + +*The `@generic` and `@typed` decorators can now decorate also instance methods, class methods and static methods (beside regular functions, as previously in 0.14.2).* -For what kind of things can be done with this, see particularly the [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/) example in [`unpythonic.tests.test_dispatch`](../unpythonic/tests/test_dispatch.py). +**Added in v0.14.2**. + +The `generic` decorator allows creating [multiple-dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) generic functions with type annotation syntax. We also provide some friendly utilities: `augment` adds a new multimethod to an existing generic function, `typed` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type-checking code), and `isoftype` (which powers the first three) is the big sister of `isinstance`, with support for many (but unfortunately not all) features of the `typing` standard library module. -**NOTE**: This was inspired by the [multi-methods of CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) (the Common Lisp Object System), and the [generic functions of Julia](https://docs.julialang.org/en/v1/manual/methods/). +This is a purely run-time implementation, so it does **not** give performance benefits, but it can make code more readable, and makes it modular to add support for new input types (or different call signatures) to an existing function later. -In `unpythonic`, the terminology is as follows: +The terminology is: - The function that supports multiple call signatures is a *generic function*. - Each of its individual implementations is a *multimethod*. @@ -3755,10 +3757,12 @@ The term *multimethod* distinguishes them from the OOP sense of *method*, alread #### `generic`: multiple dispatch with type annotation syntax -The `generic` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. This not only kills boilerplate, but makes the dispatch extensible, since the dispatcher lives outside the original function definition. There is no need to monkey-patch the original to add a new case. +The `generic` decorator essentially allows replacing the `if`/`elif` dynamic type checking boilerplate of polymorphic functions with type annotations on the function parameters, with support for features from the `typing` stdlib module. This not only kills boilerplate, but makes the dispatch extensible, since the dispatcher is separate from the actual function definition, and has a mechanism to register new multimethods. If several multimethods of the same generic function match the arguments given, the most recently registered multimethod wins. +To see what multimethods are registered on a given generic function `f`, call `methods(f)`. It will print a human-readable description to stdout. + **CAUTION**: The winning multimethod is chosen differently from Julia, where the most specific multimethod wins. Doing that requires a more careful type analysis than what we have here. The details are best explained by example: @@ -3832,35 +3836,171 @@ assert kittify(x=1, y=2) == "int" assert kittify(x=1.0, y=2.0) == "float" ``` -See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see `isoftype` below. - ##### `@generic` and OOP -As of version 0.14.3, `@generic` and `@typed` can decorate instance methods, class methods and static methods (beside regular functions as in 0.14.2). +Beginning with v0.14.3, `@generic` and `@typed` can decorate instance methods, class methods and static methods (beside regular functions as in v0.14.2). -When using both `@generic` or `@typed` and OOP: +When using both `@generic` or `@typed` and OOP, important things to know are: + + - In case of `@generic`, consider first if that is what you really want. + - The method access syntax already hides a single-dispatch mechanism behind the dot-access syntax: the syntax `x.op(...)` picks the definition of `op` based on the type of `x`. This behaves exactly like a single-dispatch function where the first argument is `x`, i.e., we could as well write `op(x, ...)`. + - So the question to ask is, is the use case best served by two overlapping dispatch mechanisms? + - If not, what are the alternative strategies? Would it be better, for example, to represent the operations as top-level `@generic` *functions*, and perform the dispatch there, dispatching to OOP methods as appropriate? + - `@typed` is fine to use with OOP, because semantically, it is not really a dispatch mechanism, but a run-time type-checking mechanism, even though it is implemented in terms of the multiple-dispatch machinery. - **`self` and `cls` parameters**. - The `self` and `cls` parameters do not participate in dispatching, and need no type annotation. - - Beside appearing as the first positional-or-keyword parameter, the self-like parameter **must be named** one of `self`, `this`, `cls`, or `klass` to be detected by the ignore mechanism. This limitation is due to implementation reasons; while a class body is being evaluated, the context needed to distinguish a method (OOP sense) from a regular function is not yet present. + - Beside appearing as the first positional-or-keyword parameter, the self-like parameter **must be named** one of `self`, `this`, `cls`, or `klass` to be detected by the ignore mechanism. + + This limitation is due to implementation reasons; while a class body is being evaluated, the context needed to distinguish a method (in the OOP sense) from a regular function is not yet present. In Python, OOP method binding is performed by the [descriptor](https://docs.python.org/3/howto/descriptor.html) that triggers when the method attribute is read on an instance. + + If curious, try this (tested in Python 3.8): + + ```python + class Thing: + def f(self): + pass + + print(type(Thing.f)) # --> "function", i.e. the same type as a bare function + assert Thing.f is Thing.f # it's always the same function object + + thing = Thing() + print(type(thing.f)) # --> "method", i.e. a bound method of Thing instance at 0x... + assert thing.f is not thing.f # each read produces a **new** bound method object + + lst = [1, 2, 3] + print(type(lst.append)) # --> "builtin_function_or_method" + assert lst.append is not lst.append # this happens even for builtins + ``` - **OOP inheritance**. - When `@generic` is installed on a method (instance method, or `@classmethod`), then at call time, classes are tried in [MRO](https://en.wikipedia.org/wiki/C3_linearization) order. All multimethods of the method defined in the class currently being looked up are tested for matches first, before moving on to the next class in the MRO. This has subtle consequences, related to in which class in the hierarchy the various multimethods for a particular method are defined. - To work with OOP inheritance, `@generic` must be the outermost decorator (except `@classmethod` or `@staticmethod`, which are essentially compiler annotations). - - However, when installed on a `@staticmethod`, the `@generic` decorator does not support MRO lookup, because that would make no sense. See discussions on interaction between `@staticmethod` and `super` in Python: [[1]](https://bugs.python.org/issue31118) [[2]](https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879). + - However, when installed on a `@staticmethod`, the `@generic` decorator does not support MRO lookup, because that would make no sense. A static method is just a bare function that happens to be stored in a class namespace. See discussions on the interaction between `@staticmethod` and `super` in Python: [[1]](https://bugs.python.org/issue31118) [[2]](https://stackoverflow.com/questions/26788214/super-and-staticmethod-interaction/26807879). + - When inspecting an **instance method** that is `@generic`, be sure to call the `methods` function **on an instance**: -##### Notes + ```python + class Thing: + @generic + def f(self, x: int): + pass -In both CLOS and in Julia, *function* is the generic entity, while *method* refers to its specialization to a particular combination of argument types. Note that *no object instance or class is needed*. Contrast with the classical OOP sense of *method*, i.e. a function that is associated with an object instance or class, with single dispatch based on the class (or in exotic cases, such as monkey-patched instances, on the instance). + @classmethod + @generic + def g(cls, x: int): + pass -Based on my own initial experiments with this feature, the machinery itself works well enough, but to really shine - just like resumable exceptions - multiple dispatch needs to be used everywhere, throughout the language's ecosystem. Python obviously doesn't do that. + thing = Thing() + methods(thing.f) -The machinery itself is also missing some advanced features, such as matching the most specific multimethod candidate instead of the most recently defined one; an `issubclass` equivalent that understands `typing` type specifications; and a mechanism to remove previously declared multimethods. + methods(Thing.g) + ``` -**CAUTION**: Multiple dispatch can be dangerous. Particularly, `@augment` can be dangerous to the readability of your codebase. If a new multimethod is added for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))). In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). Keep in mind that the multiple-dispatch table is global state! + This allows seeing registered multimethods also from linked dispatchers in the MRO. -If you need multiple dispatch, but not the other features of `unpythonic`, see the [multipledispatch](https://github.com/mrocklin/multipledispatch) library, which likely runs faster. + If we instead call it as `methods(Thing.f)`, the `self` argument is not bound yet (because `Thing.f` is just a bare function), so the dispatch machinery cannot get a reference to the MRO. This is obviously not an issue when actually using `f`, since an instance method is pretty much always invoked on an instance. + + For class methods, `methods(Thing.g)` sees the MRO, because `cls` is already bound. + +For usage examples of `@generic` with OOP, see [the unit tests](../unpythonic/tests/test_dispatch.py). + + +#### `augment`: add a new multimethod to an existing generic function + +The `@augment` decorator adds a new multimethod to an existing generic function. With this system, it is possible to implement [*holy traits*](https://ahsmart.com/pub/holy-traits-design-patterns-and-best-practice-book/): + +```python +import typing +from unpythonic import generic, augment + +class FunninessTrait: + pass +class IsFunny(FunninessTrait): + pass +class IsNotFunny(FunninessTrait): + pass + +@generic +def funny(x: typing.Any): # default + raise NotImplementedError(f"`funny` trait not registered for anything matching {type(x)}") + +@augment(funny) +def funny(x: str): # noqa: F811 + return IsFunny() +@augment(funny) +def funny(x: int): # noqa: F811 + return IsNotFunny() + +@generic +def laugh(x: typing.Any): + return laugh(funny(x), x) + +@augment(laugh) +def laugh(traitvalue: IsFunny, x: typing.Any): + return f"Ha ha ha, {x} is funny!" +@augment(laugh) +def laugh(traitvalue: IsNotFunny, x: typing.Any): + return f"{x} is not funny." + +assert laugh("that") == "Ha ha ha, that is funny!" +assert laugh(42) == "42 is not funny." +``` + +See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see `isoftype` below. + +**CAUTION**: `@augment` can be dangerous to the readability of your codebase. Keep in mind that the multiple-dispatch table is global state. If you add a new multimethod for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))), because it may change the meaning of existing code. In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). + +As Alexis King points out, no type piracy occurs if **at least one** of the following conditions holds: + + 1. At least one of the types in the call signature of the new multimethod is defined by you. + + 2. The generic function you are augmenting is defined by you. + + +##### How to augment a function that is not already `@generic` + +Given this: + +```python +# thirdparty.py +def op(x): + if isinstance(x, int): + return 2 * x + elif isinstance(x, float): + return 2.0 * x + raise TypeError(f"unsupported argument: {type(x)} with value {repr(x)}") +``` + +you do not have to change that code, but you will have to know which argument types the existing function supports (because that information is not available in an inspectable form at its interface), and then overwrite the original binding, with something like this: + +```python +# ours.py +import thirdparty + +original_op = thirdparty.op + +# Multimethod implementations for the types supported by the original `op`. +# We just re-dispatch here. +@generic +def op(x: int): + return original_op(x) +@generic +def op(x: float): + return original_op(x) + +thirdparty.op = op # unavoidable bit of monkey-patching +``` + +Then it can be augmented as usual: + +```python +@augment(op) +def op(x: str): # "ha" -> "ha, ha" + return ", ".join(x for _ in range(2)) +``` + +while preserving the meaning of all existing code that uses `thirdparty.op`. #### `typed`: add run-time type checks with type annotation syntax @@ -3962,7 +4102,20 @@ See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. **CAUTION**: The `isoftype` function is one big hack. In Python 3.6 through 3.9, there is no consistent way to handle a type specification at run time. We must access some private attributes of the `typing` meta-utilities, because that seems to be the only way to get what we need to do this. -If you need a run-time type checker, but not the other features of `unpythonic`, see the [`typeguard`](https://github.com/agronholm/typeguard) library. + +#### Notes + +The multiple-dispatch subsystem of `unpythonic` was inspired by the [multi-methods of CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) (the Common Lisp Object System), and the [generic functions of Julia](https://docs.julialang.org/en/v1/manual/methods/). + +In both CLOS and in Julia, *function* is the generic entity, while *method* refers to its specialization to a particular combination of argument types. Note that *no object instance or class is needed*. Contrast with the classical OOP sense of *method*, i.e. a function that is associated with an object instance or class, with single dispatch based on the class (or in exotic cases, such as monkey-patched instances, on the instance). + +Based on my own initial experiments with this feature in Python, the machinery itself works well enough, but to really shine - just like conditions and restarts - multiple dispatch needs to be used everywhere, throughout the language's ecosystem. Julia is impressive here. Python obviously does not do that. + +Our machinery is missing some advanced features, such as matching the most specific multimethod candidate instead of the most recently defined one; an `issubclass` equivalent that understands `typing` type specifications; and a mechanism to remove previously declared multimethods. + +*If you need multiple dispatch, but not the other features of `unpythonic`, see the [multipledispatch](https://github.com/mrocklin/multipledispatch) library, which likely runs faster.* + +*If you need a run-time type checker, but not the other features of `unpythonic`, see the [`typeguard`](https://github.com/agronholm/typeguard) library. If you are fine with a separate static type checker (which is the step where type checking arguably belongs), just use [`Mypy`](http://mypy-lang.org/).* ## Exception tools From 9dbe596a3cf95f4704a782ea5c42aa15f072315d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:16:23 +0300 Subject: [PATCH 555/832] wording --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 11b94f10..22afc796 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3664,7 +3664,7 @@ Handlers are established using the `with handlers` context manager (Common Lisp: A handler catches signals of the types it is bound to, and their subtypes. The code in the handler may invoke a restart by calling `invoke` (Common Lisp: `INVOKE-RESTART`), with the desired restart name as a string. In case of duplicate names, the most recently established restart (that is still in scope) with the given name wins. Any extra args and kwargs are passed through to the restart. The `invoke` function always transfers control, it never returns normally. -A handler **may** take one optional positional argument, the exception instance being signaled. Roughly, API-wise signal handlers are similar to exception handlers (`except` clauses). A handler that accepts an argument is like an `except ... as ...`, whereas one that does not is like `except ...`. **The main difference** to an exception handler is that a **signal handler should not try to recover from the error itself**; instead, **it should just choose** which strategy the lower-level code should use to recover from the error. Usually, the only thing a signal handler needs to do is to invoke a particular restart. +A handler **may** take one optional positional argument, the exception instance being signaled. Roughly, API-wise signal handlers are similar to exception handlers (`except` clauses). A handler that accepts an argument is like an `except ... as ...`, whereas one that does not is like `except ...`. **The main difference** to an exception handler is that a **signal handler should not try to recover from the error by itself**; instead, **it should just choose** which strategy the lower-level code should use to recover from the error. Usually, the only thing a signal handler needs to do is to invoke a particular restart. To create a simple handler that does not take an argument, and just invokes a pre-specified restart, see `invoker`. If you instead want to create a function that you can call from a handler, in order to invoke a particular restart immediately (so to define a shorthand notation similar to `use_value`), use `functools.partial(invoke, "my_restart_name")`. From edc1911ded3c48900942c00904e625833e863cee Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:24:20 +0300 Subject: [PATCH 556/832] change InvokeRestart, Escape to inherit from BaseException Now they are no longer inadvertently caught by `except Exception` handlers. --- CHANGELOG.md | 1 + unpythonic/conditions.py | 2 +- unpythonic/ec.py | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1db2931a..708e939f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -122,6 +122,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Asking for the `len` returns the current length of the memo. For subscripting, both a single `int` index and a slice are accepted. Note that memoized generators do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose. - `fup`/`fupdate`/`ShadowedSequence` can now walk the start of a memoized infinite replacement backwards. (Use `imemoize` on the original iterable, instantiate the generator, and use that generator instance as the replacement.) - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. + - The internal exception types `unpythonic.conditions.InvokeRestart` and `unpythonic.ec.Escape` now inherit from `BaseException`, so that they are not inadvertently caught by `except Exception` handlers. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). - CI: Test coverage improved to 94%. diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index b17af772..44088bc3 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -500,7 +500,7 @@ def __init__(self, *bindings): super().__init__(bindings) self.dq = _stacks.handlers -class InvokeRestart(Exception): +class InvokeRestart(BaseException): def __init__(self, restart, *args, **kwargs): # e is the context self.restart, self.a, self.kw = restart, args, kwargs # message when uncaught diff --git a/unpythonic/ec.py b/unpythonic/ec.py index c612aa31..303d3d5c 100644 --- a/unpythonic/ec.py +++ b/unpythonic/ec.py @@ -59,7 +59,7 @@ def throw(value, tag=None, allow_catchall=True): """ raise Escape(value, tag, allow_catchall) -class Escape(Exception): +class Escape(BaseException): """Exception that essentially represents the invocation of an escape continuation. Constructor parameters: see ``throw()``. From 670bb73d43ae9f1dac6fa4cf2da00dc282346a67 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:28:56 +0300 Subject: [PATCH 557/832] oops, misplaced note --- doc/features.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/features.md b/doc/features.md index 22afc796..7a996d73 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3836,6 +3836,9 @@ assert kittify(x=1, y=2) == "int" assert kittify(x=1.0, y=2.0) == "float" ``` +See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see `isoftype` below. + + ##### `@generic` and OOP Beginning with v0.14.3, `@generic` and `@typed` can decorate instance methods, class methods and static methods (beside regular functions as in v0.14.2). @@ -3947,8 +3950,6 @@ assert laugh("that") == "Ha ha ha, that is funny!" assert laugh(42) == "42 is not funny." ``` -See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see `isoftype` below. - **CAUTION**: `@augment` can be dangerous to the readability of your codebase. Keep in mind that the multiple-dispatch table is global state. If you add a new multimethod for a generic function defined elsewhere, for types defined elsewhere, this may lead to [*spooky action at a distance*](https://lexi-lambda.github.io/blog/2016/02/18/simple-safe-multimethods-in-racket/) (as in [action at a distance](https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming))), because it may change the meaning of existing code. In the Julia community, this is known as [*type piracy*](https://docs.julialang.org/en/v1/manual/style-guide/#Avoid-type-piracy). As Alexis King points out, no type piracy occurs if **at least one** of the following conditions holds: From a7d465566dc97c9b070adba180dc530e26f90d59 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 02:29:02 +0300 Subject: [PATCH 558/832] styling --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 7a996d73..df2016d0 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3902,7 +3902,7 @@ When using both `@generic` or `@typed` and OOP, important things to know are: This allows seeing registered multimethods also from linked dispatchers in the MRO. - If we instead call it as `methods(Thing.f)`, the `self` argument is not bound yet (because `Thing.f` is just a bare function), so the dispatch machinery cannot get a reference to the MRO. This is obviously not an issue when actually using `f`, since an instance method is pretty much always invoked on an instance. + If we instead call it as `methods(Thing.f)`, the `self` argument is not bound yet (because `Thing.f` is just a bare function), so the dispatch machinery cannot get a reference to the MRO. This is obviously not an issue when actually *using* `f`, since an instance method is pretty much always invoked on an instance. For class methods, `methods(Thing.g)` sees the MRO, because `cls` is already bound. From 9767e0345de4ac62af2bf7be681fd93bf9594d43 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 17 Jun 2021 15:52:28 +0300 Subject: [PATCH 559/832] readings: add Pyodide, scientific Python in the browser! --- doc/readings.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/doc/readings.md b/doc/readings.md index 4cbcbad6..7acdf802 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -84,10 +84,19 @@ The common denominator is programming. Some relate to language design, some to c - [PyPy3](http://pypy.org/), fast, JIT-ing Python 3 that's mostly a drop-in replacement for CPythons 3.6 and 3.7. As of April 2021, support for 3.8 is in the works. Macro expanders (`macropy`, `mcpyrate`) work, too. -- [Brython](https://brython.info/): Python 3 in the browser, as a replacement for JavaScript. - - No separate compile step - the compiler is implemented in JS. Including a script tag of type text/python invokes it. - - Doesn't have the `ast` module, so no way to run macro expanders. - - Also quite a few other parts are missing, understandably. Keep in mind the web client is rather different as an environment from the server side or the desktop. So for new apps, Brython is ok, but if you have some existing Python code you want to move into the browser, it might or might not work, depending on what your code needs. +- [Pyodide](https://github.com/pyodide/pyodide): Python with the scientific stack, compiled to WebAssembly. + - [Docs](https://pyodide.org/en/stable/). + - [Online REPL](https://pyodide.org/en/stable/console.html). + - Has **the scientific Python stack**, and also supports **any pure-Python PyPI wheel**. + - The `ast` module works. This should be able to run `mcpyrate` and `unpythonic` in the browser! + +- Historical Python-in-the-browser efforts: + - [Brython](https://brython.info/): Python 3 in the browser, as a replacement for JavaScript. + - No separate compile step - the compiler is implemented in JS. Including a script tag of type text/python invokes it. + - Doesn't have the `ast` module, so no way to run macro expanders. + - Also quite a few other parts are missing, understandably. Keep in mind the web client is rather different as an environment from the server side or the desktop. So for new apps, Brython is ok, but if you have some existing Python code you want to move into the browser, it might or might not work, depending on what your code needs. + - [PyPy.js](http://pypyjs.org/): PyPy python interpreter, compiled for the web via [emscripten](http://emscripten.org/), with a custom JIT backend that emits [asm.js](http://asmjs.org/) code at runtime. + - Last updated in 2015, no longer working. - Counterpoint: [Eric Torreborre (2019): When FP does not save us](https://medium.com/barely-functional/when-fp-does-not-save-us-92b26148071f) From fcdf1e67b640bf1d55de5ca16d01e277d54f4050 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:31:15 +0300 Subject: [PATCH 560/832] 0.15.0: improve exception utils docs --- doc/features.md | 50 +++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/doc/features.md b/doc/features.md index df2016d0..d56e99d1 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4125,11 +4125,13 @@ Utilities for dealing with exceptions. ### `raisef`, `tryf`: `raise` and `try` as functions +**Changed in v0.15.0.** *Deprecated parameters for `raisef` removed.* + **Changed in v0.14.3**. *Now we have also `tryf`.* -**Changed in v0.14.2**. *The parameters of `raisef` now more closely match what would be passed to `raise`. See examples below. Old-style parameters are now deprecated, and support for them will be dropped in v0.15.0.* +**Changed in v0.14.2**. *The parameters of `raisef` now more closely match what would be passed to `raise`. See examples below. Old-style parameters are now deprecated.* -Raise an exception from an expression position: +The `raisef` function allows to raise an exception from an expression position: ```python from unpythonic import raisef @@ -4142,7 +4144,7 @@ exc = TypeError("oof") g = lambda x: raisef(RuntimeError("I'm in ur lambda raising exceptions"), cause=exc) ``` -Catch an exception in an expression position: +The `tryf` function is a `try`/`except`/`else`/`finally` construct for an expression position: ```python from unpythonic import raisef, tryf @@ -4152,16 +4154,18 @@ test[tryf(lambda: raise_instance(), (ValueError, lambda err: f"got a ValueError: '{err.args[0]}'")) == "got a ValueError: 'all ok'"] ``` -The exception handler is a function. It may optionally accept one argument, the exception instance. +The exception handler is a function. It may optionally accept one argument, the exception instance. Just like in an `except` clause, the exception specification can be either an exception type, or a `tuple` of exception types. + +Functions can also be specified to represent the `else` and `finally` blocks; the keyword parameters to do this are `elsef` and `finallyf`. Each of them is a thunk (a 0-argument function). See the docstring of `unpythonic.misc.tryf` for details. -Functions can also be specified for the `else` and `finally` behavior; see the docstring of `unpythonic.misc.tryf` for details. +Examples can be found in [the unit tests](../unpythonic/tests/test_excutil.py). ### `equip_with_traceback` **Added in v0.14.3**. -In Python 3.7 and later, equip a manually created exception instance with a traceback. This is useful mainly in special cases, where `raise` cannot be used for some reason. (The `signal` function in the conditions-and-restarts system uses this.) +In Python 3.7 and later, the `equip_with_traceback` function equips a manually created exception instance with a traceback. This is useful mainly in special cases, where `raise` cannot be used for some reason. (The `signal` function in the conditions-and-restarts system uses this.) ```python e = SomeException(...) @@ -4170,22 +4174,24 @@ e = equip_with_traceback(e) The traceback is automatically extracted from the call stack of the calling thread. -Optionally, you can cull a number of the topmost frames by passing the optional argument `stacklevel=...`. Typically, for direct use of this function `stacklevel` should be the default `1` (so it excludes `equip_with_traceback` itself, but shows all stack levels from your code), and for use in a utility function that itself is called from your code, it should be `2` (so it excludes the utility function, too). +Optionally, you can cull a number of the topmost frames by passing the optional argument `stacklevel=...`. Typically, for direct use of this function `stacklevel` should be the default `1` (so it excludes `equip_with_traceback` itself, but shows all stack levels from your code), and for use in a utility function that itself is called from your code, it should be `2` (so it excludes the utility function, too). If the utility function itself calls a separate low-level utility, `3` can be useful (see [the source code](../unpythonic/conditions.py) of the conditions-and-restarts system for an example). ### `async_raise`: inject an exception to another thread **Added in v0.14.2**. -*Currently CPython only, because as of this writing (March 2020) PyPy3 does not expose the required functionality to the Python level, nor there seem to be any plans to do so.* +**CAUTION**: *Currently this is supported by CPython only, because as of June 2021, PyPy3 does not expose the required functionality to the Python level, nor there seem to be any plans to do so.* + +Usually injecting an exception into an unsuspecting thread makes absolutely no sense. But there are special cases, notably `KeyboardInterrupt`. Especially, a REPL server may need to send a `KeyboardInterrupt` into a REPL session thread that is happily stuck waiting for input inside [`InteractiveConsole.interact`](https://docs.python.org/3/library/code.html#code.InteractiveConsole.interact) - while the client that receives the actual `Ctrl+C` is running in a separate process, possibly even on a different machine. This and similar awkward situations in network programming are pretty much the only use case for this feature. -Usually injecting an exception into an unsuspecting thread makes absolutely no sense. But there are special cases, such as a REPL server which needs to send a `KeyboardInterrupt` into a REPL session thread that's happily stuck waiting for input at [`InteractiveConsole.interact()`](https://docs.python.org/3/library/code.html#code.InteractiveConsole.interact) - while the client that receives the actual `Ctrl+C` is running in a separate process. This and similar awkward situations in network programming are pretty much the only legitimate use case for this feature. +The function is named `async_raise`, because it injects an *asynchronous exception*. This has nothing to do with `async`/`await`. Synchronous vs. asynchronous exceptions [mean something different](https://en.wikipedia.org/wiki/Exception_handling#Exception_synchronicity). -The name is `async_raise`, because it injects an *asynchronous exception*. This has nothing to do with `async`/`await`. Synchronous vs. asynchronous exceptions [mean something different](https://en.wikipedia.org/wiki/Exception_handling#Exception_synchronicity). +In a nutshell, a *synchronous* exception (which is the usual kind of exception) has an explicit `raise` somewhere in the code that the thread that encountered the exception is running. In contrast, an *asynchronous* exception **does not**, it just suddenly magically materializes from the outside. As such, it can in principle happen *anywhere*, with absolutely no hint about it in any obvious place in the code. -In a nutshell, a *synchronous* exception (which is the usual kind of exception) has an explicit `raise` somewhere in the code that the thread that encountered the exception is running. In contrast, an *asynchronous* exception **doesn't**, it just suddenly magically materializes from the outside. As such, it can in principle happen *anywhere*, with absolutely no hint about it in any obvious place in the code. +Obviously, this can be very confusing, so this feature should be used sparingly, if at all. **We only provide it because the REPL server needs it**, and it would be silly to have such a feature but not make it public. -Needless to say this can be very confusing, so this feature should be used sparingly, if at all. **We only have it because the REPL server needs it.** +Here is an example: ```python from unpythonic import async_raise, box @@ -4201,16 +4207,16 @@ def worker(): t = threading.Thread(target=worker) t.start() sleep(0.1) # make sure the worker has entered the loop -async_raise(t, KeyboardInterrupt) +async_raise(t, KeyboardInterrupt) # CPython only! This will gracefully error out on PyPy. t.join() assert unbox(out) < 9 # thread terminated early due to the injected KeyboardInterrupt ``` -#### So this is how KeyboardInterrupt works under the hood? +#### Is this how KeyboardInterrupt works under the hood? -No, this is **not** how `KeyboardInterrupt` usually works. Rather, the OS sends a [SIGINT](https://en.wikipedia.org/wiki/Signal_(IPC)#SIGINT), which is then trapped by an [OS signal handler](https://docs.python.org/3/library/signal.html) that runs in the main thread. +**No, it is not.** The way `KeyboardInterrupt` usually works is, the OS sends a [SIGINT](https://en.wikipedia.org/wiki/Signal_(IPC)#SIGINT), which is then trapped by an [OS signal handler](https://docs.python.org/3/library/signal.html) that runs in the main thread. -(Note OS signal, in the *nix sense; this is unrelated to the Lisp sense, as in conditions-and-restarts.) +Note that it is an OS signal, in the *nix sense; which is unrelated to the Lisp/`unpythonic` sense, as in conditions-and-restarts. At that point the magic has already happened: the control of the main thread is now inside the signal handler, as if the signal handler was called from the otherwise currently innermost point on the call stack. All the handler needs to do is to perform a regular `raise`, and the exception will propagate correctly. @@ -4218,9 +4224,11 @@ At that point the magic has already happened: the control of the main thread is Original detective work by [Federico Ficarelli](https://gist.github.com/nazavode/84d1371e023bccd2301e) and [LIU Wei](https://gist.github.com/liuw/2407154). -Raising async exceptions is a [documented feature of Python's public C API](https://docs.python.org/3/c-api/init.html#c.PyThreadState_SetAsyncExc), but it was never meant to be invoked from within pure Python code. But then the CPython devs gave us [ctypes.pythonapi](https://docs.python.org/3/library/ctypes.html#accessing-values-exported-from-dlls), which allows access to Python's C API from within Python. (If you think ctypes.pythonapi is too quirky, the [pycapi](https://pypi.org/project/pycapi/) PyPI package smooths over the rough edges.) Combining the two gives `async_raise` without the need to compile a C extension. +Raising async exceptions is a [documented feature of Python's public C API](https://docs.python.org/3/c-api/init.html#c.PyThreadState_SetAsyncExc), but it was never meant to be invoked from within pure Python code. But then the CPython devs gave us [ctypes.pythonapi](https://docs.python.org/3/library/ctypes.html#accessing-values-exported-from-dlls), which allows access to CPython's C API from within Python. Combining the two gives `async_raise` without the need to compile a C extension. -Unfortunately PyPy doesn't currently (March 2020) implement this function in its CPython C API emulation layer, `cpyext`. See `unpythonic` issue [#58](https://github.com/Technologicat/unpythonic/issues/58). +(If you think `ctypes.pythonapi` is too quirky, the [pycapi](https://pypi.org/project/pycapi/) PyPI package smooths over the rough edges.) + +Unfortunately PyPy does **not** currently (June 2021) implement this function in its CPython C API emulation layer, `cpyext`. See `unpythonic` issue [#58](https://github.com/Technologicat/unpythonic/issues/58). ### `reraise_in`, `reraise`: automatically convert exception types @@ -4290,14 +4298,16 @@ except ApplicationException: ``` -If that's not much shorter than the hand-written `try`/`except`/`raise from`, consider that you can create the mapping once and then use it from a variable - this shortens it to just `with reraise(my_mapping)`. +If that does not seem much shorter than a hand-written `try`/`except`/`raise from`, consider that you can create the mapping once and then use it from a variable - this shortens it to just `with reraise(my_mapping)`. -Any exceptions that don't match anything in the mapping are passed through. When no exception occurs, `reraise_in` passes the return value of `thunk` through, and `reraise` does nothing. +Any exceptions that do not match anything in the mapping are passed through. When no exception occurs, `reraise_in` passes the return value of `thunk` through, and `reraise` does nothing. Full details in docstrings. If you use the conditions-and-restarts system, see also `resignal_in`, `resignal`, which perform the same job for conditions. The new signal is sent using the same error handling protocol as the original signal, so e.g. an `error` will remain an `error` even if re-signaling changes its type. +Examples can be found in [the unit tests](../unpythonic/tests/test_excutil.py). + ## Function call and return value tools From b954ce7756e31d90ee0c76196f055fa68da31b7c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:31:41 +0300 Subject: [PATCH 561/832] 0.15.0: improve @call and @callwith docs --- doc/features.md | 57 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/doc/features.md b/doc/features.md index d56e99d1..a3c957e9 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4313,11 +4313,15 @@ Examples can be found in [the unit tests](../unpythonic/tests/test_excutil.py). ### `def` as a code block: `@call` -Fuel for different thinking. Compare `call-with-something` in Lisps - but without parameters, so just `call`. A `def` is really just a new lexical scope to hold code to run later... or right now! +Fuel for different thinking. Compare `call-with-something` in Lisps - but without parameters, so just `call`. A `def` is really just a new lexical scope to hold code to run later... or as `@call` does, right now! At the top level of a module, this is seldom useful, but keep in mind that Python allows nested function definitions. Used with an inner `def`, this becomes a versatile tool. -*Make temporaries fall out of scope as soon as no longer needed*: +Note that beside use as a decorator, `call` can also be used as a normal function: `call(f, *a, **kw)` is the same as `f(*a, **kw)`. This is occasionally useful. + +Let us consider some example use cases of `@call`. + +#### Make temporaries fall out of scope as soon as no longer needed ```python from unpythonic import call @@ -4331,9 +4335,13 @@ def x(): print(x) # 30 ``` -*Multi-break out of nested loops* - `continue`, `break` and `return` are really just second-class [ec](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._call%2Fec%29%29)s. So `def` to make `return` escape to exactly where you want: +#### Multi-break out of nested loops + +As was noted in the section on escape continuations, `continue`, `break` and `return` are really just second-class [ec](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._call%2Fec%29%29)s. So use a `def` to make `return` escape to exactly where you want: ```python +from unpythonic import call + @call def result(): for x in range(10): @@ -4343,7 +4351,7 @@ def result(): print(result) # (6, 7) ``` -(But see `@catch`, `throw`, and `call_ec`.) +But if you need a *multi-return*, see `@catch`, `throw`, and `call_ec`. Compare the sweet-exp Racket: @@ -4361,6 +4369,8 @@ displayln result ; (6 7) Noting [what `let/ec` does](https://docs.racket-lang.org/reference/cont.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%2Fec%29%29), using `call_ec` we can make the Python even closer to the Racket: ```python +from unpythonic import call_ec + @call_ec def result(rtn): for x in range(10): @@ -4370,20 +4380,24 @@ def result(rtn): print(result) # (6, 7) ``` -*Twist the meaning of `def` into a "let statement"*: +#### Twist the meaning of `def` into a "let statement" ```python +from unpythonic import call + @call def result(x=1, y=2, z=3): return x * y * z print(result) # 6 ``` -(But see `blet`, `bletrec` if you want an `env` instance.) +If you want an `env` instance, see `blet` and `bletrec`. -*Letrec without `letrec`*, when it doesn't have to be an expression: +#### Letrec without `letrec`*, when a statement is acceptable ```python +from unpythonic import call + @call def t(): def evenp(x): return x == 0 or oddp(x - 1) @@ -4392,22 +4406,22 @@ def t(): print(t) # True ``` -Essentially the implementation is just `def call(thunk): return thunk()`. The point is to: +#### Notes - - Make it explicit right at the definition site that this block is *going to be called now* (in contrast to an explicit call and assignment *after* the definition). Centralize the related information. Align the presentation order with the thought process. +Essentially the implementation is just `def call(thunk): return thunk()`. The point of this seemingly trivial construct is to: - - Help eliminate errors, in the same way as the habit of typing parentheses only in pairs. No risk of forgetting to call the block after writing the definition. + - Make it explicit right at the definition site that this block is *going to be called now*, in contrast to an explicit call and assignment *after* the definition. This centralizes the related information, and aligns the presentation order with the thought process. - - Document that the block is going to be used only once. Tell the reader there's no need to remember this definition. + - Help eliminate errors, in the same way as the habit of typing parentheses only in pairs (or using a tool like Emacs's `smartparens-mode` to enforce that). With `@call`, there is no risk of forgetting to call the block after writing the definition. -Note [the grammar](https://docs.python.org/3/reference/grammar.html) requires a newline after a decorator. + - Document that the block is going to be used only once. Tell your readers there is no need to remember this definition. -**NOTE**: `call` can also be used as a normal function: `call(f, *a, **kw)` is the same as `f(*a, **kw)`. This is occasionally useful. +Note [the grammar](https://docs.python.org/3/reference/grammar.html) requires a newline after a decorator. ### `@callwith`: freeze arguments, choose function later -If you need to pass arguments when using `@call` as a decorator, use its cousin `@callwith`: +If you need to pass arguments when using `@call` as a decorator, use its sister `@callwith`: ```python from unpythonic import callwith @@ -4418,9 +4432,11 @@ def result(x): assert result == 9 ``` -Like `call`, it can also be called normally. It's essentially an argument freezer: +Like `call`, beside use as a decorator, `callwith` can also be called normally. It is essentially an argument freezer: ```python +from unpythonic import callwith + def myadd(a, b): return a + b def mymul(a, b): @@ -4430,16 +4446,17 @@ assert apply23(myadd) == 5 assert apply23(mymul) == 6 ``` -When called normally, the two-step application is mandatory. The first step stores the given arguments. It returns a function `f(callable)`. When `f` is called, it calls its `callable` argument, passing in the arguments stored in the first step. +When `callwith` is called normally, the two-step application is mandatory. The first step stores the given arguments. It then returns a function `f(callable)`. When `f` is called, it calls its `callable` argument, passing in the arguments stored in the first step. In other words, `callwith` is similar to `functools.partial`, but without specializing to any particular function. The function to be called is given later, in the second step. -Hence, `callwith(2, 3)(myadd)` means "make a function that passes in two positional arguments, with values `2` and `3`. Then call this function for the callable `myadd`". But if we instead write`callwith(2, 3, myadd)`, it means "make a function that passes in three positional arguments, with values `2`, `3` and `myadd` - not what we want in the above example. +Hence, `callwith(2, 3)(myadd)` means *make a function that passes in two positional arguments, with values `2` and `3`. Then call this function for the callable `myadd`*. But if we instead write `callwith(2, 3, myadd)`, it means *make a function that passes in three positional arguments, with values `2`, `3` and `myadd`* - not what we want in the above example. -If you want to specialize some arguments now and some later, combine with `partial`: +If you want to specialize some arguments now and some later, combine `callwith` with `partial`: ```python from functools import partial +from unpythonic import callwith p1 = partial(callwith, 2) p2 = partial(p1, 3) @@ -4458,11 +4475,13 @@ If the code above feels weird, it should. Arguments are gathered first, and the Another use case of `callwith` is `map`, if we want to vary the function instead of the data: ```python +from unpythonic import callwith + m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) assert tuple(m) == (6, 9, 3**(1/2)) ``` -If you use the quick lambda macro `f[]` (underscore notation for Python), this combines nicely: +If you use the quick lambda macro `f[]` (underscore notation for Python), these features combine nicely: ```python from unpythonic.syntax import macros, f From 28302fa1070c84adb6c1a5835623909e6b435e9d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:43:35 +0300 Subject: [PATCH 562/832] 0.15.0: improve funutil docs --- doc/features.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/doc/features.md b/doc/features.md index a3c957e9..74ed5765 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4525,9 +4525,9 @@ Inspired by *Function application with $* in [LYAH: Higher Order Functions](http `Values` is a structured multiple-return-values type. -With `Values`, you can return multiple values positionally and by name. This completes the symmetry between passing function arguments and returning values from a function: Python itself allows passing arguments by name, but has no concept of returning values by name. This class adds that concept. +With `Values`, you can return multiple values positionally, and **return values by name**. This completes the symmetry between passing function arguments and returning values from a function. Python itself allows passing arguments by name, but has no concept of returning values by name. This class adds that concept. -Having a `Values` type separate from `tuple` also helps with semantic accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value now means just that - one value that is a `tuple`. It is different from a `Values` that contains several positional return values (that are meant to be treated separately e.g. by a function composition utility). +Having a `Values` type separate from `tuple` helps with semantic accuracy. In `unpythonic` 0.15.0 and later, a `tuple` return value means just that - one value that is a `tuple`. It is distinct from a `Values` that contains several positional return values (that are meant to be treated separately e.g. by a function composition utility). Inspired by the [`values`](https://docs.racket-lang.org/reference/values.html) form of Racket. @@ -4541,7 +4541,9 @@ Accordingly, various parts of `unpythonic` that deal with function composition u #### Behavior -`Values` is a duck-type with some features of both sequences and mappings, but not the full `collections.abc` API of either. +`Values` is a duck-type with some features of both sequences and mappings, but not the full [`collections.abc`](https://docs.python.org/3/library/collections.abc.html) API of either. + +If there are no named return values in a `Values` object, it can be unpacked like a tuple. This covers the common use case of multiple positional return values with a minimum of fuss. Each operation that obviously and without ambiguity makes sense only for the positional or named part, accesses that part. @@ -4553,9 +4555,13 @@ If you need to explicitly access either part (and its full API), use the `rets` `Values` objects can be compared for equality. Two `Values` objects are equal if both their `rets` and `kwrets` (respectively) are. +See the docstrings, [the source code](../unpythonic/funutil.py), and [the unit tests](../unpythonic/tests/test_funutil.py) for full details. + Examples: ```python +from unpythonic import Values + def f(): return Values(1, 2, 3) result = f() @@ -4602,13 +4608,15 @@ The last example is silly, but legal, because it is preferable to just omit the ### `valuify` -We also provide `valuify`, a decorator that converts the pythonic tuple-as-multiple-return-values idiom into `Values`, for compatibility with our function composition utilities. +The `valuify` decorator converts the pythonic tuple-as-multiple-return-values idiom into `Values`, to easily use existing code with our function composition utilities. It converts a `tuple` return value, exactly; no subclasses. -Demonstrating just the conversion: +Demonstrating only the conversion: ```python +from unpythonic import valuify, Values + @valuify def f(x, y, z): return x, y, z From 4e2ec4f7f9397d6b38a03b1b5226a4dc0c5705c3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:48:57 +0300 Subject: [PATCH 563/832] improve "not to be confused with" notes --- doc/features.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 74ed5765..5f5a4a12 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4654,9 +4654,9 @@ For `float`, we use the strategy suggested in [the floating point guide](https:/ **Added in v0.14.2.** -Compute the (arithmetic) fixed point of a function, starting from a given initial guess. The fixed point must be attractive for this to work. See the [Banach fixed point theorem](https://en.wikipedia.org/wiki/Banach_fixed-point_theorem). +*Not to be confused with the logical fixed point with respect to the definedness ordering, which is what Haskell's `fix` function relates to.* -(Not to be confused with the logical fixed point with respect to the definedness ordering, which is what Haskell's `fix` function relates to.) +Compute the (arithmetic) fixed point of a function, starting from a given initial guess. The fixed point must be attractive for this to work. See the [Banach fixed point theorem](https://en.wikipedia.org/wiki/Banach_fixed-point_theorem). If the fixed point is attractive, and the values are represented in floating point (hence finite precision), the computation should eventually converge down to the last bit (barring roundoff or catastrophic cancellation in the final few steps). Hence the default tolerance is zero; but a desired tolerance can be passed as an argument. @@ -4686,13 +4686,12 @@ assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) **Added in v0.14.2.** **Changed in v0.15.0.** *Added `partition_int_triangular`.* +*Not to be confused with `unpythonic.partition`, which partitions an iterable based on a predicate.* The `partition_int` function [partitions](https://en.wikipedia.org/wiki/Partition_(number_theory)) a small positive integer, i.e., splits it in all possible ways, into smaller integers that sum to it. This is useful e.g. to determine the number of letters to allocate for each component of an anagram that may consist of several words. The `partition_int_triangular` function is like `partition_int`, but accepts only triangular numbers (1, 3, 6, 10, ...) as components of the partition. This function answers a timeless question: if I have `n` stackable plushies, what are the possible stack configurations? -(These are not to be confused with `unpythonic.partition`, which partitions an iterable based on a predicate.) - Examples: ```python From e1176a28a1f67f3b1e4190c0283f756bbffd949d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:49:43 +0300 Subject: [PATCH 564/832] ordering of changed/added notes: chronological, most recent first --- doc/features.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/features.md b/doc/features.md index 5f5a4a12..4462dd75 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4683,9 +4683,10 @@ assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) ### `partition_int`, `partition_int_triangular`: partition integers +**Changed in v0.15.0.** *Added `partition_int_triangular`.* + **Added in v0.14.2.** -**Changed in v0.15.0.** *Added `partition_int_triangular`.* *Not to be confused with `unpythonic.partition`, which partitions an iterable based on a predicate.* The `partition_int` function [partitions](https://en.wikipedia.org/wiki/Partition_(number_theory)) a small positive integer, i.e., splits it in all possible ways, into smaller integers that sum to it. This is useful e.g. to determine the number of letters to allocate for each component of an anagram that may consist of several words. @@ -4753,10 +4754,10 @@ Stuff that didn't fit elsewhere. ### `callsite_filename` -**Added in v0.14.3**. - **Changed in v0.15.0.** *This utility now ignores `unpythonic`'s call helpers, and gives the filename from the deepest stack frame that does not match one of our helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. `autocurry`, `lazify`).* +**Added in v0.14.3**. + Return the filename from which this function is being called. Useful as a building block for debug utilities and similar. @@ -4856,12 +4857,12 @@ assert getattrrec(w, "x") == 23 ### `arities`, `kwargs`, `resolve_bindings`: Function signature inspection utilities -**Added in v0.14.2**: `resolve_bindings`. *Get the parameter bindings a given callable would establish if it was called with the given args and kwargs. This is mainly of interest for implementing memoizers, since this allows them to see (e.g.) `f(1)` and `f(a=1)` as the same thing for `def f(a): pass`.* - **Changed in v0.15.0.** *Now `resolve_bindings` is a thin wrapper on top of `inspect.Signature.bind`, which was added in Python 3.5. In `unpythonic` 0.14.2 and 0.14.3, we used to have our own implementation of the parameter binding algorithm (that ran also on Python 3.4), but it is no longer needed, since now we support only Python 3.6 and later. Now `resolve_bindings` returns an `inspect.BoundArguments` object.* *Now `tuplify_bindings` accepts an `inspect.BoundArguments` object instead of its previous input format. The function is only ever intended to be used to postprocess the output of `resolve_bindings`, so this change shouldn't affect your own code.* +**Added in v0.14.2**: `resolve_bindings`. *Get the parameter bindings a given callable would establish if it was called with the given args and kwargs. This is mainly of interest for implementing memoizers, since this allows them to see (e.g.) `f(1)` and `f(a=1)` as the same thing for `def f(a): pass`.* + Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by `inspect`. Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit `self` and `cls` are ignored). From 3b72c6c418182cb0af6c17fbc43df266f132d827 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:53:31 +0300 Subject: [PATCH 565/832] added/changed notes formatting/ordering --- doc/macros.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 918781bb..ae432d50 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -694,16 +694,16 @@ The naming is performed using the function `unpythonic.misc.namelambda`, which w - Single-item assignment to a local name, `f = lambda ...: ...` - - **Added in v0.15.0**: Named expressions (a.k.a. walrus operator, Python 3.8+), `f := lambda ...: ...` + - Named expressions (a.k.a. walrus operator, Python 3.8+), `f := lambda ...: ...`. **Added in v0.15.0.** - Expression-assignment to an unpythonic environment, `f << (lambda ...: ...)` - Env-assignments are processed lexically, just like regular assignments. This should not cause problems, because left-shifting by a literal lambda most often makes no sense (whence, that syntax is *almost* guaranteed to mean an env-assignment). - Let bindings, `let[[f << (lambda ...: ...)] in ...]`, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). - - **Added in v0.14.2**: Named argument in a function call, as in `foo(f=lambda ...: ...)`. + - Named argument in a function call, as in `foo(f=lambda ...: ...)`. **Added in v0.14.2.** - - **Added in v0.14.2**: In a dictionary literal `{...}`, an item with a literal string key, as in `{"f": lambda ...: ...}`. + - In a dictionary literal `{...}`, an item with a literal string key, as in `{"f": lambda ...: ...}`. **Added in v0.14.2.** Support for other forms of assignment may or may not be added in a future version. We will maintain a list here; but if you want the gritty details, see the `_namedlambda` syntax transformer in [`unpythonic.syntax.lambdatools`](../unpythonic/syntax/lambdatools.py). @@ -2055,7 +2055,7 @@ Look at the implementation of `testset` as an example. Because `unpythonic` is effectively a language extension, the standard options were not applicable. -The standard library's [`unittest`](https://docs.python.org/3/library/unittest.html) fails with `unpythonic` due to technical reasons related to `unpythonic`'s unfortunate choice of module names. The `unittest` framework chokes if a module in a library exports anything that has the same name as the module itself, and the library's top-level init then `from`-imports that construct into its namespace, causing the *module reference*, that was [implicitly brought in](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-submodules-are-added-to-the-package-namespace-trap) by the `from`-import itself, to be overwritten with what was explicitly imported: a reference to the construct that has the same name as the module. (Bad naming on my part, yes, but we're stuck with it at least until v0.15.0. As of v0.14.3, I see no reason to cross that particular bridge yet.) +The standard library's [`unittest`](https://docs.python.org/3/library/unittest.html) fails with `unpythonic` due to technical reasons related to `unpythonic`'s unfortunate choice of module names. The `unittest` framework chokes if a module in a library exports anything that has the same name as the module itself, and the library's top-level init then `from`-imports that construct into its namespace, causing the *module reference*, that was [implicitly brought in](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-submodules-are-added-to-the-package-namespace-trap) by the `from`-import itself, to be overwritten with what was explicitly imported: a reference to the construct that has the same name as the module. (Bad naming on my part, yes, but as of v0.15.0, I see no reason to cross that particular bridge yet.) Also, in my opinion, `unittest` is overly verbose to use; automated tests are already a particularly verbose kind of program, even if the testing syntax is minimal. From 6f4932e015ddca9ee356ccefcc47878b430070ec Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 01:59:03 +0300 Subject: [PATCH 566/832] 0.15.0: improve numutil docs --- doc/features.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/features.md b/doc/features.md index 4462dd75..72a01d98 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4628,15 +4628,15 @@ assert f(1, 2, 3) == Values(1, 2, 3) ## Numerical tools -We briefly introduce the functions below. More details and examples can be found in the docstrings and [the unit tests](../unpythonic/tests/test_numutil.py). +We briefly introduce the functions below. More details and examples can be found in the docstrings and in [the unit tests](../unpythonic/tests/test_numutil.py). **CAUTION** for anyone new to numerics: -When working with floating-point numbers, keep in mind that they are, very roughly speaking, a finite-precision logarithmic representation of [ℝ](https://en.wikipedia.org/wiki/Real_line). They are, necessarily, actually a subset of [ℚ](https://en.wikipedia.org/wiki/Rational_number), that's not even [dense](https://en.wikipedia.org/wiki/Dense_set). The spacing between adjacent floats depends on where you are on the real line; see `ulp` below. +When working with floating-point numbers, keep in mind that they are, very roughly speaking, a finite-precision logarithmic representation of [ℝ](https://en.wikipedia.org/wiki/Real_line). They are, necessarily, actually a subset of [ℚ](https://en.wikipedia.org/wiki/Rational_number), that is not even [dense](https://en.wikipedia.org/wiki/Dense_set). The spacing between adjacent floats depends on where you are on the real line; see `ulp` below. For finer points concerning the behavior of floating-point numbers, see [David Goldberg (1991): What every computer scientist should know about floating-point arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), or for a [tl;dr](http://catplanet.org/tldr-cat-meme/) version, [the floating point guide](https://floating-point-gui.de/). -Or you could look at [my lecture slides from 2018](https://github.com/Technologicat/python-3-scicomp-intro/tree/master/lecture_slides); particularly, [lecture 7](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/lecture_slides/lectures_tut_2018_7.pdf) covers the floating-point representation. It collects the most important details, and some more links to further reading. +Or you could look at [my lecture slides from 2018](https://github.com/Technologicat/python-3-scicomp-intro/tree/master/lecture_slides); particularly, [lecture 7](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/lecture_slides/lectures_tut_2018_7.pdf) covers the floating-point representation. It collects the most important details in a few slides, and contains some more links to further reading. ### `almosteq`: floating-point almost-equality @@ -4658,7 +4658,7 @@ For `float`, we use the strategy suggested in [the floating point guide](https:/ Compute the (arithmetic) fixed point of a function, starting from a given initial guess. The fixed point must be attractive for this to work. See the [Banach fixed point theorem](https://en.wikipedia.org/wiki/Banach_fixed-point_theorem). -If the fixed point is attractive, and the values are represented in floating point (hence finite precision), the computation should eventually converge down to the last bit (barring roundoff or catastrophic cancellation in the final few steps). Hence the default tolerance is zero; but a desired tolerance can be passed as an argument. +If the fixed point is attractive, and the values are represented in floating point (hence finite precision), the computation should eventually converge down to the last bit (barring roundoff or catastrophic cancellation in the final few steps). Hence the default tolerance is zero; but any desired tolerance can be passed as an argument. **CAUTION**: an arbitrary function from ℝ to ℝ **does not** necessarily have a fixed point. Limit cycles and chaotic behavior of the function will cause non-termination. Keep in mind the classic example, [the logistic map](https://en.wikipedia.org/wiki/Logistic_map). @@ -4710,7 +4710,7 @@ assert (frozenset(tuple(sorted(c)) for c in partition_int_triangular(78, lower=1 (78,)})) ``` -As the first example demonstrates, most of the splits are a ravioli consisting mostly of ones. It is much faster to not generate such splits than to filter them out from the result. Use the `lower` parameter to set the smallest acceptable value for one component of the split; the default value `lower=1` generates all splits. Similarly, the `upper` parameter sets the largest acceptable value for one component of the split. The default `upper=None` sets no upper limit. +As the first example demonstrates, most of the splits are a ravioli consisting mostly of ones. It is much faster to not generate such splits than to filter them out from the result. Use the `lower` parameter to set the smallest acceptable value for one component of the split; the default value `lower=1` generates all splits. Similarly, the `upper` parameter sets the largest acceptable value for one component of the split. The default `upper=None` sets no upper limit, so in effect the upper limit becomes `n`. In `partition_int_triangular`, the `lower` and `upper` parameters work exactly the same. The only difference to `partition_int` is that each component of the split must be a triangular number. From ad535eef3729d3bb80f81101f6403f5bbdc47807 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 02:28:12 +0300 Subject: [PATCH 567/832] 0.15.0: improve misc utils docs --- doc/features.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/doc/features.md b/doc/features.md index 72a01d98..c3ab2cfb 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4770,7 +4770,9 @@ Convenience function. Like `issubclass(cls)`, but if `cls` is not a class, swall ### `pack`: multi-arg constructor for tuple -The default `tuple` constructor accepts a single iterable. But sometimes one needs to pass in the elements separately. Most often a literal tuple such as `(1, 2, 3)` is then the right solution, but there are situations that do not admit a literal tuple. Enter `pack`: +The default `tuple` constructor accepts a single iterable. But sometimes one needs to pass in the elements separately. Most often a literal tuple such as `(1, 2, 3)` is then the right solution, but there are situations that do not admit a literal tuple. + +In such cases it is possible to use `pack`: ```python from unpythonic import pack @@ -4783,11 +4785,11 @@ assert tuple(myzip(lol)) == ((1, 3, 5), (2, 4, 6)) ### `namelambda`: rename a function -Rename any function object (including lambdas). The return value of `namelambda` is a modified copy; the original function object is not mutated. The input can be any function object (`isinstance(f, (types.LambdaType, types.FunctionType))`). It will be renamed even if it already has a name. +Rename any function object, even a lambda. The return value of `namelambda` is a modified copy; the original function object is not mutated. The input can be any function object (`isinstance(f, (types.LambdaType, types.FunctionType))`). It will be renamed even if it already has a name. This is mainly useful in those situations where you return a lambda as a closure, call it much later, and it happens to crash - so you can tell from the stack trace *which* of the *N* lambdas in your codebase it is. -For technical reasons, `namelambda` conforms to the parametric decorator API. Usage: +`namelambda` conforms to the parametric decorator API. Usage: ```python from unpythonic import namelambda @@ -4818,6 +4820,8 @@ The inner lambda does not see the outer's new name; the parent scope names are b ### `timer`: a context manager for performance testing +This is a small convenience utility, used as follows: + ```python from unpythonic import timer @@ -4831,7 +4835,7 @@ with timer(p=True): # if p, auto-print result pass ``` -The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the `with ... as ...` syntax. After the context exits, the *Δt* is available in its `dt` attribute. +The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the `with ... as ...` syntax. After the context exits, the *Δt* is available in its `dt` attribute. The timer instance itself stays alive due to Python's scoping rules. ### `getattrrec`, `setattrrec`: access underlying data in an onion of wrappers @@ -4861,9 +4865,9 @@ assert getattrrec(w, "x") == 23 *Now `tuplify_bindings` accepts an `inspect.BoundArguments` object instead of its previous input format. The function is only ever intended to be used to postprocess the output of `resolve_bindings`, so this change shouldn't affect your own code.* -**Added in v0.14.2**: `resolve_bindings`. *Get the parameter bindings a given callable would establish if it was called with the given args and kwargs. This is mainly of interest for implementing memoizers, since this allows them to see (e.g.) `f(1)` and `f(a=1)` as the same thing for `def f(a): pass`.* +**Added in v0.14.2**: `resolve_bindings`. *Get the parameter bindings a given callable would establish if it was called with the given args and kwargs. This is mainly of interest for implementing memoizers, since this allows them to see (e.g.) `f(1)` and `f(a=1)` as the same thing for `def f(a): pass`. Thanks to Graham Dumpleton, the author of the [`wrapt`](https://pypi.org/project/wrapt/) library, for [noticing and documenting this gotcha](https://wrapt.readthedocs.io/en/latest/decorators.html#processing-function-arguments).* -Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by `inspect`. +These are convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by `inspect`. Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit `self` and `cls` are ignored). @@ -4924,7 +4928,7 @@ We special-case the builtin functions that either fail to return any arity (are If the arity cannot be inspected, and the function is not one of the special-cased builtins, the `UnknownArity` exception is raised. -These functions are internally used in various places in unpythonic, particularly `curry`, `fix`, and `@generic`. The `let` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. +Up to v0.14.3, various places in unpythonic used to internally use `arities`; particularly `curry`, `fix`, and `@generic`. As of v0.15.0, we have started to prefer `resolve_bindings`, because often what matters are the parameter bindings established, and performing the binding covers all possible ways to pass arguments. The `let` and FP looping constructs still use `arities` to emit a meaningful error message if the signature of user-provided function does not match what is expected. Inspired by various Racket functions such as `(arity-includes?)` and `(procedure-keywords)`. @@ -4984,13 +4988,13 @@ assert inp == deque([]) assert out == [(0, 1), (1, 2), (2, 10), (10, 11), (11, 12)] ``` -(Although `window` invokes `iter()` on the `Popper`, this works because the `Popper` never invokes `iter()` on the underlying container. Any mutations to the input container performed by the loop body will be understood by `Popper` and thus also seen by the `window`. The first `n` elements, though, are read before the loop body gets control, because the window needs them to initialize itself.) +Although `window` invokes `iter()` on the `Popper` instance, this works because the `Popper` never invokes `iter()` on the underlying container. Any mutations to the input container performed by the loop body will be understood by `Popper` and thus also seen by the `window`. The first `n` elements, though, are read before the loop body gets control, because the window needs them to initialize itself. One possible real use case for `Popper` is to split sequences of items, stored as lists in a deque, into shorter sequences where some condition is contiguously `True` or `False`. When the condition changes state, just commit the current subsequence, and push the rest of that input sequence (still requiring analysis) back to the input deque, to be dealt with later. -The argument to `Popper` (here `lst`) contains the **remaining** items. Each iteration pops an element **from the left**. The loop terminates when `lst` is empty. +The argument to `Popper` contains the **remaining** items. Each iteration pops an element **from the left**. The loop terminates when, at the start of an iteration, there are no more items remaining. -The input container must support either `popleft()` or `pop(0)`. This is fully duck-typed. At least `collections.deque` and any `collections.abc.MutableSequence` (including `list`) are fine. +The input container must support either `popleft()` or `pop(0)`. This is fully duck-typed. At least `collections.deque` and any [`collections.abc.MutableSequence`](https://docs.python.org/3/library/collections.abc.html) (including `list`) are fine. Per-iteration efficiency is O(1) for `collections.deque`, and O(n) for a `list`. From 3eea6ecfdafcc20a02caa843f5a6b1173afb039a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 02:50:03 +0300 Subject: [PATCH 568/832] fix borked links to unit test files after the great rename --- doc/macros.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index ae432d50..5207c703 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -15,11 +15,11 @@ Our extensions to the Python language are built on [`mcpyrate`](https://github.com/Technologicat/mcpyrate), from the PyPI package [`mcpyrate`](https://pypi.org/project/mcpyrate/). -Because in Python macro expansion occurs *at import time*, Python programs whose main module uses macros, such as [our unit tests that contain usage examples](../unpythonic/syntax/test/), cannot be run directly. Instead, run them via `macropython`, included in `mcpyrate`. +Because in Python macro expansion occurs *at import time*, Python programs whose main module uses macros, such as [our unit tests that contain usage examples](../unpythonic/syntax/tests/), cannot be run directly by `python3`. Instead, run them via the `macropython` bootstrapper, included in `mcpyrate`. **Our macros expect a from-import style** for detecting uses of `unpythonic` constructs, *even when those constructs are regular functions*. For example, the function `curry` is detected from its bare name. So if you intend to use these macros, then, for regular imports from `unpythonic`, use `from unpythonic import ...` and avoid renaming (`as`). -*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* +*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests do - and optionally file an issue on GitHub so that the documentation can be fixed.* **Changed in v0.15.0.** *To run macro-enabled programs, use the [`macropython`](https://github.com/Technologicat/mcpyrate/blob/master/doc/repl.md#macropython-the-universal-bootstrapper) bootstrapper from [`mcpyrate`](https://github.com/Technologicat/mcpyrate).* @@ -686,7 +686,7 @@ with namedlambda: Lexically inside a `with namedlambda` block, any literal `lambda` that is assigned to a name using one of the supported assignment forms is named to have the name of the LHS of the assignment. The name is captured at macro expansion time. -Decorated lambdas are also supported, as is a `curry` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/test/test_lambdatools.py) for detailed examples. +Decorated lambdas are also supported, as is a `curry` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/tests/test_lambdatools.py) for detailed examples. The naming is performed using the function `unpythonic.misc.namelambda`, which will return a modified copy with its `__name__`, `__qualname__` and `__code__.co_name` changed. The original function object is not mutated. @@ -1127,7 +1127,7 @@ Observe that while our outermost `call_cc` already somewhat acts like a prompt ( For various possible program topologies that continuations may introduce, see [these clarifying pictures](callcc_topology.pdf). -For full documentation, see the docstring of `unpythonic.syntax.continuations`. The unit tests [[1]](../unpythonic/syntax/test/test_conts.py) [[2]](../unpythonic/syntax/test/test_conts_escape.py) [[3]](../unpythonic/syntax/test/test_conts_gen.py) [[4]](../unpythonic/syntax/test/test_conts_topo.py) may also be useful as usage examples. +For full documentation, see the docstring of `unpythonic.syntax.continuations`. The unit tests [[1]](../unpythonic/syntax/tests/test_conts.py) [[2]](../unpythonic/syntax/tests/test_conts_escape.py) [[3]](../unpythonic/syntax/tests/test_conts_gen.py) [[4]](../unpythonic/syntax/tests/test_conts_topo.py) may also be useful as usage examples. **Note on debugging**: If a function containing a `call_cc[]` crashes below the `call_cc[]`, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so you can pinpoint the source code line where the error occurred. (For a function `f`, it is named `f_cont_`) But be aware that especially in complex macro combos (e.g. `continuations, curry, lazify`), the other block macros may spit out many internal function calls *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. @@ -1233,7 +1233,7 @@ Code within a `with continuations` block is treated specially. #### Differences between `call/cc` and certain other language features - - Unlike **generators**, `call_cc[]` allows resuming also multiple times from an earlier checkpoint, even after execution has already proceeded further. Generators can be easily built on top of `call/cc`. [Python version](../unpythonic/syntax/test/test_conts_gen.py), [Racket version](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt). + - Unlike **generators**, `call_cc[]` allows resuming also multiple times from an earlier checkpoint, even after execution has already proceeded further. Generators can be easily built on top of `call/cc`. [Python version](../unpythonic/syntax/tests/test_conts_gen.py), [Racket version](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt). - The Python version is a pattern that could be packaged into a macro with `mcpyrate`; the Racket version has been packaged as a macro. - Both versions are just demonstrations for teaching purposes. In production code, use the language's native functionality. - Python's built-in generators have no restriction on where `yield` can be placed, and provide better performance. @@ -1723,7 +1723,7 @@ Nested autoref blocks are allowed (lookups are lexically scoped). Reading with `autoref` can be convenient e.g. for data returned by [SciPy's `.mat` file loader](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html). -See the [unit tests](../unpythonic/syntax/test/test_autoref.py) for more usage examples. +See the [unit tests](../unpythonic/syntax/tests/test_autoref.py) for more usage examples. This is similar to the JavaScript [`with` construct](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/with), which is nowadays [deprecated](https://2ality.com/2011/06/with-statement.html). See also [the ES6 reference on `with`](https://www.ecma-international.org/ecma-262/6.0/#sec-with-statement). @@ -2005,7 +2005,7 @@ The `the[]` mechanism is smart enough to skip reporting trivialities for literal If nothing but such trivialities were captured, the failure message will instead report the value of the whole expression. (The captures still remain inspectable in the exception instance.) -To make testing/debugging macro code more convenient, the `the[]` mechanism automatically unparses an AST value into its source code representation for display in the test failure message. This is meant for debugging macro utilities, to which a test case hands some quoted code (i.e. code lifted into its AST representation using mcpyrate's `q[]` macro). See [`unpythonic.syntax.test.test_letdoutil`](unpythonic/syntax/test/test_letdoutil.py) for some examples. (Note the unparsing is done for display only; the raw value remains inspectable in the exception instance.) +To make testing/debugging macro code more convenient, the `the[]` mechanism automatically unparses an AST value into its source code representation for display in the test failure message. This is meant for debugging macro utilities, to which a test case hands some quoted code (i.e. code lifted into its AST representation using mcpyrate's `q[]` macro). See [`unpythonic.syntax.tests.test_letdoutil`](unpythonic/syntax/tests/test_letdoutil.py) for some examples. (Note the unparsing is done for display only; the raw value remains inspectable in the exception instance.) **CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See `mcpyrate.unparse`. From 78643c766bcdbe581a234c2c7b1b6e26780b3237 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 03:05:16 +0300 Subject: [PATCH 569/832] 0.15.0: improve let macro docs --- doc/macros.md | 63 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 26 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 5207c703..0b1da75e 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -101,7 +101,7 @@ Macros that introduce new ways to bind identifiers. **Changed in v0.15.0.** *Added support for env-assignment syntax in the bindings subform. For consistency with other env-assignments, this is now the preferred syntax to establish let bindings. Additionally, the old lispy syntax now accepts also brackets, for consistency with the use of brackets for macro invocations.* -Properly lexically scoped `let` constructs, no boilerplate: +These macros provide properly lexically scoped `let` constructs, no boilerplate: ```python from unpythonic.syntax import macros, let, letseq, letrec @@ -111,7 +111,7 @@ let[x << 17, # parallel binding, i.e. bindings don't see each other print(x, y)] letseq[x << 1, # sequential binding, i.e. Scheme/Racket let* - y << x+1][ + y << x + 1][ print(x, y)] letrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding, sequentially evaluated @@ -133,12 +133,13 @@ The same syntax for the bindings subform is used by: - `let`, `letseq`, `letrec` (expressions) - `dlet`, `dletseq`, `dletrec`, `blet`, `bletseq`, `bletrec` (decorators) + - As of v0.15.0, it is possible to use `@dlet(...)` instead of `@dlet[...]` in Python 3.8 and earlier. - `let_syntax`, `abbrev` (expression mode) #### Haskelly let-in, let-where -The following Haskell-inspired, perhaps more pythonic alternate syntaxes are also available: +The following Haskell-inspired, perhaps more pythonic alternative syntaxes are also available: ```python let[[x << 21, @@ -170,16 +171,16 @@ The `where` operator, if used, must be macro-imported. It may only appear at the > >In the first variant above (the *let-in*), note that even there, the bindings block needs the brackets. This is due to Python's precedence rules; `in` binds more strongly than the comma (which makes sense almost everywhere else), so to make the `in` refer to all of the bindings, the bindings block must be bracketed. If the `let` expander complains your code does not look like a `let` form and you have used *let-in*, check your brackets. > ->In the second variant (the *let-where*), note the comma between the body and `where`; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression. It's not only syntactically valid Python, it's also syntactically valid English (at least for mathematicians).) +>In the second variant (the *let-where*), note the comma between the body and `where`; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression. It's not only syntactically valid Python, it is also syntactically valid English, at least for mathematicians.)
-#### Alternate syntaxes for the bindings subform +#### Alternative syntaxes for the bindings subform **Changed in v0.15.0.** -Beginning with v0.15.0, the env-assignment syntax presented above is the preferred syntax to establish let bindings, for consistency with other env-assignments. (Let variables live in an `env`, which is created by the `let`.) +Beginning with v0.15.0, the env-assignment syntax presented above is the preferred syntax to establish let bindings, for consistency with other env-assignments. This reminds that let variables live in an `env`, which is created by the `let` form. -There is also an alternate, lispy notation for the bindings subform, where each name-value pair is given using brackets: +There is also an alternative, lispy notation for the bindings subform, where each name-value pair is given using brackets: ```python let[[x, 42], [y, 9001]][...] @@ -218,7 +219,7 @@ The issue has been fixed in Python 3.9. If you already only use 3.9 and later, p #### Multiple expressions in body -The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body ([like in `multilambda`](#multilambda-supercharge-your-lambdas)): +The `let` constructs can use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body ([like in `multilambda`](#multilambda-supercharge-your-lambdas)): ```python let[x << 1, @@ -237,9 +238,9 @@ let[[y << x + y, y << 2]] ``` -The let macros implement this by inserting a `do[...]` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the `let`; see [`do` for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). +The let macros implement this by inserting a `do[...]` (see below). In a multiple-expression body, a separate internal definition context exists for local variables that are not part of the `let`; see [the `do` macro for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). -Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a `let` form with only one body expression, use three sets of brackets: +Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a `let` form with only one body expression, double the brackets on the *body* part: ```python let[x << 1, @@ -255,7 +256,7 @@ let[[[x, y]], y << 2]] ``` -The outermost brackets delimit the `let` form, the middle ones activate multiple-expression mode, and the innermost ones denote a list. +The outermost brackets delimit the `let` form itself, the middle ones activate multiple-expression mode, and the innermost ones denote a list. Only brackets are affected; parentheses are interpreted as usual, so returning a literal tuple works as expected: @@ -275,11 +276,11 @@ let[(x, y), #### Notes -The main difference of the `let` family to Python's own named expressions (a.k.a. walrus operator, added in Python 3.8) is that `x := 42` does not create a scope, but `let[(x, 42)][...]` does. The walrus operator assigns to the name `x` in the scope it appears in, whereas in the `let` expression, the `x` only exists in that expression. +The main difference of the `let` family to Python's own named expressions (a.k.a. the walrus operator, added in Python 3.8) is that `x := 42` does not create a scope, but `let[x << 42][...]` does. The walrus operator assigns to the name `x` in the scope it appears in, whereas in the `let` expression, the `x` only exists in that expression. `let` and `letrec` expand into the `unpythonic.lispylet` constructs, implicitly inserting the necessary boilerplate: the `lambda e: ...` wrappers, quoting variable names in definitions, and transforming `x` to `e.x` for all `x` declared in the bindings. Assignment syntax `x << 42` transforms to `e.set('x', 42)`. The implicit environment parameter `e` is actually named using a gensym, so lexically outer environments automatically show through. `letseq` expands into a chain of nested `let` expressions. -Nesting utilizes an inside-out macro expansion order: +All the `let` macros respect lexical scope, so this works as expected: ```python letrec[z << 1][[ @@ -288,12 +289,12 @@ letrec[z << 1][[ print(z)]]] ``` -Hence the `z` in the inner scope expands to the inner environment's `z`, which makes the outer expansion leave it alone. (This works by transforming only `ast.Name` nodes, stopping recursion when an `ast.Attribute` is encountered.) +The `z` in the inner `letrec` expands to the inner environment's `z`, and the `z` in the outer `letrec` to the outer environment's `z`. ### `dlet`, `dletseq`, `dletrec`, `blet`, `bletseq`, `bletrec`: decorator versions -Similar to `let`, `letseq`, `letrec`, these sugar the corresponding `unpythonic.lispylet` constructs, with the `dletseq` and `bletseq` constructs existing only as macros (expanding to nested `dlet` or `blet`, respectively). +Similar to `let`, `letseq`, `letrec`, these macros sugar the corresponding `unpythonic.lispylet` constructs, with the `dletseq` and `bletseq` constructs existing only as macros. They expand to nested `dlet` or `blet`, respectively. Lexical scoping is respected; each environment is internally named using a gensym. Nesting is allowed. @@ -304,7 +305,7 @@ from unpythonic.syntax import macros, dlet, dletseq, dletrec, blet, bletseq, ble @dlet[x << 0] # up to Python 3.8, use `@dlet(x << 0)` instead def count(): - x << x + 1 + x << x + 1 # update `x` in let env return x assert count() == 1 assert count() == 2 @@ -351,7 +352,7 @@ The write of a `name << value` always occurs to the lexically innermost environm As an exception to the rule, for the purposes of the scope analysis performed by `unpythonic.syntax`, creations and deletions *of lexical local variables* take effect from the next statement, and remain in effect for the **lexically** remaining part of the current scope. This allows `x = ...` to see the old bindings on the RHS, as well as allows the client code to restore access to a surrounding env's `x` (by deleting a local `x` shadowing it) when desired. -To clarify, here's a sampling from the unit tests: +To clarify, here is a sampling from [the unit tests](../unpythonic/syntax/tests/test_letdo.py): ```python @dlet[x << "the env x"] @@ -408,7 +409,7 @@ else: *To rename existing macros, you can as-import them. As of `unpythonic` v0.15.0, doing so for `unpythonic.syntax` constructs is not recommended, though, because there is still a lot of old analysis code in the macro implementations that may scan for the original name. This may or may not be fixed in a future release.* -These constructs allow to locally splice code at macro expansion time (it's almost like inlining functions): +These constructs allow to locally splice code at macro expansion time. It is almost like inlining functions. #### `let_syntax` @@ -482,9 +483,9 @@ The `expr` and `block` operators, if used, must be macro-imported. They may only > >Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each `a` in the body of `twice` separately expands to a copy of whatever code was given as the macro argument `a`. > ->When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. (The single expression can be an explicit `do[]` if multiple expressions are needed.) When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. +>When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. The single expression can be an explicit `do[]`, if multiple expressions are needed. When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. > ->(If you know about Python ASTs, don't worry about the `ast.Expr` wrapper needed to place an expression in a statement position; this is handled automatically.) +>(If you know about Python ASTs, do not worry about the `ast.Expr` wrapper needed to place an expression in a statement position; this is handled automatically.)

@@ -507,13 +508,13 @@ The `expr` and `block` operators, if used, must be macro-imported. They may only

-Nesting `let_syntax` is allowed. Lexical scoping is supported (inner definitions of substitutions shadow outer ones). +Nesting `let_syntax` is allowed. Lexical scoping is respected. Inner definitions of substitutions shadow outer ones. -When used as an expr macro, all bindings are registered first, and then the body is evaluated. When used as a block macro, a new binding (substitution declaration) takes effect from the next statement onward, and remains active for the lexically remaining part of the `with let_syntax:` block. +When used as an expr macro, all bindings are registered first, and then the body is evaluated. When used as a block macro, a new binding (substitution declaration) takes effect from the next statement onward, and remains active for the lexically remaining part of the `with let_syntax` block. #### `abbrev` -The `abbrev` macro is otherwise exactly like `let_syntax`, but it expands outside-in. Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the `abbrev` itself expands before any macros invoked in its body. This allows things like: +The `abbrev` macro is otherwise exactly like `let_syntax`, but it expands outside-in. Hence, it has no lexically scoped nesting support, but it has the power to locally rename also macros, because the `abbrev` itself expands before any macros invoked in its body. This allows things like: ```python abbrev[m << macrowithverylongname][ @@ -524,18 +525,18 @@ abbrev[m[tree1] if m[tree2] else m[tree3], where[m << macrowithverylongname]] ``` -which can be useful when writing macros. +which is sometimes useful when writing macros. (But using `mcpyrate`, note that you can just as-import a macro if you need to rename it.) **CAUTION**: `let_syntax` is essentially a toy macro system within the real macro system. The usual caveats of macro systems apply. Especially, `let_syntax` and `abbrev` support absolutely no form of hygiene. Be very, very careful to avoid name conflicts. The `let_syntax` macro is meant for simple local substitutions where the elimination of repetition can shorten the code and improve its readability, in cases where the final "unrolled" code should be written out at compile time. If you need to do something complex (or indeed save a definition and reuse it somewhere else, non-locally), write a real macro directly in `mcpyrate`. -This was inspired by Racket's [`let-syntax`](https://docs.racket-lang.org/reference/let.html) and [`with-syntax`](https://docs.racket-lang.org/reference/stx-patterns.html). +This was inspired by Racket's [`let-syntax`](https://docs.racket-lang.org/reference/let.html) and [`with-syntax`](https://docs.racket-lang.org/reference/stx-patterns.html) forms. ### Bonus: barebones `let` -As a bonus, we provide classical simple `let` and `letseq`, wholly implemented as AST transformations, providing true lexical variables but no assignment support (because in Python, assignment is a statement) or multi-expression body support. Just like in Lisps, this version of `letseq` (Scheme/Racket `let*`) expands into a chain of nested `let` expressions, which expand to lambdas. +As a bonus, we provide classical simple `let` and `letseq`, wholly implemented as AST transformations, providing true lexical variables, but no multi-expression body support. Just like in some Lisps, this version of `letseq` (Scheme/[Racket `let*`](https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%2A%29%29)) expands into a chain of nested `let` expressions, which expand to lambdas. These are provided in the separate module `unpythonic.syntax.simplelet`, and are not part of the `unpythonic.syntax` macro API. For simplicity, they support only the lispy list syntax in the bindings subform (using brackets, specifically!), and no haskelly syntax at all: @@ -548,6 +549,16 @@ letseq[[x, 1], [x, x + 1]][...] letseq[[x, 1]][...] ``` +Starting with Python 3.8, assignment (rebinding) is possible also in these barebones `let` constructs via the walrus operator. For example: + +```python +assert let[[x, 42]][x] == 42 +assert let[[x, 42]][(x := 5)] == 5 +``` + +However, this only works for variables created by the innermost `let` (viewed from the point where the assignment happens), because `nonlocal` is a statement and so cannot be used in expressions. + + ## Sequencing Macros that run multiple expressions, in sequence, in place of one expression. From 352ea7e452c10be68d84733a9830e2886b59a54c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 03:09:30 +0300 Subject: [PATCH 570/832] fix borked formatting --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index c3ab2cfb..f1eab196 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4393,7 +4393,7 @@ print(result) # 6 If you want an `env` instance, see `blet` and `bletrec`. -#### Letrec without `letrec`*, when a statement is acceptable +#### Letrec without `letrec`, when a statement is acceptable ```python from unpythonic import call From a87d8505f316061a4ecaba08a1cd59372c227dbd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 03:11:25 +0300 Subject: [PATCH 571/832] styling --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index f1eab196..0e330db3 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4603,7 +4603,7 @@ assert result.rets[0] == 42 assert result.ret == 42 # shorthand for single-value case ``` -The last example is silly, but legal, because it is preferable to just omit the `Values` if it is known that there is only one return value. (This also applies when that value is a `tuple`, when the intent is to return it as a single `tuple`, in contexts where this distinction matters.) +The last example is silly, but legal, because it is preferable to just omit the `Values` if it is known that there is only one return value. This also applies when that value is a `tuple`, when the intent is to return it as a single `tuple`, in contexts where this distinction matters. ### `valuify` From 40cd291a3826f403d6081a2414756f1bc7a90ba3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 03:13:18 +0300 Subject: [PATCH 572/832] styling --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 0e330db3..5d92e743 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4928,7 +4928,7 @@ We special-case the builtin functions that either fail to return any arity (are If the arity cannot be inspected, and the function is not one of the special-cased builtins, the `UnknownArity` exception is raised. -Up to v0.14.3, various places in unpythonic used to internally use `arities`; particularly `curry`, `fix`, and `@generic`. As of v0.15.0, we have started to prefer `resolve_bindings`, because often what matters are the parameter bindings established, and performing the binding covers all possible ways to pass arguments. The `let` and FP looping constructs still use `arities` to emit a meaningful error message if the signature of user-provided function does not match what is expected. +Up to v0.14.3, various places in `unpythonic` used to internally use `arities`; particularly `curry`, `fix`, and `@generic`. As of v0.15.0, we have started to prefer `resolve_bindings`, because often what matters are the parameter bindings established, and performing the binding covers all possible ways to pass arguments. The `let` and FP looping constructs still use `arities` to emit a meaningful error message if the signature of user-provided function does not match what is expected. Inspired by various Racket functions such as `(arity-includes?)` and `(procedure-keywords)`. From 14e4c7f2955b97e17078f5668150c5d6c452865e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 03:14:26 +0300 Subject: [PATCH 573/832] wording: check what the tests *do* It doesn't matter what they *say*. --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index 5d92e743..48d381cd 100644 --- a/doc/features.md +++ b/doc/features.md @@ -126,7 +126,7 @@ The exception are the features marked **[M]**, which are primarily intended as a For many examples, see [the unit tests](unpythonic/tests/), the docstrings of the individual features, and this guide. -*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out-of-date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* +*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out-of-date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests do - and optionally file an issue on GitHub so that the documentation can be fixed.* **This document is up-to-date for v0.15.0.** From ad45c8add14b6cae4ab4d2ab55e778a62e1b4795 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 18 Jun 2021 10:47:51 +0300 Subject: [PATCH 574/832] readings: add Matthew Might's post on first-class macros --- doc/readings.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/readings.md b/doc/readings.md index 7acdf802..1d6b9ada 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -214,6 +214,9 @@ The common denominator is programming. Some relate to language design, some to c - Discussion on how programming languages *have* improved. - Contains interesting viewpoints, such as dmbarbour's suggestion that much of modern hardware is essentially "compiled" from a hardware description language such as VHDL. +- [Matthew Might: First-class (run-time) macros and meta-circular evaluation](https://matt.might.net/articles/metacircular-evaluation-and-first-class-run-time-macros/) + - *First-class macros are macros that can be bound to variables, passed as arguments and returned from functions. First-class macros expand and evaluate syntax at run-time.* + # Python-related FP resources From 2e1d55c3fcdf02b88573f6e88656a067c3fa209f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 19 Jun 2021 02:15:51 +0300 Subject: [PATCH 575/832] fix: as of 0.15.0, the underscore macro is `fn[]` --- doc/features.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/features.md b/doc/features.md index 48d381cd..7a0f8801 100644 --- a/doc/features.md +++ b/doc/features.md @@ -4481,13 +4481,13 @@ m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) assert tuple(m) == (6, 9, 3**(1/2)) ``` -If you use the quick lambda macro `f[]` (underscore notation for Python), these features combine nicely: +If you use the quick lambda macro `fn[]` (underscore notation for Python), these features combine nicely: ```python -from unpythonic.syntax import macros, f +from unpythonic.syntax import macros, fn from unpythonic import callwith -m = map(callwith(3), [f[2 * _], f[_**2], f[_**(1/2)]]) +m = map(callwith(3), [fn[2 * _], fn[_**2], fn[_**(1/2)]]) assert tuple(m) == (6, 9, 3**(1/2)) ``` From 69533c4e18c33273d79885d4ba9b277e697d0ee6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 00:16:17 +0300 Subject: [PATCH 576/832] add partition_int_custom --- CHANGELOG.md | 1 + doc/features.md | 18 ++++++++---- unpythonic/numutil.py | 49 ++++++++++++++++++++------------ unpythonic/tests/test_numutil.py | 21 +++++++++++++- 4 files changed, 65 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 708e939f..74d63a86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,6 +98,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - Add `resolve_bindings_partial`, useful for analyzing partial application. - Add `triangular`, to generate the triangular numbers (1, 3, 6, 10, ...). - Add `partition_int_triangular` to answer a timeless question concerning stackable plushies. + - Add `partition_int_custom` to answer unanticipated similar questions. - All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.) - Python 3.8 and 3.9 support added. diff --git a/doc/features.md b/doc/features.md index 7a0f8801..5129af60 100644 --- a/doc/features.md +++ b/doc/features.md @@ -111,7 +111,7 @@ The exception are the features marked **[M]**, which are primarily intended as a [**Numerical tools**](#numerical-tools) - [`almosteq`: floating-point almost-equality](#almosteq-floating-point-almost-equality) - [`fixpoint`: arithmetic fixed-point finder](#fixpoint-arithmetic-fixed-point-finder) - - [`partition_int`, `partition_int_triangular`: partition integers](#partition_int-partition_int_triangular-partition-integers) + - [`partition_int`: partition integers](#partition_int-partition-integers) - [`ulp`: unit in last place](#ulp-unit-in-last-place) [**Other**](#other) @@ -4681,9 +4681,9 @@ assert abs(sqrt_newton(2) - sqrt(2)) <= ulp(1.414) ``` -### `partition_int`, `partition_int_triangular`: partition integers +### `partition_int`: partition integers -**Changed in v0.15.0.** *Added `partition_int_triangular`.* +**Changed in v0.15.0.** *Added `partition_int_triangular` and `partition_int_custom`.* **Added in v0.14.2.** @@ -4693,10 +4693,13 @@ The `partition_int` function [partitions](https://en.wikipedia.org/wiki/Partitio The `partition_int_triangular` function is like `partition_int`, but accepts only triangular numbers (1, 3, 6, 10, ...) as components of the partition. This function answers a timeless question: if I have `n` stackable plushies, what are the possible stack configurations? +The `partition_int_custom` function is like `partition_int`, but lets you specify which numbers are acceptable as components of the partition. + Examples: ```python -from unpythonic import partition_int, partition_int_triangular +from itertools import count, takewhile +from unpythonic import partition_int, partition_int_triangular, rev assert tuple(partition_int(4)) == ((4,), (3, 1), (2, 2), (2, 1, 1), (1, 3), (1, 2, 1), (1, 1, 2), (1, 1, 1, 1)) assert tuple(partition_int(5, lower=2)) == ((5,), (3, 2), (2, 3)) @@ -4708,13 +4711,18 @@ assert (frozenset(tuple(sorted(c)) for c in partition_int_triangular(78, lower=1 (15, 21, 21, 21), (21, 21, 36), (78,)})) + +evens_upto_n = lambda n: takewhile(lambda m: m <= n, count(start=2, step=2)) +assert tuple(partition_int_custom(6, rev(evens_upto_n(6)))) == ((6,), (4, 2), (2, 4), (2, 2, 2)) ``` As the first example demonstrates, most of the splits are a ravioli consisting mostly of ones. It is much faster to not generate such splits than to filter them out from the result. Use the `lower` parameter to set the smallest acceptable value for one component of the split; the default value `lower=1` generates all splits. Similarly, the `upper` parameter sets the largest acceptable value for one component of the split. The default `upper=None` sets no upper limit, so in effect the upper limit becomes `n`. In `partition_int_triangular`, the `lower` and `upper` parameters work exactly the same. The only difference to `partition_int` is that each component of the split must be a triangular number. -**CAUTION**: The number of possible partitions grows very quickly with `n`, so in practice these functions are only useful for small numbers, or with a lower limit that is not too much smaller than `n / 2`. +In `partition_int_custom`, the components are given as an iterable, which is immediately forced (so if it is consumable, it will be completely consumed; and if it is infinite, the function will use up all available RAM and not terminate). Each component `x` must be an integer that satisfies `1 <= x <= n`. + +**CAUTION**: The number of possible partitions grows very quickly with `n`, so in practice these functions are only useful for small numbers, or when the smallest allowed component is not too much smaller than `n / 2`. ### `ulp`: unit in last place diff --git a/unpythonic/numutil.py b/unpythonic/numutil.py index 3f6defef..e70ff29d 100644 --- a/unpythonic/numutil.py +++ b/unpythonic/numutil.py @@ -3,13 +3,13 @@ __all__ = ["almosteq", "ulp", "fixpoint", - "partition_int", "partition_int_triangular"] + "partition_int", "partition_int_triangular", "partition_int_custom"] from itertools import takewhile from math import floor, log2 import sys -from .it import iterate1, last, within +from .it import iterate1, last, within, rev from .symbol import sym # HACK: break dependency loop mathseq -> numutil -> mathseq @@ -162,7 +162,7 @@ def partition_int(n, lower=1, upper=None): if lower < 1 or upper < 1 or lower > n or upper > n or lower > upper: raise ValueError(f"it must hold that 1 <= lower <= upper <= n; got lower={lower}, upper={upper}") - return _partition_int(n, range(min(n, upper), lower - 1, -1)) # instantiate the generator + return partition_int_custom(n, range(min(n, upper), lower - 1, -1)) # instantiate the generator def partition_int_triangular(n, lower=1, upper=None): """Like `partition_int`, but allow only triangular numbers in the result. @@ -199,26 +199,39 @@ def partition_int_triangular(n, lower=1, upper=None): triangulars_upto_n = takewhile(lambda m: m <= n, triangular()) - return _partition_int(n, filter(lambda m: lower <= m <= upper, - triangulars_upto_n)) + return partition_int_custom(n, rev(filter(lambda m: lower <= m <= upper, + triangulars_upto_n))) -def _partition_int(n, components): - """Implementation for `partition_int`, `partition_triangular`. +def partition_int_custom(n, components): + """Partition an integer in a custom way. `n`: integer to partition. `components`: iterable of ints; numbers that are allowed to appear in the partitioning result. Each number `m` must satisfy `1 <= m <= n`. + + See `partition_int`, `partition_triangular`. """ - # TODO: Check contracts on input? This is an internal function for now, so no validation. + if not isinstance(n, int): + raise TypeError(f"n must be integer; got {type(n)} with value {repr(n)}") + if n < 1: + raise ValueError(f"n must be positive; got {n}") components = tuple(components) - for k in components: - m = n - k - if m == 0: - yield (k,) - else: - out = [] - for item in _partition_int(m, (x for x in components if x <= m)): - out.append((k,) + item) - for term in out: - yield term + invalid_components = [not isinstance(x, int) for x in components] + if any(invalid_components): + raise TypeError(f"each component must be an integer; got invalid components {invalid_components}") + invalid_components = [not (1 <= x <= n) for x in components] + if any(invalid_components): + raise ValueError(f"each component x must be 1 <= x <= n; got n = {n}, with invalid components {invalid_components}") + def rec(components): + for k in components: + m = n - k + if m == 0: + yield (k,) + else: + out = [] + for item in partition_int_custom(m, tuple(x for x in components if x <= m)): + out.append((k,) + item) + for term in out: + yield term + return rec(components) diff --git a/unpythonic/tests/test_numutil.py b/unpythonic/tests/test_numutil.py index 984d0bc3..7870a4bd 100644 --- a/unpythonic/tests/test_numutil.py +++ b/unpythonic/tests/test_numutil.py @@ -3,10 +3,13 @@ from ..syntax import macros, test, test_raises, error, the # noqa: F401 from ..test.fixtures import session, testset +from itertools import count, takewhile from math import cos, sqrt import sys -from ..numutil import almosteq, fixpoint, partition_int, partition_int_triangular, ulp +from ..numutil import (almosteq, fixpoint, ulp, + partition_int, partition_int_triangular, partition_int_custom) +from ..it import rev def runtests(): with testset("ulp (unit in the last place; float utility)"): @@ -87,6 +90,22 @@ def sqrt_iter(x): # has an attractive fixed point at sqrt(n) (21, 21, 36), (78,)})] + # partition_int_custom: like partition_int, but lets you specify allowed components manually. + # Can be used to build other functions like `partition_int` and `partition_int_triangular`. + with testset("partition_int_custom"): + test[tuple(partition_int_custom(4, [1])) == ((1, 1, 1, 1),)] + test[tuple(partition_int_custom(4, [1, 3])) == ((1, 1, 1, 1), (1, 3), (3, 1))] + + evens_upto_n = lambda n: takewhile(lambda m: m <= n, count(start=2, step=2)) + test[tuple(partition_int_custom(4, rev(evens_upto_n(4)))) == ((4,), (2, 2))] + test[tuple(partition_int_custom(6, rev(evens_upto_n(6)))) == ((6,), (4, 2), (2, 4), (2, 2, 2))] + + test_raises[TypeError, partition_int_custom("not a number", evens_upto_n("blah"))] + test_raises[TypeError, tuple(partition_int_custom(4, [2.0]))] + test_raises[ValueError, partition_int_custom(-3, evens_upto_n(-3))] + test_raises[ValueError, tuple(partition_int_custom(4, [-1]))] + test_raises[ValueError, tuple(partition_int_custom(4, [1, -1]))] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From b4df10e36d8b63313d9910da502989e12ae0513b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 02:01:45 +0300 Subject: [PATCH 577/832] add name resolution caution --- doc/macros.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/doc/macros.md b/doc/macros.md index 0b1da75e..7011ebf6 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -31,6 +31,7 @@ Because in Python macro expansion occurs *at import time*, Python programs whose [**Bindings**](#bindings) - [`let`, `letseq`, `letrec` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. - [`dlet`, `dletseq`, `dletrec`, `blet`, `bletseq`, `bletrec`: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) +- [Caution on name resolution and scoping](#caution-on-name-resolution-and-scoping) - [`let_syntax`, `abbrev`: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. - [Bonus: barebones `let`](#bonus-barebones-let): pure AST transformation of `let` into a `lambda`. @@ -401,6 +402,49 @@ else: ``` +### Caution on name resolution and scoping + +The name resolution behavior described above **does not fully make sense**, because to define things this way is to conflate static (lexical) and dynamic (run-time) concepts. This feature unfortunately got built before I understood the matter clearly. + +Python itself performs name resolution purely lexically, which is arguably the right thing to do. In any given lexical scope, an identifier such as `x` always refers to the same variable. Whether that variable has been initialized, or has already been deleted, is another matter, which has to wait until run time - but `del x` will **not** cause the identifier `x` to point to a different variable for the remainder of the same scope, like `delete[x]` **does** in the body of an `unpythonic` `let[]` or `do[]`. + +#### Aside: Names and variables + +To be technically correct, in Python, an identifier `x` refers to a *name*, not to a "variable". Python, like Lisp, has [*names and values*](https://nedbatchelder.com/text/names.html). + +Roughly, an *identifier* is a certain kind of token in the source code text - something that everyday English calls a "name". However, in programming, a *name* is technically the *key* component of a key-value pair that is stored in a particular *environment*. + +Very roughly speaking, an *environment* is just a place to store such pairs, for the purposes of "the variables subsystem" of the language. There are important details, such as that each *activation* of a function (think: "a particular call of the function") will create a new environment instance, to hold the local variables of that activation; this detail allows [lexical closures](https://en.wikipedia.org/wiki/Closure_(computer_programming)) to work. The piece of bookkeeping for this is termed an *activation record*. But the important point here is, an environment stores name-value pairs. + +An identifier *refers to* a name. Scoping rules concern themselves with the details of mapping identifiers to names. In *lexical scoping* (like in Python), the position of the identifier in the source code text determines the search order of environments for the target name, when resolving a particular instance of an identifier in the source code text. Python uses the LEGB ordering (local, enclosing, global, builtin). + +Finally, *values* are the run-time things names point to. They are the *value* component of the key-value pair. + +In this simple example: + +```python +def outer(): + x = 17 + def inner(): + x = 23 +``` + + - The piece of source code text `x` is an *identifier*. + - *The outer `x`* and *the inner `x`* are *names*, both of which have the textual representation `x`. + - *Which one of these the identifier `x` refers to depends on where it appears.* + - The integers `17` and `23` are *values*. + +Note that classically, names have no type; values do. + +Nowadays, a name may have a type annotation, which reminds the programmer about the type of *value* that is safe to bind to that particular name. In other words, the code that defines that name (e.g. as a function parameter) promises (in the sense of a contract) that the code knows how to behave if a value of that type is bound to that name (e.g. by passing such a value as a function argument that will be bound to that name). + +Here *type* may be a concrete [nominal type](https://en.wikipedia.org/wiki/Nominal_type_system) such as `int`, or for example, it may represent a particular interface (such as the types in [`collections.abc`](https://docs.python.org/3/library/collections.abc.html)), or it may allow multiple mutually exclusive options (a *union*). + +By default, Python treats type annotations as a form of comments; to actually statically type-check Python, [Mypy](http://mypy-lang.org/) can be used. + +Compare the *name*/*value* concept to the concept of a *variable* in the classical sense, such as in C, or `cdef` in Cython. In such *low-level* [HLLs](https://en.wikipedia.org/wiki/High-level_programming_language), a *variable* is a named, fixed memory location, with a static data type determining how to interpret the bits at that memory location. The contents of the memory location can be changed, hence "variable" is an apt description. + + ### `let_syntax`, `abbrev`: syntactic local bindings **Note v0.15.0.** *Now that we use `mcpyrate` as the macro expander, `let_syntax` and `abbrev` are not really needed. We are keeping them mostly for backwards compatibility, and because they exercise a different feature set in the macro expander, making the existence of these constructs particularly useful for system testing.* From 1d37edd4698b50abe975f1e9b156a38aeb6ae7c0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 02:02:15 +0300 Subject: [PATCH 578/832] some wording changes for `do[]` docs --- doc/macros.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 7011ebf6..effd7377 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -611,7 +611,7 @@ Macros that run multiple expressions, in sequence, in place of one expression. We provide an `expr` macro wrapper for `unpythonic.seq.do`, with some extra features. -This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see [the functions in `unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). +This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see [the functions in `unpythonic.fploop`](../unpythonic/fploop.py) (`looped` and `looped_over`). ```python from unpythonic.syntax import macros, do, local, delete @@ -630,7 +630,7 @@ y = do[local[a << 17], True] ``` -Local variables are declared and initialized with `local[var << value]`, where `var` is a bare name. To explicitly denote "no value", just use `None`. `delete[...]` allows deleting a `local[...]` binding. This uses `env.pop()` internally, so a `delete[...]` returns the value the deleted local variable had at the time of deletion. (So if you manually use the `do()` function in some code without macros, feel free to `env.pop()` in a do-item if needed.) +Local variables are declared and initialized with `local[var << value]`, where `var` is a bare name. To explicitly denote "no value", just use `None`. The syntax `delete[...]` allows deleting a `local[...]` binding. This uses `env.pop()` internally, so a `delete[...]` returns the value the deleted local variable had at the time of deletion. (This also means that if you manually use the `do()` function in some code without macros, you can `env.pop(...)` in a do-item if needed.) The `local[]` and `delete[]` declarations may only appear at the top level of a `do[]`, `do0[]`, or implicit `do` (extra bracket syntax, e.g. for the body of a `let` form). In any invalid position, `local[]` and `delete[]` are considered a syntax error at macro expansion time. @@ -659,7 +659,8 @@ Already declared local variables are updated with `var << value`. Updating varia

-**CAUTION**: `do[]` supports local variable deletion, but the `let[]` constructs don't, by design. When `do[]` is used implicitly with the extra bracket syntax, any `delete[]` refers to the scope of the implicit `do[]`, not any surrounding `let[]` scope. +**CAUTION**: `do[]` supports local variable deletion, but the `let[]` constructs do **not**, by design. When `do[]` is used implicitly with the extra bracket syntax, any `delete[]` refers to the scope of the implicit `do[]`, not any surrounding `let[]` scope. + ## Tools for lambdas From f33b0d07a648229d5032e6d2b753952a63e98d49 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 16:30:18 +0300 Subject: [PATCH 579/832] 0.15.0: do macro docs wording changes vol 2 --- doc/macros.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index effd7377..ae569521 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -609,7 +609,7 @@ Macros that run multiple expressions, in sequence, in place of one expression. ### `do` as a macro: stuff imperative code into an expression, *with style* -We provide an `expr` macro wrapper for `unpythonic.seq.do`, with some extra features. +We provide an `expr` macro wrapper for `unpythonic.seq.do` and `unpythonic.seq.do0`, with some extra features. This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see [the functions in `unpythonic.fploop`](../unpythonic/fploop.py) (`looped` and `looped_over`). @@ -651,11 +651,9 @@ Already declared local variables are updated with `var << value`. Updating varia >Assignments are recognized anywhere inside the `do`; but note that any `let` constructs nested *inside* the `do`, that define variables of the same name, will (inside the `let`) shadow those of the `do` - as expected of lexical scoping. > ->The necessary boilerplate (notably the `lambda e: ...` wrappers) is inserted automatically, so the expressions in a `do[]` are only evaluated when the underlying `seq.do` actually runs. +>The boilerplate needed by the underlying `unpythonic.seq.do` form (notably the `lambda e: ...` wrappers) is inserted automatically. The expressions in a `do[]` are only evaluated when the underlying `unpythonic.seq.do` actually runs. > ->When running, `do` behaves like `letseq`; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites (this is afterall an imperative tool). -> ->We also provide a `do0` macro, which returns the value of the first expression, instead of the last. +>When running, `do` behaves like `letseq`; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites.

From 76b98c54d83454a924c28c7a2715b6ce68d727c3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 16:56:47 +0300 Subject: [PATCH 580/832] 0.15.0: update envify macro docs --- doc/macros.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index ae569521..f4b8174e 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -830,7 +830,7 @@ When a function whose definition (`def` or `lambda`) is lexically inside a `with Wherever could *that* be useful? For an illustrative caricature, consider [PG's accumulator puzzle](http://paulgraham.com/icad.html). -The modern pythonic solution: +The Python 3 solution: ```python def foo(n): @@ -841,11 +841,23 @@ def foo(n): return accumulate ``` -This avoids allocating an extra place to store the accumulator `n`. If you want optimal bytecode, this is the best solution in Python 3. +This avoids allocating an extra place to store the accumulator `n`. The Python 3.8+ solution, using the new walrus operator, is one line shorter: -But what if, instead, we consider the readability of the unexpanded source code? The definition of `accumulate` requires many lines for something that simple. What if we wanted to make it a lambda? Because all forms of assignment are statements in Python, the above solution is not admissible for a lambda, even with macros. +```python +def foo(n): + def accumulate(i): + nonlocal n + return (n := n + i) + return accumulate +``` + +This is rather clean, but still needs the `nonlocal` declaration, which is available as a statement only. + +If you want optimal bytecode, these two are the best solutions of the puzzle in Python. -So if we want to use a lambda, we have to create an `env`, so that we can write into it. Let's use the let-over-lambda idiom: +But what if we want to shorten the source code even more, for readability? We could make `accumulate` a lambda. But then, to rebind the `n` that lives in an enclosing scope - because Python does not support doing that from an expression position - we must make it live in an `unpythonic` `env`. + +Let's use the let-over-lambda idiom: ```python def foo(n0): @@ -853,7 +865,7 @@ def foo(n0): (lambda i: n << n + i)] ``` -Already better, but the `let` is used only for (in effect) altering the passed-in value of `n0`; we don't place any other variables into the `let` environment. Considering the source text already introduces an `n0` which is just used to initialize `n`, that's an extra element that could be eliminated. +This is already shorter, but the `let` is used only for (in effect) altering the passed-in value of `n0`; we do not place any other variables into the `let` environment. Considering the source text already introduces a name `n0` which is just used to initialize `n`, that's an extra element that could be eliminated. Enter the `envify` macro, which automates this: @@ -863,7 +875,7 @@ with envify: return lambda i: n << n + i ``` -Combining with `autoreturn` yields the fewest-elements optimal solution to the accumulator puzzle: +Combining with `autoreturn` yields the fewest-source-code-elements optimal solution to the accumulator puzzle: ```python with autoreturn, envify: @@ -871,7 +883,8 @@ with autoreturn, envify: lambda i: n << n + i ``` -The `with` block adds a few elements, but if desired, it can be refactored into the definition of a custom dialect in [Pydialect](https://github.com/Technologicat/pydialect). +The `with` block adds a few elements, but if desired, it can be refactored into the definition of a custom dialect using `mcpyrate`. See [dialect examples](dialects.md). + ## Language features From 9a90dcb1614941b0ac5838cb1d0f306f866f7201 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 17:08:17 +0300 Subject: [PATCH 581/832] update comment --- unpythonic/fun.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/fun.py b/unpythonic/fun.py index a8788f63..e3afecf7 100644 --- a/unpythonic/fun.py +++ b/unpythonic/fun.py @@ -38,7 +38,7 @@ # -------------------------------------------------------------------------------- -#def memoize_simple(f): # essential idea, without exception handling +#def memoize_simple(f): # essential idea, without exception handling or thread-safety. # memo = {} # @wraps(f) # def memoized(*args, **kwargs): From 7da9663a4e4c16feeaf90bd679bcf55f84d2ef61 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 17:08:29 +0300 Subject: [PATCH 582/832] 0.15.0: update `autocurry` macro docs --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index f4b8174e..7ead768b 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -920,7 +920,7 @@ assert add3(1)(2)(3) == 6 - Function calls are autocurried, and run `unpythonic.fun.curry` in a special mode that no-ops on uninspectable functions (triggering a standard function call with the given args immediately) instead of raising `TypeError` as usual. -**CAUTION**: Some built-ins are uninspectable or may report their arities incorrectly; in those cases, `curry` may fail, occasionally in mysterious ways. The function `unpythonic.arity.arities`, which `unpythonic.fun.curry` internally uses, has a workaround for the inspectability problems of all built-ins in the top-level namespace (as of Python 3.7), but e.g. methods of built-in types are not handled. +**CAUTION**: Some built-ins are uninspectable or may report their call signature incorrectly; in those cases, `curry` may fail, occasionally in mysterious ways. When inspection fails, `curry` raises ``ValueError``, like `inspect.signature` does. Manual uses of the `curry` decorator (on both `def` and `lambda`) are detected, and in such cases the macro skips adding the decorator. From f0d75e4c0e22fbb5517ece8d6f16315fa89f2ee3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 20 Jun 2021 17:24:25 +0300 Subject: [PATCH 583/832] 0.15.0: update lazify macro docs, vol 1 --- doc/macros.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 7ead768b..e76e2a31 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -926,13 +926,15 @@ Manual uses of the `curry` decorator (on both `def` and `lambda`) are detected, ### `lazify`: call-by-need for Python -**Changed in v0.15.0.** *Up to 0.14.x, the `lazy[]` macro, that is used together with `with lazify`, used to be provided by `macropy`, but now that we use `mcpyrate`, we provide it ourselves. If you use `lazy[]`, change your import of that macro to `from unpythonic.syntax import macros, lazy`*. +**Changed in v0.15.0.** *The `lazy[]` macro, that is used together with `with lazify`, used to be provided by `macropy` up to `unpythonic` v0.14.3. But now that we use `mcpyrate`, we provide a `lazy[]` macro and an underlying `Lazy` class ourselves. For details, see the separate section about `lazy[]` and `lazyrec[]` below.* Also known as *lazy functions*. Like [lazy/racket](https://docs.racket-lang.org/lazy/index.html), but for Python. Note if you want *lazy sequences* instead, Python already provides those; just use the generator facility (and decorate your gfunc with `unpythonic.gmemoize` if needed). Lazy function example: ```python +from unpythonic.syntax import macros, lazify + with lazify: def my_if(p, a, b): if p: @@ -957,7 +959,7 @@ Note `my_if` in the example is a regular function, not a macro. Only the `with l ```python from unpythonic.syntax import macros, lazy -from unpythonic.syntax import force +from unpythonic import force def my_if(p, a, b): if force(p): @@ -990,7 +992,11 @@ Inspired by Haskell, Racket's `(delay)` and `(force)`, and [lazy/racket](https:/ #### `lazy[]` and `lazyrec[]` macros -**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. Note that a lazy value now no longer has a `__call__` operator; instead, it has a `force()` method. The utility `unpythonic.lazyutil.force` (previously exported in `unpythonic.syntax`; now moved to the top-level namespace of `unpythonic`) abstracts away this detail.* +**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. We also now provide the underlying `Lazy` class ourselves.* + +*Note that a lazy value (an instance of `Lazy`) now no longer has a `__call__` operator; instead, it has a `force()` method. The preferred way to force a lazy value, however, is to use the top-level utility function `force`, which abstracts away this detail. It also helpfully passes its argument through if it is not a `Lazy`.* + +*The `force` function was previously exported in `unpythonic.syntax`; now it is available in the top-level namespace of `unpythonic`. This follows the general convention that regular functions live in the top-level `unpythonic` package, while macros (and in general, syntactic constructs) live in `unpythonic.syntax`.* We provide the macros `unpythonic.syntax.lazy`, which explicitly lazifies a single expression, and `unpythonic.syntax.lazyrec`, which can be used to lazify expressions inside container literals, recursively. From 29eb2fec788066e8c7891065c2782155df47bace Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 00:46:52 +0300 Subject: [PATCH 584/832] Some final wording changes for 0.15.0 --- doc/features.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/doc/features.md b/doc/features.md index 5129af60..7e881077 100644 --- a/doc/features.md +++ b/doc/features.md @@ -373,7 +373,7 @@ letrec[[evenp << (lambda x: ### `env`: the environment -The environment used by all the `let` constructs and `assignonce` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. It is somewhat similar to [`types.SimpleNamespace`](https://docs.python.org/3/library/types.html#types.SimpleNamespace), but with many extra features. For details, see `unpythonic.env`. +The environment used by all the `let` constructs and `assignonce` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. It is somewhat similar to [`types.SimpleNamespace`](https://docs.python.org/3/library/types.html#types.SimpleNamespace), but with many extra features. For details, see `unpythonic.env.env` (and note the unfortunate module name). Our `env` allows things like: @@ -1014,9 +1014,9 @@ f2 = lambda x: begin0(42 * x, f2(2) # --> 84 ``` -The `begin` and `begin0` forms are actually tuples in disguise; evaluation of all items occurs before the `begin` or `begin0` form gets control. Items are evaluated left-to-right due to Python's argument passing rules. +The `begin` and `begin0` forms are actually tuples in disguise; evaluation of **all** items occurs before the `begin` or `begin0` form gets control. Items are evaluated left-to-right due to Python's argument passing rules. -We provide also `lazy_begin` and `lazy_begin0`, which use loops. The price is the need for a lambda wrapper for each expression to delay evaluation, see [`unpythonic.seq`](../unpythonic/seq.py) for details. +We provide also `lazy_begin` and `lazy_begin0`, which use loops. The price is the need for a lambda wrapper for each expression to delay evaluation. See the module [`unpythonic.seq`](../unpythonic/seq.py) for details. ### `do`: stuff imperative code into an expression @@ -1665,7 +1665,7 @@ As of v0.15.0, the actual algorithm by which `curry` decides what to do, in the - Then, try for a partial match that passes the type check. **If any such match is found**, keep currying. - If none of the above match, it implies that no matter which multimethod we pick, at least one parameter will get a binding that fails the type check. Raise `TypeError`. -If interested in the gritty details, see [the source code](../unpythonic/fun.py) of `unpythonic.fun.curry`. It calls some functions from `unpythonic.dispatch` for its `@generic` support, but otherwise it is pretty much self-contained. +If interested in the gritty details, see [the source code](../unpythonic/fun.py) of `unpythonic.curry`, in the module `unpythonic.fun`. It calls some functions from the module `unpythonic.dispatch` for its `@generic` support, but otherwise it is pretty much self-contained. Getting back to the simple case, in the above example: @@ -2479,12 +2479,16 @@ The view can be efficiently iterated over. As usual, iteration assumes that no i Getting/setting an item (subscripting) checks whether the index cache needs updating during each access, so it can be a bit slow. Setting a slice checks just once, and then updates the underlying iterable directly. Setting a slice to a scalar value broadcasts the scalar à la NumPy. -The `unpythonic.collections` module also provides the `SequenceView` and `MutableSequenceView` abstract base classes; `view` is a `MutableSequenceView`. +Beside `view` itself, the `unpythonic.collections` module provides also some other related abstractions. -There is also the read-only cousin `roview`, which is like `view`, except it has no `__setitem__` or `reverse`. This can be useful for providing explicit read-only access to a sequence, when it is undesirable to have clients write into it. +There is the read-only sister of view, `roview`, which is like `view`, except it has no `__setitem__` or `reverse`. This can be useful for providing explicit read-only access to a sequence, when it is undesirable to have clients write into it. The constructor of the writable `view` checks that the input is not read-only (`roview`, or a `Sequence` that is not also a `MutableSequence`) before allowing creation of the writable view. +Finally, there are the `SequenceView` and `MutableSequenceView` abstract base classes. The concrete `view` and `roview` are instances of them. + +**NOTE**: A writable view supports also the read-only API, so `isinstance(MutableSequenceView, SequenceView) is True`; as well as `isinstance(view, roview) is True`. Keep in mind the [Liskov substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle). + ### `mogrify`: update a mutable container in-place @@ -2519,9 +2523,9 @@ For convenience, we support some special cases: If you want to process strings, implement it in your function that is called by `mogrify`. You can e.g. `tuple(thestring)` and then call `mogrify` on that. - - The `box`, `ThreadLocalBox` and `Some` containers from `unpythonic.collections`. Although the first two are mutable, their update is not conveniently expressible by the `collections.abc` APIs. + - The `box`, `ThreadLocalBox` and `Some` containers from the module `unpythonic.collections`. Although the first two are mutable, their update is not conveniently expressible by the `collections.abc` APIs. - - The `cons` container from `unpythonic.llist` (including the `ll`, `llist` linked lists). This is treated with the general tree strategy, so nested linked lists will be flattened, and the final `nil` is also processed. + - The `cons` container from the module `unpythonic.llist`, including linked lists created using `ll` or `llist`. This is treated with the general tree strategy, so nested linked lists will be flattened, and the final `nil` is also processed. Note that since `cons` is immutable, anyway, if you know you have a long linked list where you need to update the values, just iterate over it and produce a new copy - that will work as intended. @@ -3112,7 +3116,7 @@ def outer_result(outer_loop, y, outer_acc): assert outer_result == ((1, 2), (2, 4), (3, 6)) ``` -If you feel the trailing commas ruin the aesthetics, see `unpythonic.misc.pack`. +If you feel the trailing commas ruin the aesthetics, see `unpythonic.pack`. #### Accumulator type and runtime cost @@ -3509,7 +3513,7 @@ This `forall` is essentially a tuple comprehension that: - Allows filters to be placed at any level of the nested looping. - Presents the source code in the same order as it actually runs. -The `unpythonic.amb` module defines four operators: +The module `unpythonic.amb` defines four operators: - `forall` is the control structure, which marks a section that uses nondeterministic evaluation. - `choice` binds a name: `choice(x=range(3))` essentially means `for e.x in range(3):`. @@ -3752,7 +3756,7 @@ The terminology is: The term *multimethod* distinguishes them from the OOP sense of *method*, already established in Python, as well as reminds that multiple arguments participate in dispatching. -**CAUTION**: Code using the `with lazify` macro cannot usefully use `@generic` or `@typed`, because all arguments of each function call will be wrapped in a promise (`unpythonic.lazyutil.Lazy`) that carries no type information on its contents. +**CAUTION**: Code using the `with lazify` macro cannot usefully use `@generic` or `@typed`, because all arguments of each function call will be wrapped in a promise (`unpythonic.Lazy`) that carries no type information on its contents. #### `generic`: multiple dispatch with type annotation syntax @@ -4156,7 +4160,7 @@ test[tryf(lambda: raise_instance(), The exception handler is a function. It may optionally accept one argument, the exception instance. Just like in an `except` clause, the exception specification can be either an exception type, or a `tuple` of exception types. -Functions can also be specified to represent the `else` and `finally` blocks; the keyword parameters to do this are `elsef` and `finallyf`. Each of them is a thunk (a 0-argument function). See the docstring of `unpythonic.misc.tryf` for details. +Functions can also be specified to represent the `else` and `finally` blocks; the keyword parameters to do this are `elsef` and `finallyf`. Each of them is a thunk (a 0-argument function). See the docstring of `unpythonic.tryf` for details. Examples can be found in [the unit tests](../unpythonic/tests/test_excutil.py). @@ -4551,7 +4555,7 @@ The only exception is `__getitem__` (subscripting), which makes sense for both p If you need to explicitly access either part (and its full API), use the `rets` and `kwrets` attributes. The names are in analogy with `args` and `kwargs`. -`rets` is a `tuple`, and `kwrets` is an `unpythonic.collections.frozendict`. +`rets` is a `tuple`, and `kwrets` is an `unpythonic.frozendict`. `Values` objects can be compared for equality. Two `Values` objects are equal if both their `rets` and `kwrets` (respectively) are. From cc9b2875a4525282e7e41fe10dd9000605bafdac Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 00:51:21 +0300 Subject: [PATCH 585/832] 0.15.0: update lazify macro docs --- doc/macros.md | 49 ++++++++++++++++++++++++++----------------------- 1 file changed, 26 insertions(+), 23 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index e76e2a31..5d658763 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -992,25 +992,25 @@ Inspired by Haskell, Racket's `(delay)` and `(force)`, and [lazy/racket](https:/ #### `lazy[]` and `lazyrec[]` macros -**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. We also now provide the underlying `Lazy` class ourselves.* +**Changed in v0.15.0.** *Previously, the `lazy[]` macro was provided by MacroPy. Now that we use `mcpyrate`, which doesn't provide it, we provide it ourselves, in `unpythonic.syntax`. We now provide also the underlying `Lazy` class ourselves.* -*Note that a lazy value (an instance of `Lazy`) now no longer has a `__call__` operator; instead, it has a `force()` method. The preferred way to force a lazy value, however, is to use the top-level utility function `force`, which abstracts away this detail. It also helpfully passes its argument through if it is not a `Lazy`.* +*Note that a lazy value (an instance of `Lazy`) now no longer has a `__call__` operator; instead, it has a `force()` method. However, the preferred way is to use the top-level function `force`, which abstracts away this detail.* *The `force` function was previously exported in `unpythonic.syntax`; now it is available in the top-level namespace of `unpythonic`. This follows the general convention that regular functions live in the top-level `unpythonic` package, while macros (and in general, syntactic constructs) live in `unpythonic.syntax`.* We provide the macros `unpythonic.syntax.lazy`, which explicitly lazifies a single expression, and `unpythonic.syntax.lazyrec`, which can be used to lazify expressions inside container literals, recursively. -Essentially, `lazy[...]` achieves the same result as `memoize(lambda: ...)`, with the practical difference that a `lazy[]` promise `p` is evaluated by calling `unpythonic.lazyutil.force(p)` or `p.force()`. In `unpythonic`, the promise datatype (`unpythonic.lazyutil.Lazy`) does not have a `__call__` method, because the word `force` better conveys the intent. +Essentially, `lazy[...]` achieves the same result as `memoize(lambda: ...)`, with the practical difference that the `lazify` subsystem expects the `lazy[...]` notation in its analyzer, and will not recognize `memoize(lambda: ...)` as a delayed value. -It is preferable to use the `force` function instead of the `.force` method, because the function will also pass through any non-promise value, whereas (obviously) a non-promise value will not have a `.force` method. Using the function, you can `force` a value just to be sure, without caring whether that value was a promise. The `force` function is available in the top-level namespace of `unpythonic`. +A `lazy[]` promise `p` is evaluated by calling `force(p)` or `p.force()`. In `unpythonic`, the promise datatype (`Lazy`) does not have a `__call__` method, because the word `force` better conveys the intent. -The `lazify` subsystem expects the `lazy[...]` notation in its analyzer, and will not recognize `memoize(lambda: ...)` as a delayed value. +It is preferable to use the `force` top-level function instead of the `.force` method, because the function will also pass through any non-promise value, whereas (obviously) a non-promise value will not have a `.force` method. Using the function, you can `force` a value just to be sure, without caring whether that value was a promise. The `force` function is available in the top-level namespace of `unpythonic`. -The `lazyrec[]` macro allows code like `tpl = lazyrec[(1*2*3, 4*5*6)]`. Each item becomes wrapped with `lazy[]`, but the container itself is left alone, to avoid interfering with unpacking. Because `lazyrec[]` is a macro and must work by names only, it supports a fixed set of container types: `list`, `tuple`, `set`, `dict`, `frozenset`, `unpythonic.collections.frozendict`, `unpythonic.collections.box`, and `unpythonic.llist.cons` (specifically, the constructors `cons`, `ll` and `llist`). +The `lazyrec[]` macro allows code like `tpl = lazyrec[(1*2*3, 4*5*6)]`. Each item becomes wrapped with `lazy[]`, but the container itself is left alone, to avoid interfering with its unpacking. Because `lazyrec[]` is a macro and must work by names only, it supports a fixed set of container types: `list`, `tuple`, `set`, `dict`, `frozenset`, `unpythonic.frozendict`, `unpythonic.box`, and `unpythonic.cons` (specifically, the constructors `cons`, `ll` and `llist`). The `unpythonic` containers **must be from-imported** for `lazyrec[]` to recognize them. Either use `from unpythonic import xxx` (**recommended**), where `xxx` is a container type, or import the `containers` subpackage by `from unpythonic import containers`, and then use `containers.xxx`. (The analyzer only looks inside at most one level of attributes. This may change in the future.) -(The analysis in `lazyrec[]` must work by names only, because in an eager language any lazification must be performed as a syntax transformation before the code actually runs, so the analysis must be performed statically - and locally, because `lazyrec[]` is an expr macro. [Fexprs](https://fexpr.blogspot.com/2011/04/fexpr.html) (along with [a new calculus to go with them](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html)) are the clean, elegant solution, but this requires redesigning the whole language from ground up. Of course, if you're fine with a language not particularly designed for extensibility, and lazy evaluation is your top requirement, you could just use Haskell.) +Observe that the analysis in `lazyrec[]` must work by names only, because in an eager language any lazification must be performed as a syntax transformation before the code actually runs. Hence, the analysis must be performed statically - and locally, because `lazyrec[]` is an expr macro. [Fexprs](https://fexpr.blogspot.com/2011/04/fexpr.html) (along with [a new calculus to go with them](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html)) are the clean, elegant solution, but this requires redesigning the whole language from ground up. Of course, if you are fine with a language not particularly designed for extensibility, and lazy evaluation is your top requirement, you could just use Haskell. #### Forcing promises manually @@ -1020,57 +1020,60 @@ This is mainly useful if you `lazy[]` or `lazyrec[]` something explicitly, and w We provide the functions `force1` and `force`. Using `force1`, if `x` is a `lazy[]` promise, it will be forced, and the resulting value is returned. If `x` is not a promise, `x` itself is returned, à la Racket. The function `force`, in addition, descends into containers (recursively). When an atom `x` (i.e. anything that is not a container) is encountered, it is processed using `force1`. -Mutable containers are updated in-place; for immutables, a new instance is created, but as a side effect the promise objects **in the input container** will be forced. Any container with a compatible `collections.abc` is supported. (See `unpythonic.collections.mogrify` for details.) In addition, as special cases `unpythonic.collections.box` and `unpythonic.llist.cons` are supported. +Mutable containers are updated in-place; for immutables, a new instance is created, but as a side effect the promise objects **in the input container** will be forced. Any container with a compatible `collections.abc` is supported. (See `unpythonic.mogrify` for details.) In addition, as special cases `unpythonic.box` and `unpythonic.cons` are supported. #### Binding constructs and auto-lazification -Why do we auto-lazify in certain kinds of binding constructs, but not in others? Function calls and let-bindings have one feature in common: both are guaranteed to bind only new names (even if that name is already in scope, they are distinct; the new binding will shadow the old one). Auto-lazification of all assignments, on the other hand, in a language that allows mutation is dangerous, because then this superficially innocuous code will fail: +Why do we auto-lazify in certain kinds of binding constructs, but not in others? Function calls and let-bindings have one feature in common: both are guaranteed to bind only new names. Even if a name that uses the same identifier is already in scope, they are distinct; the new binding will shadow the old one. Auto-lazification of all assignments, on the other hand, in a language that allows mutation is dangerous, because then this superficially innocuous code will fail: ```python -a = 10 -a = 2*a -print(a) # 20, right? +from unpythonic.syntax import macros, lazify + +with lazify: + a = 10 + a = 2 * a + print(a) # 20, right? ``` -If we chose to auto-lazify assignments, then assuming a `with lazify` around the example, it would expand to: +If we chose to auto-lazify assignments, then the example would expand to: ```python from unpythonic.syntax import macros, lazy from unpythonic.syntax import force a = lazy[10] -a = lazy[2*force(a)] +a = lazy[2 * force(a)] print(force(a)) ``` -In the second assignment, the `lazy[]` sets up a promise, which will force `a` *at the time when the containing promise is forced*, but at that time the name `a` points to a promise, which will force... +Scan that again: in the second assignment, the `lazy[]` sets up a promise, which will force `a` *at the time when the containing promise is forced*, but at that time the name `a` points to a promise, which will force... -The fundamental issue is that `a = 2*a` is an imperative update. Therefore, to avoid this infinite loop trap for the unwary, assignments are not auto-lazified. Note that if we use two different names, this works just fine: +The fundamental issue is that `a = 2 * a` is an imperative update. Therefore, to avoid this infinite loop trap for the unwary, assignments are not auto-lazified. Note that if we use two *different* names, this works just fine: ```python from unpythonic.syntax import macros, lazy from unpythonic.syntax import force a = lazy[10] -b = lazy[2*force(a)] +b = lazy[2 * force(a)] print(force(b)) ``` -because now at the time when `b` is forced, the name `a` still points to the value we intended it to. +because now at the time when `b` is forced, the name `a` still points to the value we intended it to. That is, code that is normalized to [static single assignment (SSA) form](https://en.wikipedia.org/wiki/Static_single_assignment_form) could be auto-lazified. -If you're sure you have *new definitions* and not *imperative updates*, just manually use `lazy[]` (or `lazyrec[]`, as appropriate) on the RHS. Or if it's fine to use eager evaluation, just omit the `lazy[]`, thus allowing Python to evaluate the RHS immediately. +If you are sure you have *new definitions* and not *imperative updates*, you can just manually use `lazy[]` (or `lazyrec[]`, as appropriate) on the RHS. Or if it is fine to use eager evaluation, just omit the `lazy[]`, thus allowing Python to evaluate the RHS immediately. Beside function calls (which bind the parameters of the callee to the argument values of the call) and assignments, there are many other binding constructs in Python. For a full list, see [here](http://excess.org/article/2014/04/bar-foo/), or locally [here](../unpythonic/syntax/scopeanalyzer.py), in function `get_names_in_store_context`. Particularly noteworthy in the context of lazification are the `for` loop and the `with` context manager. In Python's `for`, the loop counter is an imperatively updated single name. In many use cases a rapid update is desirable for performance reasons, and in any case, the whole point of the loop is (almost always) to read the counter (and do something with the value) at least once per iteration. So it is much simpler, faster, and equally correct not to lazify there. -In `with`, the whole point of a context manager is that it is eagerly initialized when the `with` block is entered (and finalized when the block exits). Since our lazy code can transparently use both bare values and promises (due to the semantics of our `force1`), and the context manager would have to be eagerly initialized anyway, we can choose not to lazify there. +In `with`, the whole point of a context manager is that it is eagerly initialized when the `with` block is entered, and finalized when the block exits. Since our lazy code can transparently use both bare values and promises (due to the semantics of our `force1`), and the context manager would have to be eagerly initialized anyway, we have chosen not to lazify there. #### Note about TCO To borrow a term from PG's On Lisp, to make `lazify` *pay-as-you-go*, a special mode in `unpythonic.tco.trampolined` is automatically enabled by `with lazify` to build lazify-aware trampolines in order to avoid a drastic performance hit (~10x) in trampolines built for regular strict code. -The idea is that the mode is enabled while any function definitions in the `with lazify` block run, so they get a lazify-aware trampoline when the `trampolined` decorator is applied. This should be determined lexically, but that's complicated to do API-wise, so we currently enable the mode for the dynamic extent of the `with lazify`. Usually this is close enough; the main case where this can behave unexpectedly is: +The idea is that the mode is enabled while any function definitions in the `with lazify` block run, so they get a lazify-aware trampoline when the `trampolined` decorator is applied. This should be determined lexically, but that is complicated to do, because the decorator is applied at run time; so we currently enable the mode for the dynamic extent of the `with lazify`. Usually this is close enough. The main case where this can behave unexpectedly is: ```python @trampolined # strict trampoline @@ -1093,13 +1096,13 @@ with lazify: f2 = make_f() # f2 gets the lazify-aware trampoline ``` -TCO chains with an arbitrary mix of lazy and strict functions should work as long as the first function in the chain has a lazify-aware trampoline, because the chain runs under the trampoline of the first function (the trampolines of any tail-called functions are stripped away by the TCO machinery). +TCO chains with an arbitrary mix of lazy and strict functions should work as long as the first function in the chain has a lazify-aware trampoline, because the chain runs under the trampoline of the first function. The trampolines of any tail-called functions are skipped by the TCO machinery. Tail-calling from a strict function into a lazy function should work, because all arguments are evaluated at the strict side before the call is made. But tail-calling `strict -> lazy -> strict` will fail in some cases. The second strict callee may get promises instead of values, because the strict trampoline does not have the `maybe_force_args` (the mechanism `with lazify` uses to force the args when lazy code calls into strict code). -The reason we have this hack is that it allows the performance of strict code using unpythonic's TCO machinery, not even caring that a `lazify` exists, to be unaffected by the additional machinery used to support automatic lazy-strict interaction. +The reason we have this hack is that it allows the performance of strict code using `unpythonic`'s TCO machinery, not even caring that a `lazify` exists, to be unaffected by the additional machinery used to support automatic lazy-strict interaction. ### `tco`: automatic tail call optimization for Python From 4db7442f698a5ec0e96c808eaf903e74bd98d582 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 01:03:37 +0300 Subject: [PATCH 586/832] summarize Gabriel and Pitman 2001 --- doc/readings.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/readings.md b/doc/readings.md index 1d6b9ada..9be0e931 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -192,7 +192,10 @@ The common denominator is programming. Some relate to language design, some to c - [Example of Wat in Manuel Simoni's blog (2013)](http://axisofeval.blogspot.com/2013/05/green-threads-in-browser-in-20-lines-of.html) - [Richard P. Gabriel, Kent M. Pitman (2001): Technical Issues of Separation in Function Cells and Value Cells](https://dreamsongs.com/Separation.html) - - A discussion of [Lisp-1 vs. Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2). + - A discussion of [Lisp-1 vs. Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2), particularly of historical interest. + - Summary: Lisp-1 often leads to more readable code than Lisp-2, but by the time this became clear, for Common Lisp that train had already sailed. The authors suggest that instead of fixing CL with a backward compatibility breaking change, future Lisps would do well to take lessons learned from both Scheme and Common Lisp. In my own opinion, [Racket](https://racket-lang.org/) indeed has. + - Interestingly, there are more namespaces in Lisps than just values and functions, so, as the authors note, the popular names "Lisp-1" and "Lisp-2" are actually misnomers. For example, the labels for the Common Lisp construct `TAGBODY`/`GO` live in their own namespace. + - If explained using Python terminology, a Common Lisp symbol instance essentially has one attribute for each namespace, that stores the value bound to that symbol in that namespace. - [`hoon`: The C of Functional Programming](https://urbit.org/docs/hoon/) - Interesting take on an alternative computing universe where the functional camp won systems programming. These people have built [a whole operating system](https://github.com/urbit/urbit) on a Turing-complete non-lambda automaton, Nock. From dbedaf8acab7ab81431fb771fe8e47faa87a1907 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 01:20:03 +0300 Subject: [PATCH 587/832] 0.15.0: some final wording changes --- doc/macros.md | 66 +++++++++++++++++++++++++++------------------------ 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 5d658763..33483a0e 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -609,9 +609,9 @@ Macros that run multiple expressions, in sequence, in place of one expression. ### `do` as a macro: stuff imperative code into an expression, *with style* -We provide an `expr` macro wrapper for `unpythonic.seq.do` and `unpythonic.seq.do0`, with some extra features. +We provide an `expr` macro wrapper for `unpythonic.do` and `unpythonic.do0`, with some extra features. -This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see [the functions in `unpythonic.fploop`](../unpythonic/fploop.py) (`looped` and `looped_over`). +This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see the functions in the module [`unpythonic.fploop`](../unpythonic/fploop.py) (`looped` and `looped_over`). ```python from unpythonic.syntax import macros, do, local, delete @@ -651,7 +651,7 @@ Already declared local variables are updated with `var << value`. Updating varia >Assignments are recognized anywhere inside the `do`; but note that any `let` constructs nested *inside* the `do`, that define variables of the same name, will (inside the `let`) shadow those of the `do` - as expected of lexical scoping. > ->The boilerplate needed by the underlying `unpythonic.seq.do` form (notably the `lambda e: ...` wrappers) is inserted automatically. The expressions in a `do[]` are only evaluated when the underlying `unpythonic.seq.do` actually runs. +>The boilerplate needed by the underlying `unpythonic.do` form (notably the `lambda e: ...` wrappers) is inserted automatically. The expressions in a `do[]` are only evaluated when the underlying `unpythonic.do` actually runs. > >When running, `do` behaves like `letseq`; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites. @@ -742,7 +742,7 @@ Lexically inside a `with namedlambda` block, any literal `lambda` that is assign Decorated lambdas are also supported, as is a `curry` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/tests/test_lambdatools.py) for detailed examples. -The naming is performed using the function `unpythonic.misc.namelambda`, which will return a modified copy with its `__name__`, `__qualname__` and `__code__.co_name` changed. The original function object is not mutated. +The naming is performed using the function `unpythonic.namelambda`, which will return a modified copy with its `__name__`, `__qualname__` and `__code__.co_name` changed. The original function object is not mutated. **Supported assignment forms**: @@ -781,7 +781,7 @@ We have named the construct `fn`, because `f` is often used as a function name i The underscore `_` itself is not a macro. The `fn` macro treats the underscore magically, just like MacroPy's `f`, but anywhere else the underscore is available to be used as a regular variable. -The underscore does not need to be imported for `fn[]` to recognize it, but if you want to make your IDE happy, there is a symbol named `_` in `unpythonic.syntax` you can import to silence any "undefined name" errors regarding the use of `_`. It is a regular run-time object, not a macro. +The underscore does not need to be imported for `fn[]` to recognize it, but if you want to make your IDE happy, there is a symbol named `_` in `unpythonic.syntax` you can import to silence any "undefined name" errors regarding the use of `_`. It is a regular run-time object, not a macro. It is available in `unpythonic.syntax` (not at the top level of `unpythonic`) because it is basically an auxiliary syntactic construct, with no meaningful run-time functionality of its own. (It *could* be made into a `@namemacro` that triggers a syntax error when it appears in an improper context, like starting with v0.15.0, many auxiliary constructs in similar roles already do. But it was decided that in this particular case, it is more valuable to have the name `_` available for other uses in other contexts, because it is a standard dummy name in Python. The lambdas created using `fn[]` are likely short enough that not automatically detecting misplaced underscores does not cause problems in practice.) @@ -918,7 +918,7 @@ assert add3(1)(2)(3) == 6 - All **function calls** and **function definitions** (`def`, `lambda`) are automatically curried, somewhat like in Haskell, or in `#lang` [`spicy`](https://github.com/Technologicat/spicy). - - Function calls are autocurried, and run `unpythonic.fun.curry` in a special mode that no-ops on uninspectable functions (triggering a standard function call with the given args immediately) instead of raising `TypeError` as usual. + - Function calls are autocurried, and run `unpythonic.curry` in a special mode that no-ops on uninspectable functions (triggering a standard function call with the given args immediately) instead of raising `TypeError` as usual. **CAUTION**: Some built-ins are uninspectable or may report their call signature incorrectly; in those cases, `curry` may fail, occasionally in mysterious ways. When inspection fails, `curry` raises ``ValueError``, like `inspect.signature` does. @@ -982,13 +982,13 @@ Like `with continuations`, no state or context is associated with a `with lazify Lazy code is allowed to call strict functions and vice versa, without requiring any additional effort. -Comboing with other block macros in `unpythonic.syntax` is supported, including `autocurry` and `continuations`. See the [meta](#meta) section of this README for the correct ordering. +Comboing `lazify` with other block macros in `unpythonic.syntax` is supported, including `autocurry` and `continuations`. See the [meta](#meta) section of this README for the correct ordering. For more details, see the docstring of `unpythonic.syntax.lazify`. Inspired by Haskell, Racket's `(delay)` and `(force)`, and [lazy/racket](https://docs.racket-lang.org/lazy/index.html). -**CAUTION**: The functions in `unpythonic.fun` are lazify-aware (so that e.g. `curry` and `compose` work with lazy functions), as are `call` and `callwith` in `unpythonic.misc`, but a large part of `unpythonic` is not. Keep in mind that any call to a strict (regular Python) function will evaluate all of its arguments. +**CAUTION**: The functions in the module `unpythonic.fun` are lazify-aware (so that e.g. `curry` and `compose` work with lazy functions), as are `call` and `callwith` in the module `unpythonic.funutil`, but a large part of `unpythonic` is not. Keep in mind that any call to a strict (regular Python) function will evaluate all of its arguments. #### `lazy[]` and `lazyrec[]` macros @@ -1039,7 +1039,7 @@ If we chose to auto-lazify assignments, then the example would expand to: ```python from unpythonic.syntax import macros, lazy -from unpythonic.syntax import force +from unpythonic import force a = lazy[10] a = lazy[2 * force(a)] @@ -1052,7 +1052,7 @@ The fundamental issue is that `a = 2 * a` is an imperative update. Therefore, to ```python from unpythonic.syntax import macros, lazy -from unpythonic.syntax import force +from unpythonic import force a = lazy[10] b = lazy[2 * force(a)] @@ -1071,11 +1071,14 @@ In `with`, the whole point of a context manager is that it is eagerly initialize #### Note about TCO -To borrow a term from PG's On Lisp, to make `lazify` *pay-as-you-go*, a special mode in `unpythonic.tco.trampolined` is automatically enabled by `with lazify` to build lazify-aware trampolines in order to avoid a drastic performance hit (~10x) in trampolines built for regular strict code. +To borrow a term from PG's On Lisp, to make `lazify` *pay-as-you-go*, a special mode in `unpythonic.trampolined` is automatically enabled by `with lazify` to build lazify-aware trampolines in order to avoid a drastic performance hit (~10x) in trampolines built for regular strict code. The idea is that the mode is enabled while any function definitions in the `with lazify` block run, so they get a lazify-aware trampoline when the `trampolined` decorator is applied. This should be determined lexically, but that is complicated to do, because the decorator is applied at run time; so we currently enable the mode for the dynamic extent of the `with lazify`. Usually this is close enough. The main case where this can behave unexpectedly is: ```python +from unpythonic.syntax import macros, lazify +from unpythonic import trampolined + @trampolined # strict trampoline def g(): ... @@ -1333,7 +1336,7 @@ To keep things relatively straightforward, our `call_cc[]` is only allowed to ap Nested defs are ok; here *top level* only means the top level of the *currently innermost* `def`. -If you need to place `call_cc[]` inside a loop, use `@looped` et al. from `unpythonic.fploop`; this has the loop body represented as the top level of a `def`. +If you need to place `call_cc[]` inside a loop, use `@looped` et al. from the module `unpythonic.fploop`; this has the loop body represented as the top level of a `def`. Multiple `call_cc[]` statements in the same function body are allowed. These essentially create nested closures. @@ -1359,7 +1362,7 @@ call_cc[f(...) if p else g(...)] *NOTE*: `*xs` may need to be written as `*xs,` in order to explicitly make the LHS into a tuple. The variant without the comma seems to work when run from a `.py` file with the `macropython` bootstrapper from [`mcpyrate`](https://pypi.org/project/mcpyrate/), but fails in code run interactively in the `mcpyrate` REPL. -*NOTE*: `f()` and `g()` must be **literal function calls**. Sneaky trickery (such as calling indirectly via `unpythonic.funutil.call` or `unpythonic.fun.curry`) is not supported. (The `prefix` and `curry` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the `call_cc[]` needs to patch the `cc=...` kwarg of the call being made. +*NOTE*: `f()` and `g()` must be **literal function calls**. Sneaky trickery (such as calling indirectly via `unpythonic.call` or `unpythonic.curry`) is not supported. (The `prefix` and `curry` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the `call_cc[]` needs to patch the `cc=...` kwarg of the call being made. **Assignment targets**: @@ -1530,7 +1533,7 @@ However, as the only exception to this rule, if the continuation is meant to act (Note also that a continuation that has no `cc` parameter cannot be used as the target of an explicit tail-call in the client code, since a tail-call in a `with continuations` block will attempt to supply a `cc` argument to the function being tail-called. Likewise, it cannot be used as the target of a `call_cc[]`, since this will also attempt to supply a `cc` argument.) -These observations make `unpythonic.fun.identity` eligible as a continuation, even though it is defined elsewhere in the library and it has no `cc` parameter. +These observations make `unpythonic.identity` eligible as a continuation, even though it is defined elsewhere in the library and it has no `cc` parameter. #### This isn't `call/cc`! @@ -1610,7 +1613,7 @@ with prefix: # in case of duplicate name across kws, rightmost wins assert (f, kw(a="hi there"), kw(b="Tom"), kw(b="Jerry")) == (q, "hi there", "Jerry") - # give *args with unpythonic.fun.apply, like in Lisps: + # give *args with unpythonic.apply, like in Lisps: lst = [1, 2, 3] def g(*args): return args @@ -1691,10 +1694,11 @@ If you wish to omit `return` in tail calls, this comboes with `tco`; just apply ### `forall`: nondeterministic evaluation -Behaves the same as the multiple-body-expression tuple comprehension `unpythonic.amb.forall`, but implemented purely by AST transformation, with real lexical variables. This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see `unpythonic.syntax.forall`). +Behaves the same as the multiple-body-expression tuple comprehension `unpythonic.forall`, but implemented purely by AST transformation, with real lexical variables. This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see `unpythonic.syntax.forall`). ```python -from unpythonic.syntax import macros, forall, insist, deny +from unpythonic.syntax import macros, forall +from unpythonic.syntax import insist, deny # regular functions, not macros out = forall[y << range(3), x << range(3), @@ -1712,9 +1716,9 @@ assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), (8, 15, 17), (9, 12, 15), (12, 16, 20)) ``` -Assignment (with List-monadic magic) is `var << iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). +Assignment (**with** List-monadic magic) is `var << iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). -`insist` and `deny` are not really macros; they are just the functions from `unpythonic.amb`, re-exported for convenience. +`insist` and `deny` are not macros; they are just the functions from `unpythonic.amb`, re-exported for convenience. The error raised by an undefined name in a `forall` section is `NameError`. @@ -1867,7 +1871,7 @@ with session("simple framework demo"): By default, running this script through the `macropython` wrapper (from `mcpyrate`) will produce an ANSI-colored test report in the terminal. To actually see how the output looks like, for actual runnable examples, see `unpythonic`'s own automated tests. -If you want to turn coloring off (e.g. for redirecting stderr to a file), see the `TestConfig` bunch of constants in `unpythonic.test.fixtures`. +If you want to turn coloring off (e.g. for the purposes of redirecting stderr to a file), see the `TestConfig` bunch of constants in `unpythonic.test.fixtures`. The following is an overview of the framework. For details, look at the docstrings of the various constructs in `unpythonic.test.fixtures` (which provides much of this), those of the test macros, and finally, the automated tests of `unpythonic` itself. @@ -1877,7 +1881,7 @@ How to test macro utilities (e.g. syntax transformer functions that operate on A #### Overview -We provide the low-level syntactic constructs `test[]`, `test_raises[]` and `test_signals[]`, with the usual meanings. The last one is for testing code that uses the `signal` function and its sisters (related to conditions and restarts à la Common Lisp); see [`unpythonic.conditions`](features.md#handlers-restarts-conditions-and-restarts). +We provide the low-level syntactic constructs `test[]`, `test_raises[]` and `test_signals[]`, with the usual meanings. The last one is for testing code that uses the `signal` function and its sisters (related to conditions and restarts à la Common Lisp); see the module [`unpythonic.conditions`](../unpythonic/conditions.py), and the user manual section on [conditions and restarts](features.md#handlers-restarts-conditions-and-restarts). By default, the `test[expr]` macro asserts that the value of `expr` is truthy. If you want to assert only that `expr` runs to completion normally, use `test[returns_normally(expr)]`. @@ -1953,9 +1957,9 @@ Additional tools for code using **conditions and restarts**: The `catch_signals` context manager controls the signal barrier of `with testset` and the `test` family of syntactic constructs. It is provided for writing tests for code that uses conditions and restarts. -Used as `with catch_signals(False)`, it disables the signal barrier. Within the dynamic extent of the block, an uncaught signal (in the sense of `unpythonic.conditions.signal` and its sisters) is not considered an error. This can be useful, because sometimes leaving a signal uncaught is the right thing to do. See [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py) for examples. +Used as `with catch_signals(False)`, it disables the signal barrier for the dynamic extent of the block. When the barrier is disabled, an uncaught signal (in the sense of `unpythonic.signal` and its sisters) is not considered as an errored test. This can be useful, because sometimes leaving a signal uncaught is the right thing to do. See [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py) for examples. -It can be nested. Used as `with catch_signals(True)`, it re-enables the barrier, if currently disabled. +The `with catch_signals` construct can be nested. Used as `with catch_signals(True)`, it re-enables the barrier, if currently disabled, for the dynamic extent of that inner `with catch_signals` block. When a `with catch_signals` block exits, the previous state of the signal barrier is automatically restored. @@ -1981,7 +1985,7 @@ The constructs `test_raises`, `test_signals`, `fail`, `error` and `warn` do **no Tests can be nested; this is sometimes useful as an explicit signal barrier. -Note the macros `error[]` and `warn[]` have nothing to do with the functions with the same name in `unpythonic.conditions`. The macros are part of the test framework; the functions with the same name are signaling protocols of the conditions and restarts system. Following the usual naming conventions in both systems, this naming conflict is unfortunately what we get. +Note the macros `error[]` and `warn[]` have nothing to do with the functions with the same name in the module `unpythonic.conditions`. The macros are part of the test framework; the functions with the same name are signaling protocols of the conditions and restarts system. Following the usual naming conventions in both systems, this naming conflict is unfortunately what we get. **Block** forms: @@ -2085,7 +2089,7 @@ To make testing/debugging macro code more convenient, the `the[]` mechanism auto **CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See `mcpyrate.unparse`. -**CAUTION**: The name of the `the[]` construct was inspired by Common Lisp, but the semantics are completely different. Common Lisp's `THE` is a return-type declaration (pythonistas would say *return-type annotation*), meant as a hint for the compiler to produce performance-optimized compiled code (see [chapter 32 of Peter Seibel's Practical Common Lisp](http://www.gigamonkeys.com/book/conclusion-whats-next.html)), whereas our `the[]` captures a value for test reporting. The only common factors are the name, and that neither construct changes the semantics of the marked code, much. In `unpythonic.test.fixtures`, the reason behind picking this name was that it doesn't change the flow of the source code as English that much, specifically to suggest, between the lines, that it doesn't change the semantics much. The reasoning behind CL's `THE` may be similar. +**CAUTION**: The name of the `the[]` construct was inspired by Common Lisp, but the semantics are completely different. Common Lisp's `THE` is a return-type declaration (pythonistas would say *return-type annotation*), meant as a hint for the compiler to produce performance-optimized compiled code (see [chapter 32 of Peter Seibel's Practical Common Lisp](http://www.gigamonkeys.com/book/conclusion-whats-next.html)), whereas our `the[]` captures a value for test reporting. The only common factors are the name, and that neither construct changes the semantics of the marked code, much. In `unpythonic.test.fixtures`, the reason behind picking this name was that it doesn't change the flow of the source code as English that much, specifically to suggest, between the lines, that it doesn't change the semantics much. The reasoning behind CL's `THE` may be similar, but I have not researched its etymology. #### Test sessions and testsets @@ -2105,7 +2109,7 @@ In case of an uncaught signal, the error is reported, and the testset resumes. In case of an uncaught exception, the error is reported, and the testset terminates, because the exception model does not support resuming. -Catching of uncaught *signals*, in both the low-level `test` constructs and the high-level `testset`, can be disabled using `with catch_signals(False)`. This is useful in testing code that uses conditions and restarts; sometimes allowing a signal (e.g. from `unpythonic.conditions.warn`) to remain uncaught is the right thing to do. +Catching of uncaught *signals*, in both the low-level `test` constructs and the high-level `testset`, can be disabled using `with catch_signals(False)`. This is useful in testing code that uses conditions and restarts; sometimes allowing a signal (e.g. from `unpythonic.warn` in the conditions-and-restarts system) to remain uncaught is the right thing to do. #### Producing unconditional failures, errors, and warnings @@ -2115,15 +2119,15 @@ The helper macros `fail[message]`, `error[message]` and `warn[message]` uncondit - `error[...]` if some part of your tests is unable to run. - `warn[...]` if some tests are temporarily disabled and need future attention, e.g. for syntactic compatibility to make the code run for now on an old Python version. -Currently (v0.14.3), warnings produced by `warn[]` are not counted in the total number of tests run. But you can still get the warning count from the separate counter `unpythonic.test.fixtures.tests_warned` (see `unpythonic.collections.box`; basically you can `b.get()` or `unbox(b)` to read the value currently inside a box). +Currently (v0.14.3), warnings produced by `warn[]` are not counted in the total number of tests run. But you can still get the warning count from the separate counter `unpythonic.test.fixtures.tests_warned` (see `unpythonic.box`; basically you can `b.get()` or `unbox(b)` to read the value currently inside a box). #### Advanced: building a custom test framework -If `unpythonic.test.fixtures` does not fit your needs and you want to experiment with creating your own framework, the test asserter macros are reusable. For reference, their implementations can be found in `unpythonic.syntax.testingtools`. They refer to a few objects in `unpythonic.test.fixtures`; consider these a common ground that is not strictly part of the surrounding framework. +If `unpythonic.test.fixtures` does not fit your needs and you want to experiment with creating your own framework, the test asserter macros are reusable. Their implementations can be found in `unpythonic.syntax.testingtools`. They refer to a few objects in `unpythonic.test.fixtures`; consider these a common ground that is not strictly part of the surrounding framework. Start by reading the docstring of the `test` macro, which documents some low-level details. -Set up a condition handler to intercept test failures and errors. These will be signaled via `cerror`, using the conditions and restarts mechanism. See `unpythonic.conditions`. Report the failure/error in any way you desire, and then invoke the `proceed` restart (from your condition handler) to let testing continue. +Set up a condition handler to intercept test failures and errors. These will be signaled via `cerror`, using the conditions and restarts mechanism. See the module `unpythonic.conditions`. Report the failure/error in any way you desire, and then invoke the `proceed` restart (from your condition handler) to let testing continue. Look at the implementation of `testset` as an example. @@ -2149,9 +2153,9 @@ What we have is small, simple, custom-built for its purpose (works well with mac #### Etymology and roots -[Test fixture](https://en.wikipedia.org/wiki/Test_fixture) *is an environment used to consistently test some item, device, or piece of software*. In automated tests, it is typically a piece of code that is reused within the test suite of a project, to perform initialization and/or teardown tasks common to several test cases. +A [test fixture](https://en.wikipedia.org/wiki/Test_fixture) is defined as *an environment used to consistently test some item, device, or piece of software*. In automated tests, it is typically a piece of code that is reused within the test suite of a project, to perform initialization and/or teardown tasks common to several test cases. -A test framework can be reused across many different projects, and the error-catching and reporting code, if anything, is something that is shared across all test cases. Also, following our naming scheme, it had to be called `unpythonic.test.something`, and `fixtures` just happened to fit the theme. +A test framework can be reused across many different projects, and the error-catching and reporting code, if anything, is something that is shared across all test cases. Also, following our naming scheme, the framework had to be called `unpythonic.test.something`, and `fixtures` just happened to fit the theme. Inspired by [Julia](https://julialang.org/)'s standard-library [`Test` package](https://docs.julialang.org/en/v1/stdlib/Test/), and [chapter 9 of Peter Seibel's Practical Common Lisp](http://www.gigamonkeys.com/book/practical-building-a-unit-test-framework.html). From c296c40052ea0c67095bcc3db6a4f8dae3b7bc07 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 01:35:19 +0300 Subject: [PATCH 588/832] 0.15.0: update tco macro docs --- doc/macros.md | 34 +++++++++++++++++++++++++--------- unpythonic/syntax/tailtools.py | 2 +- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 33483a0e..0bc2d4c4 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1130,37 +1130,53 @@ with tco: assert evenp(10000) is True ``` -All function definitions (`def` and `lambda`) lexically inside the block undergo TCO transformation. The functions are automatically `@trampolined`, and any tail calls in their return values are converted to `jump(...)` for the TCO machinery. Here *return value* is defined as: +All function definitions (`def` and `lambda`) lexically inside the `with tco` block undergo TCO transformation. The functions are automatically `@trampolined`, and any tail calls in their return values are converted to `jump(...)` for the TCO machinery. Here *return value* is defined as: - In a `def`, the argument expression of `return`, or of a call to a known escape continuation. - In a `lambda`, the whole body, as well as the argument expression of a call to a known escape continuation. -What is a *known escape continuation* is explained below, in the section [TCO and `call_ec`](#tco-and-call_ec). +What is considered a *known escape continuation* is explained below, in the section [TCO and `call_ec`](#tco-and-call_ec). -To find the tail position inside a compound return value, this recursively handles any combination of `a if p else b`, `and`, `or`; and from `unpythonic.syntax`, `do[]`, `let[]`, `letseq[]`, `letrec[]`. Support for `do[]` includes also any `multilambda` blocks that have already expanded when `tco` is processed. The macros `aif[]` and `cond[]` are also supported, because they expand into a combination of `let[]`, `do[]`, and `a if p else b`. +To find the tail position inside a compound return value, we recursively handle any combination of `a if p else b`, `and`, `or`; and from `unpythonic.syntax`, `do[]`, `let[]`, `letseq[]`, `letrec[]`. Support for `do[]` includes also any `multilambda` blocks that have already expanded when `tco` is processed. The macros `aif[]` and `cond[]` are also supported, because they expand into a combination of `let[]`, `do[]`, and `a if p else b`. **CAUTION**: In an `and`/`or` expression, only the last item of the whole expression is in tail position. This is because in general, it is impossible to know beforehand how many of the items will be evaluated. -**CAUTION**: In a `def` you still need the `return`; it marks a return value. If you want the tail position to imply a `return`, use the combo `with autoreturn, tco` (on `autoreturn`, see below). +**CAUTION**: In a `def` you still need the `return`; it marks a return value. If you want tail position to imply a `return`, use the combo `with autoreturn, tco` (on `autoreturn`, see below). TCO is based on a strategy similar to MacroPy's `tco` macro, but using unpythonic's TCO machinery, and working together with the macros introduced by `unpythonic.syntax`. The semantics are slightly different; by design, `unpythonic` requires an explicit `return` to mark tail calls in a `def`. A call that is strictly speaking in tail position, but lacks the `return`, is not TCO'd, and Python's implicit `return None` then shuts down the trampoline, returning `None` as the result of the TCO chain. #### TCO and continuations -The `tco` macro detects and skips any `with continuations` blocks inside the `with tco` block, because `continuations` already implies TCO. This is done **for the specific reason** of allowing the [Lispython dialect](https://github.com/Technologicat/pydialect) to use `with continuations`, because the dialect itself implies a `with tco` for the whole module (so the user code has no way to exit the TCO context). +The `tco` macro detects and skips any `with continuations` blocks inside the `with tco` block, because `continuations` already implies TCO. This is done **for the specific reason** of allowing the [Lispython dialect](https://github.com/Technologicat/pydialect) to use `with continuations`, because the dialect itself implies a `with tco` for the whole module. Hence, in that dialect, the user code has no way to exit the TCO context. -The `tco` and `continuations` macros actually share a lot of the code that implements TCO; `continuations` just hooks into some callbacks to perform additional processing. +The `tco` and `continuations` macros actually share a lot of the code that implements TCO; `continuations`, for its TCO processing, just hooks into some callbacks to make additional AST edits. #### TCO and `call_ec` -(Mainly of interest for lambdas, which have no `return`, and for "multi-return" from a nested function.) +This is mainly of interest for lambdas, which have no `return`, and for "multi-return" from a nested function. It is important to recognize a call to an escape continuation as such, because the argument given to an escape continuation is essentially a return value. If this argument is itself a call, it needs the TCO transformation to be applied to it. -For escape continuations in `tco` and `continuations` blocks, only basic uses of `call_ec` are supported, for automatically harvesting names referring to an escape continuation. In addition, the literal function names `ec`, `brk` and `throw` are always *understood as referring to* an escape continuation. +For escape continuations in `tco` and `continuations` blocks, only basic uses of `call_ec` are supported, for automatically extracting names referring to an escape continuation. *Basic use* is defined as either of these two cases: -The name `ec`, `brk` or `throw` alone is not sufficient to make a function into an escape continuation, even though `tco` (and `continuations`) will think of it as such. The function also needs to actually implement some kind of an escape mechanism. An easy way to get an escape continuation, where this has already been done for you, is to use `call_ec`. Another such mechanism is the `catch`/`throw` pair. +```python +from unpythonic import call_ec + +# use as decorator +@call_ec +def result(ec): + ... + +# use directly on a literal lambda (effectively, as a decorator) +result = call_ec(lambda ec: ...) +``` + +When macro expansion of the ``with tco`` block starts, names of escape continuations created **anywhere lexically within** the ``with tco`` block are captured, provided that the creation takes place using one of the above *basic use* patterns. + +In addition, the literal function names `ec`, `brk` and `throw` are always *understood as referring to* an escape continuation. The name `ec` is the customary name for the parameter of a function passed to `call_ec`. The name `brk` is the customary name for the break continuation created by `@breakably_looped` and `@breakably_looped_over`. The name `throw` is understood as referring to the function `unpythonic.throw`. + +Obviously, having a name of `ec`, `brk` or `throw` is not by itself sufficient to make a function into an escape continuation, even though `tco` (and `continuations`) will think of it as such. The function also needs to actually implement some kind of an escape mechanism. An easy way to get an escape continuation, where this has already been done for you, is to use `call_ec`. Another such mechanism is the `catch`/`throw` pair. See the docstring of `unpythonic.syntax.tco` for details. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 54f9eab7..d28a59ca 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -207,7 +207,7 @@ def oddp(x): def result(ec): ... - # use directly on a literal lambda + # use directly on a literal lambda (effectively, as a decorator) result = call_ec(lambda ec: ...) When macro expansion of the ``with tco`` block starts, names of escape From df3a72467b0bfab16b05676f55a863493b640ab8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 01:40:52 +0300 Subject: [PATCH 589/832] small addition to tco macro doc --- doc/macros.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/doc/macros.md b/doc/macros.md index 0bc2d4c4..ba1adbc5 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1110,9 +1110,21 @@ The reason we have this hack is that it allows the performance of strict code us ### `tco`: automatic tail call optimization for Python +*This is the macro that applies tail call optimization (TCO) automatically. See the manual section on [`trampolined` and `jump`](features.md#trampolined-jump-tail-call-optimization-tco--explicit-continuations) on what TCO is and where it is useful.* + +Using `with tco`, there is no need to manually use `trampolined` or `jump`: + ```python from unpythonic.syntax import macros, tco +with tco: + def fact(n, acc=1): + if n == 0: + return acc + return fact(n - 1, n * acc) + print(fact(4)) # 24 + fact(5000) # no crash + with tco: evenp = lambda x: (x == 0) or oddp(x - 1) oddp = lambda x: (x != 0) and evenp(x - 1) From 2a32271896882730c7e7424825132d9a61a1d2d0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 03:00:55 +0300 Subject: [PATCH 590/832] 0.15.0: update continuations macro doc --- doc/macros.md | 160 ++++++++++++++++++++++++++++---------------------- 1 file changed, 91 insertions(+), 69 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index ba1adbc5..3a4f4bc6 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1197,46 +1197,52 @@ See the docstring of `unpythonic.syntax.tco` for details. *Where control flow is your playground.* -We provide **genuine multi-shot continuations for Python**. Compare generators and coroutines, which are resumable functions, or in other words, single-shot continuations. In single-shot continuations, once execution passes a certain point, it cannot be rewound. Multi-shot continuations [can be emulated](https://gist.github.com/yelouafi/858095244b62c36ec7ebb84d5f3e5b02), but this makes the execution time `O(n**2)`, because when we want to restart again at an already passed point, the execution must start from the beginning, replaying the history. In contrast, **we implement continuations that can natively resume execution multiple times from the same point.** +We provide **genuine multi-shot continuations for Python**. Compare generators and coroutines, which are resumable functions, or in other words, single-shot continuations. In single-shot continuations, once execution passes a certain point, it cannot be rewound. Multi-shot continuations [can be emulated](https://gist.github.com/yelouafi/858095244b62c36ec7ebb84d5f3e5b02) using single-shot continuations, but this makes the execution time `O(n**2)`, because when we want to restart again at an already passed point, the execution must start from the beginning, replaying the whole history. In contrast, **we implement continuations that can natively resume execution multiple times from the same point.** -This feature has some limitations and is mainly intended for experimenting with, and teaching, multi-shot continuations in a Python setting. +**CAUTION**: This feature has some limitations, and is mainly intended for experimenting with, and teaching, multi-shot continuations in a Python setting. Particularly: -- There are seams between continuation-enabled code and regular Python code. (This happens with any feature that changes the semantics of only a part of a program.) + - There are seams between continuation-enabled code and regular Python code. (This happens with any feature that changes the semantics of only a part of a program.) -- There is no [`dynamic-wind`](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28quote._~23~25kernel%29._dynamic-wind%29%29) (the generalization of `try/finally`, when control can jump back in to the block from outside it). + - There is no [`dynamic-wind`](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28quote._~23~25kernel%29._dynamic-wind%29%29): Scheme's generalization of `try/finally`, which beside the `finally` exit hook, has an *entry hook* for when control jumps back into the block from outside it. -- Interaction of continuations with exceptions is not fully thought out. + - Interaction of continuations with exceptions is not fully thought out. -- Interaction with async functions **is not even implemented**. An `async def` or `await` appearing inside a `with continuations` block is considered a syntax error. + - Interaction with async functions **is not even implemented**. For this reason, an `async def` or `await` appearing inside a `with continuations` block is considered a syntax error. -- The implicit `cc` parameter might not be a good idea in the long run. - - This design might or might not change in a future release. It suffers from the same lack of transparency, whence the same potential for bugs, as the implicit `this` in many languages (e.g. C++ and JavaScript). - - Because `cc` is *declared* implicitly, it is easy to forget that *every* function definition anywhere inside the `with continuations` block introduces its own `cc` parameter. - - This introduces a bug when one introduces an inner function, and attempts to use the outer `cc` inside the inner function body, forgetting that inside the inner function, the name `cc` points to **the inner function's** own `cc`. - - The correct pattern is to `outercc = cc` in the outer function, and then use `outercc` inside the inner function body. - - Not introducing its own `this` [was precisely why](http://tc39wiki.calculist.org/es6/arrow-functions/) the arrow function syntax was introduced to JavaScript in ES6. - - Python gets `self` right in that while it is conveniently *passed* implicitly, it must be *declared* explicitly, eliminating the transparency issue. - - On the other hand, a semi-explicit `cc`, like Python's `self`, was tried in an early version of this continuations subsystem, and it led to a lot of boilerplate. It is especially bad that to avoid easily avoidable bugs regarding passing in the wrong arguments, `cc` effectively must be a keyword parameter, necessitating the user to write `def f(x, *, cc)`. Not having to type out the `, *, cc` is much nicer, albeit not as pythonic. + - The implicit `cc` parameter might not be a good idea in the long run. + - This design suffers from the same lack of transparency, whence the same potential for bugs, as the implicit `this` in many languages (e.g. C++ and JavaScript). + - Because `cc` is *declared* implicitly, it is easy to forget that *every* function definition *anywhere* inside the `with continuations` block introduces its own `cc` parameter. + - Particularly, also a `lambda` is a function definition. + - This introduces a bug when one introduces an inner function, and attempts to use the outer `cc` inside the inner function body, forgetting that inside the inner function, the name `cc` points to **the inner function's** own `cc`. + - The correct pattern is to `outercc = cc` in the outer function, and then use `outercc` inside the inner function body. + - Not introducing its own `this` [was precisely why](http://tc39wiki.calculist.org/es6/arrow-functions/) the arrow function syntax was introduced to JavaScript in ES6. + - Python gets `self` right in that while it is conveniently *passed* implicitly, it must be *declared* explicitly, eliminating the transparency issue. + - On the other hand, a semi-explicit `cc`, like Python's `self`, was tried in an early version of this continuations subsystem, and it led to a lot of boilerplate. + - It is especially bad that to avoid easily avoidable bugs regarding passing in the wrong arguments, `cc` effectively must be a keyword parameter, necessitating the user to write `def f(x, *, cc)`. Not having to type out the `, *, cc` is much nicer, albeit not as pythonic. #### General remarks on continuations If you are new to continuations, see the [short and easy Python-based explanation](https://www.ps.uni-saarland.de/~duchier/python/continuations.html) of the basic idea. -We essentially provide a very loose pythonification of Paul Graham's continuation-passing macros, chapter 20 in [On Lisp](http://paulgraham.com/onlisp.html). +This continuations system in `unpythonic` began as a very loose pythonification of Paul Graham's continuation-passing macros, chapter 20 in [On Lisp](http://paulgraham.com/onlisp.html). The approach differs from native continuation support (such as in Scheme or Racket) in that the continuation is captured only where explicitly requested with `call_cc[]`. This lets most of the code work as usual, while performing the continuation magic where explicitly desired. -As a consequence of the approach, our continuations are [*delimited*](https://en.wikipedia.org/wiki/Delimited_continuation) in the very crude sense that the captured continuation ends at the end of the body where the *currently dynamically outermost* `call_cc[]` was used. Notably, in `unpythonic`, a continuation eventually terminates and returns a value, without hijacking the rest of the whole-program execution. +As a consequence of the approach, our continuations are [*delimited*](https://en.wikipedia.org/wiki/Delimited_continuation) in the very crude sense that the captured continuation ends at the end of the body where the *currently dynamically outermost* `call_cc[]` was invoked. Notably, in `unpythonic`, a continuation eventually terminates and returns a value (provided that the code contained in the continuation itself terminates), without hijacking the rest of the whole-program execution. -Hence, if porting some code that uses `call/cc` from Racket to Python, in the Python version the `call_cc[]` may be need to be placed further out to capture the relevant part of the computation. For example, see `amb` in the demonstration below; a Scheme or Racket equivalent usually has the `call/cc` placed inside the `amb` operator itself, whereas in Python we must place the `call_cc[]` at the call site of `amb`. +Hence, if porting some code that uses `call/cc` from Racket to Python, in the Python version the `call_cc[]` may be need to be placed further out to capture the relevant part of the computation. For example, see `amb` in the demonstration below; a Scheme or Racket equivalent usually has the `call/cc` placed inside the `amb` operator itself, whereas in Python we must place the `call_cc[]` at the call site of `amb`, so that the continuation captures the remainder of the call site. -Observe that while our outermost `call_cc` already somewhat acts like a prompt (in the sense of delimited continuations), we are currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and terminate the capture there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. +Observe that while our outermost `call_cc` already somewhat acts like a prompt (in the sense of delimited continuations), we are currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and make the continuation terminate there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. For various possible program topologies that continuations may introduce, see [these clarifying pictures](callcc_topology.pdf). For full documentation, see the docstring of `unpythonic.syntax.continuations`. The unit tests [[1]](../unpythonic/syntax/tests/test_conts.py) [[2]](../unpythonic/syntax/tests/test_conts_escape.py) [[3]](../unpythonic/syntax/tests/test_conts_gen.py) [[4]](../unpythonic/syntax/tests/test_conts_topo.py) may also be useful as usage examples. -**Note on debugging**: If a function containing a `call_cc[]` crashes below the `call_cc[]`, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so you can pinpoint the source code line where the error occurred. (For a function `f`, it is named `f_cont_`) But be aware that especially in complex macro combos (e.g. `continuations, curry, lazify`), the other block macros may spit out many internal function calls *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. +**Note on debugging**: If a function containing a `call_cc[]` crashes below a line that has a `call_cc[]` invocation, the stack trace will usually have the continuation function somewhere in it, containing the line number information, so as usual, you can pinpoint the source code line where the error occurred. For a function `f`, continuation definitions created by `call_cc[]` invocations within its body are named `f_cont_`. + +Be aware that especially in complex block macro combos (e.g. `with lazify, autocurry, continuations`), the other block macros may have spit out many internal function calls that, at run time, get called *after* the relevant stack frame that points to the actual user program. So check the stack trace as usual, but check further up than usual. + +Using the `with step_expansion` macro from `mcpyrate.debug` may help in understanding how the macro-expanded code actually looks like. **Note on exceptions**: Raising an exception, or [signaling and restarting](features.md#handlers-restarts-conditions-and-restarts), will partly unwind the call stack, so the continuation *from the level that raised the exception* will be cancelled. This is arguably exactly the expected behavior. @@ -1247,7 +1253,7 @@ from unpythonic.syntax import macros, continuations, call_cc with continuations: # basic example - how to call a continuation manually: - k = None # kontinuation + k = None # a kontinuation is konventionally kalled k def setk(*args, cc): global k k = cc @@ -1278,7 +1284,7 @@ with continuations: # Pythagorean triples def pt(): z = call_cc[amb(range(1, 21))] - y = call_cc[amb(range(1, z+1)))] + y = call_cc[amb(range(1, z+1))] x = call_cc[amb(range(1, y+1))] if x*x + y*y != z*z: return fail() @@ -1291,12 +1297,13 @@ with continuations: print(fail()) print(fail()) ``` + Code within a `with continuations` block is treated specially.

Roughly: > - Each function definition (`def` or `lambda`) in a `with continuations` block has an implicit formal parameter `cc`, **even if not explicitly declared** in the formal parameter list. -> - The continuation machinery will set the default value of `cc` to the default continuation (`identity`), which just returns its arguments. +> - The continuation machinery will set the default value of `cc` to the default continuation (`identity`), which just returns its argument(s). > - The default value allows these functions to be called also normally without passing a `cc`. In effect, the function will then return normally. > - If `cc` is not declared explicitly, it is implicitly declared as a by-name-only parameter named `cc`, and the default value is set automatically. > - If `cc` is declared explicitly, the default value is set automatically if `cc` is in a position that can accept a default value, and no default has been set by the user. @@ -1309,19 +1316,21 @@ Code within a `with continuations` block is treated specially. > > - In a function definition inside the `with continuations` block: > - Most of the language works as usual; especially, any non-tail function calls can be made as usual. -> - `return value` or `return v0, ..., vn` is actually a tail-call into `cc`, passing the given value(s) as arguments. -> - As in other parts of `unpythonic`, returning a `Values` means returning multiple-return-values. -> - This is important if the return value is received by the assignment targets of a `call_cc[]`. If you get a `TypeError` concerning the arguments of a function with a name ending in `_cont`, check your `call_cc[]` invocations and the `return` in the call_cc'd function. +> - `return value` or `return Values(...)` is actually a tail-call into `cc`, passing the given value(s) as arguments. +> - As in other parts of `unpythonic`, returning a `Values` means returning multiple-return-values and/or named-return-values. +> - This is important if the return value is received by the assignment targets of a `call_cc[]`. If you get a `TypeError` concerning the arguments of a function with a name ending in `_cont_`, check your `call_cc[]` invocations and the `return` in the call_cc'd function. > - **Changed in v0.15.0.** *Up to v0.14.3, multiple return values used to be represented as a `tuple`. Now returning a `tuple` means returning one value that is a tuple.* > - `return func(...)` is actually a tail-call into `func`, passing along (by default) the current value of `cc` to become its `cc`. -> - Hence, the tail call is inserted between the end of the current function body and the start of the continuation `cc`. -> - To override which continuation to use, you can specify the `cc=...` kwarg, as in `return func(..., cc=mycc)`. +> - Hence, the tail call is inserted *between* the end of the current function body and the start of the continuation `cc`. +> - To override which continuation to use, you can specify the `cc=...` kwarg, as in `return func(..., cc=mycc)`, as was done in the `amb` example above. > - The `cc` argument, if passed explicitly, **must be passed by name**. > - **CAUTION**: This is **not** enforced, as the machinery does not analyze positional arguments in any great detail. The machinery will most likely break in unintuitive ways (or at best, raise a mysterious `TypeError`) if this rule is violated. > - The function `func` must be a defined in a `with continuations` block, so that it knows what to do with the named argument `cc`. -> - Attempting to tail-call a regular function breaks the TCO chain and immediately returns to the original caller (provided the function even accepts a `cc` named argument). +> - Attempting to tail-call a regular function breaks the TCO chain and immediately returns to the original caller (provided the function even accepts a `cc` named argument; if not, you will get a `TypeError`). > - Be careful: `xs = list(args); return xs` and `return list(args)` mean different things. -> - TCO is automatically applied to these tail calls. This uses the exact same machinery as the `tco` macro. +> - Because `list(args)` is a function call, `return list(args)` will attempt to tail-call `list` as a continuation-enabled function (which it is not, you will get a `TypeError`), before passing its result into the current continuation. +> - Using `return xs` instead will pass an inert data value into the current continuation. +> - TCO is automatically applied to these tail calls. The TCO processing of `continuations` uses the exact same machinery as the `tco` macro, performing some additional AST edits via hooks. > > - The `call_cc[]` statement essentially splits its use site into *before* and *after* parts, where the *after* part (the continuation) can be run a second and further times, by later calling the callable that represents the continuation. This makes a computation resumable from a desired point. > - The continuation is essentially a closure. @@ -1329,12 +1338,12 @@ Code within a `with continuations` block is treated specially. > - Assignment targets can be used to get the return value of the function called by `call_cc[]`. > - Just like in Scheme/Racket's `call/cc`, the values that get bound to the `call_cc[]` assignment targets on second and further calls (when the continuation runs) are the arguments given to the continuation when it is called (whether implicitly or manually). > - A first-class reference to the captured continuation is available in the function called by `call_cc[]`, as its `cc` argument. -> - The continuation is a function that takes positional arguments, plus a named argument `cc`. +> - The continuation itself is a function that takes positional arguments, plus a named argument `cc`. > - The call signature for the positional arguments is determined by the assignment targets of the `call_cc[]`. > - The `cc` parameter is there only so that a continuation behaves just like any continuation-enabled function when tail-called, or when later used as the target of another `call_cc[]`. -> - Basically everywhere else, `cc` points to the identity function - the default continuation just returns its arguments. +> - Basically everywhere else, `cc` points to the identity function - the default continuation just returns its argument(s). > - This is unlike in Scheme or Racket, which implicitly capture the continuation at every expression. -> - Inside a `def`, `call_cc[]` generates a tail call, thus terminating the original (parent) function. (Hence `call_ec` does not combo well with this.) +> - Inside a `def`, `call_cc[]` generates a tail call, thus terminating the original (parent) function. Hence `call_ec` does **not** combo with `with continuations`. > - At the top level of the `with continuations` block, `call_cc[]` generates a normal call. In this case there is no return value for the block (for the continuation, either), because the use site of the `call_cc[]` is not inside a function.
@@ -1346,10 +1355,10 @@ Code within a `with continuations` block is treated specially. - Python's built-in generators have no restriction on where `yield` can be placed, and provide better performance. - Racket's standard library provides [generators](https://docs.racket-lang.org/reference/Generators.html). - - Unlike **exceptions**, which only perform escapes, `call_cc[]` allows to jump back at an arbitrary time later, also after the dynamic extent of the original function where the `call_cc[]` appears. Escape continuations are a special case of continuations, so exceptions can be built on top of `call/cc`. + - Unlike **exceptions**, which only perform escapes, `call_cc[]` allows to jump back at an arbitrary time later, also *after* the dynamic extent of the original function where the `call_cc[]` appears. Escape continuations are a special case of continuations, so exceptions can be built on top of `call/cc`. - [As explained in detail by Matthew Might](http://matt.might.net/articles/implementing-exceptions/), exceptions are fundamentally based on (escape) continuations; the *"unwinding the call stack"* mental image is ["not even wrong"](https://en.wikiquote.org/wiki/Wolfgang_Pauli). -So if all you want is generators or exceptions (or even resumable exceptions a.k.a. [conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), then a general `call/cc` mechanism is not needed. The point of `call/cc` is to provide the ability to *resume more than once* from *the same*, already executed point in the program. In other words, `call/cc` is a general mechanism for bookmarking the control state. +So if all you want is generators or exceptions (or even resumable exceptions a.k.a. [conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html)), then a general `call/cc` mechanism is not needed. The point of `call/cc` is to provide the ability to *resume more than once* from *the same*, already executed point in the program. In other words, **`call/cc` is a general mechanism for bookmarking the control state**. However, its usability leaves much to be desired. This has been noted e.g. in [Oleg Kiselyov: An argument against call/cc](http://okmij.org/ftp/continuations/against-callcc.html) and [John Shutt: Guarded continuations](http://fexpr.blogspot.com/2012/01/guarded-continuations.html). For example, Shutt writes: @@ -1360,11 +1369,11 @@ However, its usability leaves much to be desired. This has been noted e.g. in [O To keep things relatively straightforward, our `call_cc[]` is only allowed to appear **at the top level** of: - the `with continuations` block itself - - a `def` or `async def` + - a `def` inside that block Nested defs are ok; here *top level* only means the top level of the *currently innermost* `def`. -If you need to place `call_cc[]` inside a loop, use `@looped` et al. from the module `unpythonic.fploop`; this has the loop body represented as the top level of a `def`. +If you need to place `call_cc[]` inside a loop, use `@looped` et al. from the module `unpythonic.fploop`; this has the loop body represented as the top level of a `def`. Keep in mind that **only the control state is bookmarked**. Multiple `call_cc[]` statements in the same function body are allowed. These essentially create nested closures. @@ -1375,11 +1384,11 @@ In any invalid position, `call_cc[]` is considered a syntax error at macro expan In `unpythonic`, `call_cc` is a **statement**, with the following syntaxes: ```python -x = call_cc[func(...)] -*xs = call_cc[func(...)] -x0, ... = call_cc[func(...)] -x0, ..., *xs = call_cc[func(...)] -call_cc[func(...)] +x = call_cc[f(...)] +*xs = call_cc[f(...)] +x0, ... = call_cc[f(...)] +x0, ..., *xs = call_cc[f(...)] +call_cc[f(...)] x = call_cc[f(...) if p else g(...)] *xs = call_cc[f(...) if p else g(...)] @@ -1390,19 +1399,21 @@ call_cc[f(...) if p else g(...)] *NOTE*: `*xs` may need to be written as `*xs,` in order to explicitly make the LHS into a tuple. The variant without the comma seems to work when run from a `.py` file with the `macropython` bootstrapper from [`mcpyrate`](https://pypi.org/project/mcpyrate/), but fails in code run interactively in the `mcpyrate` REPL. -*NOTE*: `f()` and `g()` must be **literal function calls**. Sneaky trickery (such as calling indirectly via `unpythonic.call` or `unpythonic.curry`) is not supported. (The `prefix` and `curry` macros, however, **are** supported; just order the block macros as shown in the final section of this README.) This limitation is for simplicity; the `call_cc[]` needs to patch the `cc=...` kwarg of the call being made. +*NOTE*: `f()` and `g()` must be **literal function calls**. Sneaky trickery (such as calling indirectly via `unpythonic.call` or `unpythonic.curry`) is not supported. This limitation is for simplicity; the `call_cc[]` invocation needs to patch the `cc=...` kwarg of the call being made. + +The `prefix` and `curry` macros, however, **are** supported; just order the block macros as in [The xmas tree combo](#the-xmas-tree-combo). **Assignment targets**: - To destructure positional multiple-values (from a `Values` return value of the function called by the `call_cc`), use a tuple assignment target (comma-separated names, as usual). Destructuring *named* return values from a `call_cc` is currently not supported due to syntactic limitations. - - The last assignment target may be starred. It is transformed into the vararg (a.k.a. `*args`, star-args) of the continuation function created by the `call_cc`. (It will capture a whole tuple, or any excess items, as usual.) + - The last assignment target may be starred. It is transformed into the vararg (a.k.a. `*args`, star-args) of the continuation function created by the `call_cc`. It will capture a whole tuple, or any excess items, as usual. - - To ignore the return value, just omit the assignment part. Useful if `func` was called only to perform its side-effects (the classic side effect is to stash `cc` somewhere for later use). + - To ignore the return value of the `call_cc`'d function, just omit the assignment part. This is useful if `f` was called only to perform its side-effects. The classic side effect is to stash `cc` somewhere for later use. **Conditional variant**: - - `p` is any expression. If truthy, `f(...)` is called, and if falsey, `g(...)` is called. + - `p` is any expression. It is evaluated at run time, as usual. When the result is truthy, `f(...)` is called, and when falsey, `g(...)` is called. - Each of `f(...)`, `g(...)` may be `None`. A `None` skips the function call, proceeding directly to the continuation. Upon skipping, all assignment targets (if any are present) are set to `None`. The starred assignment target (if present) gets the empty tuple. @@ -1427,15 +1438,17 @@ Scheme and Racket implicitly capture the continuation at every position, whereas Also, since there are limitations to where a `call_cc[]` may appear, some code may need to be structured differently to do some particular thing, if porting code examples originally written in Scheme or Racket. -Unlike `call/cc` in Scheme/Racket, our `call_cc` takes **a function call** as its argument, not just a function reference. Also, there's no need for it to be a one-argument function; any other args can be passed in the call. The `cc` argument is filled implicitly and passed by name; any others are passed exactly as written in the client code. +Unlike `call/cc` in Scheme/Racket, our `call_cc` takes **a function call** as its argument, not just a function reference. Also, there is no need for it to be a one-argument function; any other args can be passed in the call. The `cc` argument is filled implicitly and passed by name; any others are passed exactly as you write in the invocation. #### Combo notes **CAUTION**: Do not use `with tco` inside a `with continuations` block; `continuations` already implies TCO. The `continuations` macro **makes no attempt** to skip `with tco` blocks inside it. -If you need both `continuations` and `multilambda` simultaneously, the incantation is: +If you want to use `multilambda` inside a `with continuations` block, it needs to go on the outside: ```python +from unpythonic.syntax import macros, continuations, multilambda + with multilambda, continuations: f = lambda x: [print(x), x**2] assert f(42) == 1764 @@ -1443,11 +1456,13 @@ with multilambda, continuations: This works, because the `continuations` macro understands already expanded `let[]` and `do[]`, and `multilambda` generates and expands a `do[]`. (Any explicit use of `do[]` in a lambda body or in a `return` is also ok; recall that macros expand from inside out.) -Similarly, if you need `quicklambda`, apply it first: +Similarly, if you want to use `quicklambda` inside a `with continuations` block, place it on the outside: ```python +from unpythonic.syntax import macros, continuations, quicklambda, fn + with quicklambda, continuations: - g = f[_**2] + g = fn[_**2] assert g(42) == 1764 ``` @@ -1457,6 +1472,8 @@ To enable both of these, use `with quicklambda, multilambda, continuations` (alt #### Continuations as an escape mechanism +An escape continuation `ec` is a continuation, too. How can we use `cc` to escape? + Pretty much by the definition of a continuation, in a `with continuations` block, a trick that *should* at first glance produce an escape is to set `cc` to the `cc` of the caller, and then return the desired value. There is however a subtle catch, due to the way we implement continuations. First, consider this basic strategy, without any macros: @@ -1467,7 +1484,7 @@ from unpythonic import call_ec def double_odd(x, ec): if x % 2 == 0: # reject even "x" ec("not odd") - return 2*x + return 2 * x @call_ec def result1(ec): y = double_odd(42, ec) @@ -1482,7 +1499,9 @@ assert result1 == "not odd" assert result2 == "not odd" ``` -Now, can we use the same strategy with the continuation machinery? +Here `ec` is the escape continuation of the `result1`/`result2` block, due to the placement of the `call_ec`. + +Now, can we use the same strategy with the general continuation machinery? ```python from unpythonic.syntax import macros, continuations, call_cc @@ -1492,9 +1511,9 @@ with continuations: if x % 2 == 0: cc = ec return "not odd" - return 2*x + return 2 * x def main1(cc): - # cc actually has a default, so it's ok to not pass anything as cc here. + # cc actually has a default (`identity`), so it's ok to not pass anything as cc here. y = double_odd(42, ec=cc) # y = "not odd" z = double_odd(21, ec=cc) # we could tail-call, but let's keep this similar to the first example. return z @@ -1506,11 +1525,13 @@ with continuations: assert main2() == "not odd" ``` -In the first example, `ec` is the escape continuation of the `result1`/`result2` block, due to the placement of the `call_ec`. In the second example, the `cc` inside `double_odd` is the implicitly passed `cc`... which, naively, should represent the continuation of the current call into `double_odd`. So far, so good. +The `cc` inside `double_odd` is the implicitly passed `cc`... which, naively, should represent the continuation of the current call into `double_odd`. So far, so good. + +However, because the example contains no `call_cc[]` statements, the actual value of `cc`, anywhere in this example, is always just `identity`. Scan that again: *in this example, `cc` is not the actual continuation, because no continuation captures were requested.* -However, because the example code contains no `call_cc[]` statements, the actual value of `cc`, anywhere in this example, is always just `identity`. *It's not the actual continuation.* Even though we pass the `cc` of `main1`/`main2` as an explicit argument "`ec`" to use as an escape continuation (like the first example does with `ec`), it is still `identity` - and hence cannot perform an escape. +Even though we pass the `cc` of `main1`/`main2` as an explicit argument "`ec`" to use as an escape continuation (like the first example does with `ec`), it is still `identity` - and hence cannot perform an escape. -We must `call_cc[]` to request a capture of the actual continuation: +We must `call_cc[]` to request a capture of the continuation, hence populating `cc` with something useful: ```python from unpythonic.syntax import macros, continuations, call_cc @@ -1520,7 +1541,7 @@ with continuations: if x % 2 == 0: cc = ec return "not odd" - return 2*x + return 2 * x def main1(cc): y = call_cc[double_odd(42, ec=cc)] # <-- the only change is adding the call_cc[] z = call_cc[double_odd(21, ec=cc)] # <-- @@ -1535,45 +1556,46 @@ with continuations: This variant performs as expected. -There's also a second, even subtler catch; instead of setting `cc = ec` and returning a value, just tail-calling `ec` with that value doesn't do what we want. This is because - as explained in the rules of the `continuations` macro, above - a tail-call is *inserted* between the end of the function, and whatever `cc` currently points to. +There is also a second, even subtler catch; instead of setting `cc = ec` and returning a value, as we did, just tail-calling `ec` with that same value does **not** do what we want. Why? Because - as explained in the rules of the `continuations` macro, above - a tail-call is *inserted* between the end of the function, and whatever continuation `cc` currently points to. -Most often that's exactly what we want, but in this particular case, it causes *both* continuations to run, in sequence. But if we overwrite `cc`, then the function's original `cc` argument (the one given by `call_cc[]`) is discarded, so it never runs - and we get the effect we want, *replacing* the `cc` by the `ec`. +Most often that is exactly what we want, but in this particular case, it causes *both* continuations to run, in sequence. But if, instead of performing a tail call to the `ec`, we set `cc = ec`, then the function's original `cc` argument (the one supplied by `call_cc[]`) is discarded, hence that continuation never runs - and we get the effect we want, *replacing* the `cc` by the `ec`. Such subtleties arise essentially from the difference between a language that natively supports continuations (Scheme, Racket) and one that has continuations hacked on top of it as macros performing a CPS conversion only partially (like Python with `unpythonic.syntax`, or Common Lisp with PG's continuation-passing macros). The macro approach works, but the programmer needs to be careful. #### What can be used as a continuation? -In `unpythonic` specifically, a continuation is just a function. ([As John Shutt has pointed out](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html), in general this is not true. The calculus underlying the language becomes much cleaner if continuations are defined as a separate control flow mechanism orthogonal to function application. Continuations are not intrinsically a whole-computation device, either.) +In `unpythonic` specifically, a continuation is just a function. ([As John Shutt has pointed out](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html), in general this is not true. The calculus underlying the language becomes much cleaner if continuations are defined as a separate control flow mechanism orthogonal to function application. Continuations are [not intrinsically a whole-computation device](https://en.wikipedia.org/wiki/Delimited_continuation), either.) The continuation function must be able to take as many positional arguments as the previous function in the TCO chain is trying to pass into it. Keep in mind that: - - In `unpythonic`, multiple return values are represented as a `Values` object. So if your function does `return Values(a, b)`, and that is being fed into the continuation, this implies that the continuation must be able to take two positional arguments. + - In `unpythonic`, multiple return values (and named return values) are represented as a `Values` object. So if your function does `return Values(a, b)`, and that is being fed into the continuation, this implies that the continuation must be able to take two positional arguments. - **Changed in v0.15.0.** *Up to v0.14.3, a `tuple` used to represent multiple-return-values; now it denotes a single return value that is a tuple. The `Values` type allows not only multiple return values, but also **named** return values. These are fed as kwargs.* + **Changed in v0.15.0.** *Up to v0.14.3, a `tuple` used to represent multiple-return-values; now it denotes a single return value that is a tuple. The `Values` type allows not only multiple return values, but also **named** return values. Named return values are fed as kwargs.* - - At the end of any function in Python, at least an implicit bare `return` always exists. It will try to pass in the value `None` to the continuation, so the continuation must be able to accept one positional argument. (This is handled automatically for continuations created by `call_cc[]`. If no assignment targets are given, `call_cc[]` automatically creates one ignored positional argument that defaults to `None`.) + - At the end of any function in Python, at least an implicit bare `return` always exists. It will try to pass in the value `None` to the continuation, so a continuation must be able to accept one positional argument. + - This is handled automatically for continuations created by `call_cc[]`. If no assignment targets are given, `call_cc[]` automatically creates one ignored positional argument that defaults to `None`. -If there is an arity mismatch, Python will raise `TypeError` as usual. (The actual error message may be unhelpful due to the macro transformations; look for a mismatch in the number of values between a `return` and the call signature of a function used as a continuation (most often, the `f` in a `cc=f`).) +If there is an arity mismatch, Python will raise `TypeError` as usual. The actual error message may be unhelpful due to macro transformations. Look for a mismatch between a `return` and the call signature of a function used as a continuation. Most often, this is the `f` in a `cc=f`. Usually, a function to be used as a continuation is defined inside the `with continuations` block. This automatically introduces the implicit `cc` parameter, and in general makes the source code undergo the transformations needed by the continuation machinery. However, as the only exception to this rule, if the continuation is meant to act as the endpoint of the TCO chain - i.e. terminating the chain and returning to the original top-level caller - then it may be defined outside the `with continuations` block. Recall that in a `with continuations` block, returning an inert data value (i.e. not making a tail call) transforms into a tail-call into the `cc` (with the given data becoming its argument(s)); it does not set the `cc` argument of the continuation being called, or even require that it has a `cc` parameter that could accept one. -(Note also that a continuation that has no `cc` parameter cannot be used as the target of an explicit tail-call in the client code, since a tail-call in a `with continuations` block will attempt to supply a `cc` argument to the function being tail-called. Likewise, it cannot be used as the target of a `call_cc[]`, since this will also attempt to supply a `cc` argument.) - These observations make `unpythonic.identity` eligible as a continuation, even though it is defined elsewhere in the library and it has no `cc` parameter. +Finally, note that a function that has no `cc` parameter cannot be used as the target of an explicit tail-call inside a `with continuations` block, since a tail-call there will attempt to supply a `cc` argument to the function being tail-called. Likewise, it cannot be used as the function called by a `call_cc[]`, since this will also attempt to supply a `cc` argument. + #### This isn't `call/cc`! Strictly speaking, `True`. The implementation is very different (much more than just [exposing a hidden parameter](https://www.ps.uni-saarland.de/~duchier/python/continuations.html)), not to mention it has to be a macro, because it triggers capture - something that would not need to be requested for separately, had we converted the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style). -The selective capture approach is however more efficient when we implement the continuation system in Python, indeed *on Python* (in the sense of [On Lisp](paulgraham.com/onlisp.html)), since we want to run most of the program the usual way with no magic attached. This way there is no need to sprinkle absolutely every statement and expression with a `def` or a `lambda`. (Not to mention Python's `lambda` is underpowered due to the existence of some language features only as statements, so we would need to use a mixture of both, which is already unnecessarily complicated.) Function definitions are not intended as [the only control flow construct](https://dspace.mit.edu/handle/1721.1/5753) in Python, so the compiler likely wouldn't optimize heavily enough (i.e. eliminate **almost all** of the implicitly introduced function definitions), if we attempted to use them as such. +The selective capture approach is however more efficient when we implement the continuation system in Python, indeed *on Python* (in the sense of [On Lisp](paulgraham.com/onlisp.html)), since we want to run most of the program the usual way with no magic attached. This way there is no need to sprinkle absolutely every statement and expression with a `def` or a `lambda`. (Not to mention Python's `lambda` is underpowered due to the existence of some language features only as statements, so we would need to use a mixture of both, which is already unnecessarily complicated.) Function definitions are not intended as [the only control flow construct](https://dspace.mit.edu/handle/1721.1/5753) in Python, so the compiler likely would not optimize heavily enough (i.e. eliminate **almost all** of the implicitly introduced function definitions), if we attempted to use them as such. Continuations only need to come into play when we explicitly request for one ([ZoP §2](https://www.python.org/dev/peps/pep-0020/)); this avoids introducing any more extra function definitions than needed. -The name is nevertheless `call_cc`, because the resulting behavior is close enough to `call/cc`. +The name is nevertheless `call_cc`, because the resulting behavior is close enough to `call/cc`. Instead of *call with current continuation*, we could retcon the name to mean *call with **captured** continuation*. -Note our implementation provides a rudimentary form of *delimited* continuations. See [Oleg Kiselyov: Undelimited continuations are co-values rather than functions](http://okmij.org/ftp/continuations/undelimited.html). Delimited continuations return a value and can be composed, so they at least resemble functions (even though are not, strictly speaking, actually functions), whereas undelimited continuations do not even return. (For two different debunkings of the continuations-are-functions myth, approaching the problem from completely different angles, see the above post by Oleg Kiselyov, and [John Shutt: Continuations and term-rewriting calculi](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html).) +Note our implementation provides a rudimentary form of *delimited* continuations. See [Oleg Kiselyov: Undelimited continuations are co-values rather than functions](http://okmij.org/ftp/continuations/undelimited.html). Delimited continuations return a value and can be composed, so they at least resemble functions (even though are not, strictly speaking, actually functions), whereas undelimited continuations do not even return. For two different debunkings of the continuations-are-functions myth, approaching the problem from completely different angles, see the above post by Oleg Kiselyov, and [John Shutt: Continuations and term-rewriting calculi](http://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html). Racket provides a thought-out implementation of delimited continuations and [prompts](https://docs.racket-lang.org/guide/prompt.html) to control them. From ff04df239023cbe5747f71c695bbad54968ea53e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 16:35:53 +0300 Subject: [PATCH 591/832] spelling: let-bindings --- doc/features.md | 2 +- doc/macros.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/features.md b/doc/features.md index 7e881077..d3131f76 100644 --- a/doc/features.md +++ b/doc/features.md @@ -368,7 +368,7 @@ letrec[[evenp << (lambda x: evenp(42)] # --> True ``` -(*The transformations made by the macros may be the most apparent when comparing these examples. Note that the macros scope the `let` bindings lexically, automatically figuring out which `let` environment, if any, to refer to.*) +(*The transformations made by the macros may be the most apparent when comparing these examples. Note that the macros scope the let-bindings lexically, automatically figuring out which `let` environment, if any, to refer to.*) ### `env`: the environment diff --git a/doc/macros.md b/doc/macros.md index 3a4f4bc6..5f92fceb 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -100,7 +100,7 @@ Macros that introduce new ways to bind identifiers. ### `let`, `letseq`, `letrec` as macros -**Changed in v0.15.0.** *Added support for env-assignment syntax in the bindings subform. For consistency with other env-assignments, this is now the preferred syntax to establish let bindings. Additionally, the old lispy syntax now accepts also brackets, for consistency with the use of brackets for macro invocations.* +**Changed in v0.15.0.** *Added support for env-assignment syntax in the bindings subform. For consistency with other env-assignments, this is now the preferred syntax to establish let-bindings. Additionally, the old lispy syntax now accepts also brackets, for consistency with the use of brackets for macro invocations.* These macros provide properly lexically scoped `let` constructs, no boilerplate: @@ -128,7 +128,7 @@ let[x << 21][2 * x] There must be at least one binding; `let[][...]` is a syntax error, since Python's parser rejects an empty subscript slice. -Bindings are established using the `unpythonic` *env-assignment* syntax, `name << value`. The let bindings can be rebound in the body with the same env-assignment syntax, e.g. `x << 42`. +Bindings are established using the `unpythonic` *env-assignment* syntax, `name << value`. The let-bindings can be rebound in the body with the same env-assignment syntax, e.g. `x << 42`. The same syntax for the bindings subform is used by: @@ -179,7 +179,7 @@ The `where` operator, if used, must be macro-imported. It may only appear at the **Changed in v0.15.0.** -Beginning with v0.15.0, the env-assignment syntax presented above is the preferred syntax to establish let bindings, for consistency with other env-assignments. This reminds that let variables live in an `env`, which is created by the `let` form. +Beginning with v0.15.0, the env-assignment syntax presented above is the preferred syntax to establish let-bindings, for consistency with other env-assignments. This reminds that let variables live in an `env`, which is created by the `let` form. There is also an alternative, lispy notation for the bindings subform, where each name-value pair is given using brackets: @@ -209,7 +209,7 @@ let[(x, 42) in ...] let[..., where(x, 42)] ``` -Even though an expr macro invocation itself is always denoted using brackets, as of `unpythonic` v0.15.0 parentheses can still be used *to pass macro arguments*, hence `let(...)[...]` is still accepted. The code that interprets the AST for the let bindings accepts both lists and tuples for each key-value pair, and the top-level container for the bindings subform in a let-in or let-where can be either list or tuple, so whether brackets or parentheses are used does not matter there, either. +Even though an expr macro invocation itself is always denoted using brackets, as of `unpythonic` v0.15.0 parentheses can still be used *to pass macro arguments*, hence `let(...)[...]` is still accepted. The code that interprets the AST for the let-bindings accepts both lists and tuples for each key-value pair, and the top-level container for the bindings subform in a let-in or let-where can be either list or tuple, so whether brackets or parentheses are used does not matter there, either. Still, brackets are now the preferred delimiter, for consistency between the bindings and body subforms. @@ -753,7 +753,7 @@ The naming is performed using the function `unpythonic.namelambda`, which will r - Expression-assignment to an unpythonic environment, `f << (lambda ...: ...)` - Env-assignments are processed lexically, just like regular assignments. This should not cause problems, because left-shifting by a literal lambda most often makes no sense (whence, that syntax is *almost* guaranteed to mean an env-assignment). - - Let bindings, `let[[f << (lambda ...: ...)] in ...]`, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). + - Let-bindings, `let[[f << (lambda ...: ...)] in ...]`, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). - Named argument in a function call, as in `foo(f=lambda ...: ...)`. **Added in v0.14.2.** @@ -1621,7 +1621,7 @@ The `call_cc[]` explicitly suggests that these are (almost) the only places wher Write Python almost like Lisp! -Lexically inside a `with prefix` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the `let` macros and the top-level tuple in a `do[]` are left alone, but `prefix` recurses inside them (in the case of bindings, on each RHS). +Lexically inside a `with prefix` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the `let` macros and the top-level tuple in a `do[]` are left alone, but `prefix` recurses inside them (in the case of let-bindings, on each RHS). The rest is best explained by example: From fb394b58748437187a5a2bdf77b1fb8960d70b7a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 16:38:15 +0300 Subject: [PATCH 592/832] macro name: autocurry --- doc/dialects/listhell.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/listhell.md b/doc/dialects/listhell.md index f2e018c9..20e29cda 100644 --- a/doc/dialects/listhell.md +++ b/doc/dialects/listhell.md @@ -47,7 +47,7 @@ assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ## Features -In terms of `unpythonic.syntax`, we implicitly enable `prefix` and `curry` for the whole module. +In terms of `unpythonic.syntax`, we implicitly enable `prefix` and `autocurry` for the whole module. The following are dialect builtins: From fe7f0d1d728d40f130e5deea2c2691536b3aeeb9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 16:38:54 +0300 Subject: [PATCH 593/832] 0.15.0: update prefix macro doc --- doc/macros.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 5f92fceb..c49b0ce0 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1674,7 +1674,7 @@ with prefix: If you use the `q`, `u` and `kw()` operators, they must be macro-imported. The `q`, `u` and `kw()` operators may only appear in a tuple inside a prefix block. In any invalid position, any of them is considered a syntax error at macro expansion time. -This comboes with `autocurry` for an authentic *Listhell* programming experience: +The `prefix` macro comboes with `autocurry` for an authentic *Listhell* programming experience: ```python from unpythonic.syntax import macros, autocurry, prefix, q, u, kw @@ -1687,6 +1687,8 @@ with prefix, autocurry: # important: apply prefix first, then autocurry assert (mymap, double, (q, 1, 2, 3)) == ll(2, 4, 6) ``` +See also [the Listhell dialect](dialects/listhell.md), which pre-packages that combo. + **CAUTION**: The `prefix` macro is experimental and not intended for use in production code. From 3755a1ff251c760701e846872f6b59f4555c5a90 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 17:03:07 +0300 Subject: [PATCH 594/832] autoreturn: return an inner function/class definition, too --- CHANGELOG.md | 1 + unpythonic/syntax/tailtools.py | 61 ++++++++++++++++--------- unpythonic/syntax/tests/test_autoret.py | 32 ++++++++++--- 3 files changed, 66 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 74d63a86..1870d201 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -122,6 +122,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - If any extra arguments (positional or named) remain when the top-level curry context exits, then by default, `TypeError` is raised. To override, use `with dyn.let(curry_context=["whatever"])`, just like before. Then you'll get a `Values` object. - The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Asking for the `len` returns the current length of the memo. For subscripting, both a single `int` index and a slice are accepted. Note that memoized generators do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose. - `fup`/`fupdate`/`ShadowedSequence` can now walk the start of a memoized infinite replacement backwards. (Use `imemoize` on the original iterable, instantiate the generator, and use that generator instance as the replacement.) + - When using the `autoreturn` macro, if the item in tail position is a function definition or class definition, return the thing that was defined. - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The internal exception types `unpythonic.conditions.InvokeRestart` and `unpythonic.ec.Escape` now inherit from `BaseException`, so that they are not inadvertently caught by `except Exception` handlers. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index d28a59ca..7f2951f2 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -9,7 +9,7 @@ from functools import partial -from ast import (Lambda, FunctionDef, AsyncFunctionDef, +from ast import (Lambda, FunctionDef, AsyncFunctionDef, ClassDef, arguments, arg, keyword, List, Tuple, Call, Name, Starred, Constant, @@ -668,28 +668,45 @@ def transform(self, tree): if is_captured_value(tree): return tree # don't recurse! if type(tree) in (FunctionDef, AsyncFunctionDef): - tree.body[-1] = transform_tailstmt(tree.body[-1]) + newtail = TailStatementTransformer().visit(tree.body[-1]) + if isinstance(newtail, list): # replaced by more than one statement? + tree.body = tree.body[:-1] + newtail + else: + tree.body[-1] = newtail return self.generic_visit(tree) - def transform_tailstmt(tree): - # TODO: For/AsyncFor/While? - if type(tree) is If: - tree.body[-1] = transform_tailstmt(tree.body[-1]) - if tree.orelse: - tree.orelse[-1] = transform_tailstmt(tree.orelse[-1]) - elif type(tree) in (With, AsyncWith): - tree.body[-1] = transform_tailstmt(tree.body[-1]) - elif type(tree) is Try: - # We don't care about finalbody; typically used for unwinding only. - if tree.orelse: # tail position is in else clause if present - tree.orelse[-1] = transform_tailstmt(tree.orelse[-1]) - else: # tail position is in the body of the "try" - tree.body[-1] = transform_tailstmt(tree.body[-1]) - # additionally, tail position is in each "except" handler - for handler in tree.handlers: - handler.body[-1] = transform_tailstmt(handler.body[-1]) - elif type(tree) is Expr: - tree = Return(value=tree.value) - return tree + + class TailStatementTransformer(ASTTransformer): + def transform(self, tree): + # TODO: For/AsyncFor/While? + if type(tree) is If: + tree.body[-1] = self.visit(tree.body[-1]) + if tree.orelse: + tree.orelse[-1] = self.visit(tree.orelse[-1]) + elif type(tree) in (With, AsyncWith): + tree.body[-1] = self.visit(tree.body[-1]) + elif type(tree) is Try: + # We don't care about finalbody; typically used for unwinding only. + if tree.orelse: # tail position is in else clause if present + tree.orelse[-1] = self.visit(tree.orelse[-1]) + else: # tail position is in the body of the "try" + tree.body[-1] = self.visit(tree.body[-1]) + # additionally, tail position is in each "except" handler + for handler in tree.handlers: + handler.body[-1] = self.visit(handler.body[-1]) + elif type(tree) in (FunctionDef, AsyncFunctionDef, ClassDef): # v0.15.0+ + # If the item in tail position is a named function definition + # or a class definition, it binds a name - that of the function/class. + # Return that object. + with q as quoted: + with a: + tree + return n[tree.name] + tree = quoted + elif type(tree) is Expr: # expr -> return expr + with q as quoted: + return a[tree.value] + tree = quoted[0] + return tree # This macro expands outside-in. Any nested macros should get clean standard Python, # not having to worry about implicit "return" statements. return AutoreturnTransformer().visit(block_body) diff --git a/unpythonic/syntax/tests/test_autoret.py b/unpythonic/syntax/tests/test_autoret.py index d1e431f9..d0baa492 100644 --- a/unpythonic/syntax/tests/test_autoret.py +++ b/unpythonic/syntax/tests/test_autoret.py @@ -16,8 +16,8 @@ def runtests(): # - if you need a loop in tail position to have a return value, # use an explicit return, or the constructs from unpythonic.fploop. # - any explicit return statements are left alone, so "return" can be used normally. - with autoreturn: - with testset("basic usage"): + with testset("basic usage"): + with autoreturn: def f(): "I'll just return this" test[f() == "I'll just return this"] @@ -26,7 +26,8 @@ def f2(): return "I'll just return this" # explicit return, not transformed test[f2() == "I'll just return this"] - with testset("if, elif, else"): + with testset("if, elif, else"): + with autoreturn: def g(x): if x == 1: "one" @@ -38,7 +39,8 @@ def g(x): test[g(2) == "two"] test[g(42) == "something else"] - with testset("except, else"): + with testset("except, else"): + with autoreturn: def h(x): try: if x == 1: @@ -50,7 +52,8 @@ def h(x): test[h(10) == 20] test[h(1) == "error"] - with testset("except, body of the try"): + with testset("except, body of the try"): + with autoreturn: def h2(x): try: if x == 1: @@ -61,12 +64,29 @@ def h2(x): test[h2(10) == 10] test[h2(1) == "error"] - with testset("with block"): + with testset("with block"): + with autoreturn: def ctx(): with env(x="hi") as e: # just need some context manager for testing, doesn't matter which e.x # tail position in a with block test[ctx() == "hi"] + with testset("function definition"): # v0.15.0+ + with autoreturn: + def outer(): + def inner(): + "inner function" + test[callable(outer())] # returned a function + test[outer()() == "inner function"] + + with testset("class definition"): # v0.15.0+ + with autoreturn: + def classdefiner(): + class InnerClassDefinition: + pass + test[isinstance(classdefiner(), type)] # returned a class + test[classdefiner().__name__ == "InnerClassDefinition"] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 00a63f7db52b629eec3224427cc7405aea72882c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 17:22:40 +0300 Subject: [PATCH 595/832] 0.15.0: update autoreturn macro docs --- doc/macros.md | 43 +++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index c49b0ce0..031462f3 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1694,9 +1694,13 @@ See also [the Listhell dialect](dialects/listhell.md), which pre-packages that c ### `autoreturn`: implicit `return` in tail position -In Lisps, a function implicitly returns the value of the expression in tail position (along the code path being executed). Python's `lambda` also behaves like this (the whole body is just one return-value expression), but `def` doesn't. +**Changed in v0.15.0.** *If the item in tail position is a function definition or class definition, return the thing that was defined. This functionality being missing in earlier versions was an oversight.* -Now `def` can, too: +In Lisps, a function implicitly returns the value of the expression in tail position along the code path being executed. That is, "the last value" is automatically returned when the function terminates normally. No `return` keyword is needed. + +Python's `lambda` also already behaves like this; the whole body is just one expression, whose value will be returned. + +However, `def` requires a `return`, even in tail position. Enter the `autoreturn` macro: ```python from unpythonic.syntax import macros, autoreturn @@ -1720,28 +1724,39 @@ with autoreturn: assert g(42) == "something else" ``` -Each `def` function definition lexically within the `with autoreturn` block is examined, and if the last item within the body is an expression `expr`, it is transformed into `return expr`. Additionally: +Each `def` or `async def` function definition lexically within the `with autoreturn` block is examined. + +Any explicit `return` statements are left alone, so `return` can still be used as usual. This is especially useful if you want to return early (before execution reaches the tail position). - - If the last item is an `if`/`elif`/`else` block, the transformation is applied to the last item in each of its branches. +To find and transform the statement(s) in tail position, we look at the last statement within the function definition. If it is: - - If the last item is a `with` or `async with` block, the transformation is applied to the last item in its body. + - An expression `expr`, it is transformed into `return expr`. - - If the last item is a `try`/`except`/`else`/`finally` block: - - **If** an `else` clause is present, the transformation is applied to the last item in it; **otherwise**, to the last item in the `try` clause. These are the positions that indicate a normal return (no exception was raised). - - In both cases, the transformation is applied to the last item in each of the `except` clauses. - - The `finally` clause is not transformed; the intention is it is usually a finalizer (e.g. to release resources) that runs after the interesting value is already being returned by `try`, `else` or `except`. + - A function or class definition, a return statement is appended to return that function/class. **Added in v0.15.0.** -If needed, the above rules are applied recursively to locate the tail position(s). + - An `if`/`elif`/`else` block, the transformation is applied recursively to the last item in each of its branches. + - **CAUTION**: If the final `else` of an `if`/`elif`/`else` is omitted, as often in Python, then only the `else` item is in tail position with respect to the function definition - likely not what you want. So with `autoreturn`, the final `else` should be written out explicitly, to include the `else` branch into the `if`/`elif`/`else` statement. -Any explicit `return` statements are left alone, so `return` can still be used as usual. + - A `with` or `async with` block, the transformation is applied recursively to the last item in its body. -**CAUTION**: If the final `else` of an `if`/`elif`/`else` is omitted, as often in Python, then only the `else` item is in tail position with respect to the function definition - likely not what you want. So with `autoreturn`, the final `else` should be written out explicitly, to make the `else` branch part of the same `if`/`elif`/`else` block. + - A `try`/`except`/`else`/`finally` block: + - **If** an `else` clause is present, the transformation is applied recursively to the last item in it; **otherwise**, to the last item in the `try` clause. These are the positions that indicate a normal return (i.e. no exception was raised). + - In both cases, the transformation is applied recursively to the last item in each of the `except` clauses. + - The `finally` clause is not transformed; it is intended as a finalizer (e.g. to release resources) that runs after the interesting value is already being returned by `try`, `else` or `except`. **CAUTION**: `for`, `async for`, `while` are currently not analyzed; effectively, these are defined as always returning `None`. If the last item in your function body is a loop, use an explicit return. -**CAUTION**: With `autoreturn` enabled, functions no longer return `None` by default; the whole point of this macro is to change the default return value. The default return value is `None` only if the tail position contains a statement other than `if`, `with`, `async with` or `try`. +**CAUTION**: With `autoreturn` enabled, functions no longer return `None` by default; the whole point of this macro is to change the default return value. The default return value becomes `None` only if the tail position contains a statement other than `if`, `with`, `async with` or `try`. + +If you wish to omit `return` in tail calls, `autoreturn` comboes with `tco`. For the correct invocation order, see [the xmas tree combo](#the-xmas-tree-combo). + +For code using **conditions and restarts**: there is no special integration between `autoreturn` and the conditions-and-restarts subsystem of `unpythonic`. However, these should work together, because: -If you wish to omit `return` in tail calls, this comboes with `tco`; just apply `autoreturn` first (either `with autoreturn, tco:` or in nested format, `with tco:`, `with autoreturn:`). + - The `with restarts` form is just a `with` block, so it gets the `autoreturn` treatment. + - The handlers in a `with handlers` form are either separately defined functions, or lambdas. + - Lambdas need no `autoreturn`. + - If you `def` the handler functions in a `with autoreturn` block (either the same one or a different one; this does not matter), they will get the `autoreturn` treatment. + - The `with handlers` form itself is just `with` block, so it also gets the `autoreturn` treatment. ### `forall`: nondeterministic evaluation From a1bd1f04c338a618a983a0b5c32604eab82ce563 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 17:33:08 +0300 Subject: [PATCH 596/832] 0.15.0: update forall macro doc --- doc/macros.md | 10 +++++++--- unpythonic/amb.py | 3 +++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 031462f3..60816444 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1761,7 +1761,11 @@ For code using **conditions and restarts**: there is no special integration betw ### `forall`: nondeterministic evaluation -Behaves the same as the multiple-body-expression tuple comprehension `unpythonic.forall`, but implemented purely by AST transformation, with real lexical variables. This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see `unpythonic.syntax.forall`). +This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad. + +The `forall[]` expr macro behaves the same as the multiple-body-expression tuple comprehension `unpythonic.forall`, but the macro is implemented purely by AST transformation, using real lexical variables. + +The implementation is generic and very short; if interested, see the module [`unpythonic.syntax.forall`](../unpythonic/syntax/forall.py). Compare the module [`unpythonic.amb`](../unpythonic/amb.py), which implements the same functionality with a source code generator and `eval`, without macros. The macro implementation is both shorter and more readable; this is effectively a textbook example of a situation where macros are the clean solution. ```python from unpythonic.syntax import macros, forall @@ -1783,11 +1787,11 @@ assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), (8, 15, 17), (9, 12, 15), (12, 16, 20)) ``` -Assignment (**with** List-monadic magic) is `var << iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). +Assignment, **with** List-monadic magic, is `var << iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). `insist` and `deny` are not macros; they are just the functions from `unpythonic.amb`, re-exported for convenience. -The error raised by an undefined name in a `forall` section is `NameError`. +The error raised by an undefined name in a `forall[]` section is `NameError`. ## Convenience features diff --git a/unpythonic/amb.py b/unpythonic/amb.py index 193af8db..cf8d3f8e 100644 --- a/unpythonic/amb.py +++ b/unpythonic/amb.py @@ -199,6 +199,9 @@ def begin(*exprs): # args eagerly evaluated by Python mlst = eval(allcode, {"e": e, "bodys": bodys, "begin": begin, "monadify": monadify}) return tuple(mlst) +# -------------------------------------------------------------------------------- +# This low-level machinery is shared with the macro version, `unpythonic.syntax.forall`. + def monadify(value, unpack=True): """Pack value into a monadic list if it is not already. From c1805a6ec80b94a9b6c2129dee50ed42a2cc26c1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 17:58:33 +0300 Subject: [PATCH 597/832] 0.15.0: improve macro docs: convenience features cond, aif, autoref --- doc/macros.md | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 60816444..39568907 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1800,7 +1800,7 @@ Small macros that are not essential but make some things easier or simpler. ### `cond`: the missing `elif` for `a if p else b` -Now lambdas too can have multi-branch conditionals, yet remain human-readable: +With `cond`, lambdas too can have multi-branch conditionals, yet remain human-readable: ```python from unpythonic.syntax import macros, cond @@ -1811,7 +1811,7 @@ answer = lambda x: cond[x == 2, "two", print(answer(42)) ``` -Syntax is `cond[test1, then1, test2, then2, ..., otherwise]`. Expansion raises an error if the `otherwise` branch is missing. +Syntax is `cond[test1, then1, test2, then2, ..., otherwise]`. A missing `otherwise` branch is considered a syntax error at macro expansion time. Any part of `cond` may have multiple expressions by surrounding it with brackets: @@ -1822,24 +1822,32 @@ cond[[pre1, ..., test1], [post1, ..., then1], [postn, ..., otherwise]] ``` -To denote a single expression that is a literal list, use an extra set of brackets: `[[1, 2, 3]]`. +This is just the extra bracket syntax that denotes an implicit `do[]`. To denote a single expression that is a literal list, double the brackets: `[[1, 2, 3]]`. Just like in a `let[]` form, the outer brackets enable multiple-expression mode, and then the inner brackets denote a list. The multiple-expression mode is allowed also when there is just one expression. + +Inspired by the `cond` form of many Lisps. There is some variation between Lisp dialects on whether `cond` or `if` is preferable if the dialect provides both. For example, in [Racket](https://racket-lang.org/), `cond` is the [preferred](https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Conditionals%29) construct for writing conditionals. ### `aif`: anaphoric if -This is mainly of interest as a point of [comparison with Racket](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/aif.rkt); `aif` is about the simplest macro that relies on either the lack of hygiene or breaking thereof. +**Changed in v0.15.0.** *The `it` helper macro may only appear in the `then` and `otherwise` branches of an `aif[]`. Anywhere else, it is considered a syntax error at macro expansion time.* + +In linguistics, an [*anaphor*](https://en.wikipedia.org/wiki/Anaphora_(linguistics)) is an expression that refers to another, such as the English word *"it"*. [Anaphoric macros](https://en.wikipedia.org/wiki/Anaphoric_macro) are a lispy take on the concept. An anaphoric macro may, for example, implicitly define an `it` that the user code can then use, with the meaning defined by the macro. This is sometimes a useful technique to shorten code, but it can also make code unreadable by hiding definitions, so it should be used sparingly. + +Particularly, the *anaphoric if* is a classic macro, where `it` is automatically bound to the result of the test. We provide that macro as `aif[]`. + +Concerning readability, the anaphoric if is relatively harmless, because it is *almost* obvious from context that the only `it` that makes sense for a human to refer to is the test expression. ```python from unpythonic.syntax import macros, aif, it -aif[2*21, +aif[2 * 21, print(f"it is {it}"), print("it is falsey")] ``` -Syntax is `aif[test, then, otherwise]`. The magic identifier `it` (which **must** be imported as a macro, if used) refers to the test result while (lexically) inside the `then` and `otherwise` parts of `aif`, and anywhere else is considered a syntax error at macro expansion time. +Syntax is `aif[test, then, otherwise]`. The magic identifier `it` (which **must** be imported as a macro) refers to the test result while (lexically) inside the `then` and `otherwise` parts of `aif`, and anywhere else is considered a syntax error at macro expansion time. -Any part of `aif` may have multiple expressions by surrounding it with brackets (implicit `do[]`): +Any part of `aif` may have multiple expressions by surrounding it with brackets: ```python aif[[pre, ..., test], @@ -1847,11 +1855,15 @@ aif[[pre, ..., test], [post_false, ..., otherwise]] # "otherwise" branch ``` -To denote a single expression that is a literal list, use an extra set of brackets: `[[1, 2, 3]]`. +This is just the extra bracket syntax that denotes an implicit `do[]`. To denote a single expression that is a literal list, double the brackets: `[[1, 2, 3]]`. Just like in a `let[]` form, the outer brackets enable multiple-expression mode, and then the inner brackets denote a list. The multiple-expression mode is allowed also when there is just one expression. + +If interested, [compare with a Racket implementation](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/aif.rkt); `aif` is probably *the* simplest macro that relies on either the lack of [macro hygiene](https://en.wikipedia.org/wiki/Hygienic_macro) or intentional *breaking* thereof. ### `autoref`: implicitly reference attributes of an object +**CAUTION**: *This is a really, really bad idea that comes with serious readability and security implications. Python does not provide this construct itself, for good reason. Details below. Use with care, if at all.* + Ever wish you could `with(obj)` to say `x` instead of `obj.x` to read attributes of an object? Enter the `autoref` block macro: ```python @@ -1866,7 +1878,7 @@ with autoref(e): assert c == 3 # no c in e, so just c ``` -The transformation is applied for names in `Load` context only, including names found in `Attribute` or `Subscript` nodes. +The transformation is applied for names in `Load` context only, including names found inside `Attribute` or `Subscript` AST nodes, so things like `a[1]` and `a.x` are also valid (looking up `a` in `e`). Names in `Store` or `Del` context are not redirected. To write to or delete attributes of `o`, explicitly refer to `o.x`, as usual. @@ -1878,9 +1890,11 @@ See the [unit tests](../unpythonic/syntax/tests/test_autoref.py) for more usage This is similar to the JavaScript [`with` construct](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/with), which is nowadays [deprecated](https://2ality.com/2011/06/with-statement.html). See also [the ES6 reference on `with`](https://www.ecma-international.org/ecma-262/6.0/#sec-with-statement). -**CAUTION**: This construct was deprecated in JavaScript **for security reasons**. Since the autoref'd object **will hijack all name lookups**, use `with autoref` only with an object you trust! +**NOTE**: The JavaScript `with` and the Python `with` have nothing in common except the name. + +**CAUTION**: The `with` construct of JavaScript was deprecated **for security reasons**. Since the autoref'd object **will hijack all name lookups**, use `with autoref` only with an object you trust! In most Python code, this does not matter, as we are all adults here, but this *may* matter if a Python object arrives from an untrusted source in a networked app. -**CAUTION**: `with autoref` also complicates static code analysis or makes it outright infeasible, for the same reason. It is impossible to statically know whether something that looks like a bare name in the source code is actually a true bare name, or a reference to an attribute of the autoref'd object. That status can also change at any time, since the lookup is dynamic, and attributes can be added and removed dynamically. +**CAUTION**: `with autoref` complicates static code analysis or makes it outright infeasible. It is impossible to statically know whether something that looks like a bare name in the source code is actually a true bare name, or a reference to an attribute of the autoref'd object. That status can also change at any time, since the lookup is dynamic, and attributes can be added and removed dynamically. ## Testing and debugging From 247f552d4fa4bb23de2b449354dbc8fac5052305 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 18:00:10 +0300 Subject: [PATCH 598/832] wording --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 39568907..8c2df6d8 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1845,7 +1845,7 @@ aif[2 * 21, print("it is falsey")] ``` -Syntax is `aif[test, then, otherwise]`. The magic identifier `it` (which **must** be imported as a macro) refers to the test result while (lexically) inside the `then` and `otherwise` parts of `aif`, and anywhere else is considered a syntax error at macro expansion time. +Syntax is `aif[test, then, otherwise]`. The magic identifier `it` (which **must** be imported as a macro) refers to the test result while (lexically) inside the `then` and `otherwise` branches of an `aif[]`, and anywhere else is considered a syntax error at macro expansion time. Any part of `aif` may have multiple expressions by surrounding it with brackets: From e0ccd1c3a64ff31399dafbde62f4468848120222 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 18:49:43 +0300 Subject: [PATCH 599/832] 0.15.0: update test framework docs --- doc/macros.md | 90 ++++++++++++++++++++++++++++----------------------- 1 file changed, 50 insertions(+), 40 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 8c2df6d8..d670d7d3 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1954,29 +1954,35 @@ with session("simple framework demo"): test[2 * 2 == 4] # not reached ``` -By default, running this script through the `macropython` wrapper (from `mcpyrate`) will produce an ANSI-colored test report in the terminal. To actually see how the output looks like, for actual runnable examples, see `unpythonic`'s own automated tests. +By default, running this script through the `macropython` wrapper (from `mcpyrate`) will produce an ANSI-colored test report in the terminal. To actually see how the output looks like, and for actual runnable examples, see `unpythonic`'s own automated tests. If you want to turn coloring off (e.g. for the purposes of redirecting stderr to a file), see the `TestConfig` bunch of constants in `unpythonic.test.fixtures`. -The following is an overview of the framework. For details, look at the docstrings of the various constructs in `unpythonic.test.fixtures` (which provides much of this), those of the test macros, and finally, the automated tests of `unpythonic` itself. +The following is an overview of the framework. For details, look at the docstrings of the various constructs in `unpythonic.test.fixtures` (which provides much of this), those of the testing macros, and finally, the automated tests of `unpythonic` itself. Tests can be found in subfolders named `tests`: [regular code](../unpythonic/tests/), [macros](../unpythonic/syntax/tests/), [dialects](../unpythonic/dialects/tests/). -How to test code using conditions and restarts can be found in [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py). +Examples of how to test code using conditions and restarts can be found in [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py). -How to test macro utilities (e.g. syntax transformer functions that operate on ASTs) can be found in [`unpythonic.syntax.tests.test_letdoutil`](../unpythonic/syntax/tests/test_letdoutil.py). +Examples of how to test macro utilities (e.g. syntax transformer functions that operate on ASTs) can be found in [`unpythonic.syntax.tests.test_letdoutil`](../unpythonic/syntax/tests/test_letdoutil.py). + +**NOTE**: If you want to compartmentalize macro expansion in your tests (so that an error during macro expansion will not crash your test unit), `mcpyrate` offers more than one way to invoke the macro expander at run time ([*of your test unit*](https://github.com/Technologicat/mcpyrate/blob/master/doc/troubleshooting.md#macro-expansion-time-where-exactly)), depending on what exactly you want to do. One is the `mcpyrate.metatools.expand` family of macros, and another are the functions in the module `mcpyrate.compiler`. See [the `mcpyrate` user manual](https://github.com/Technologicat/mcpyrate/blob/master/doc/main.md): specifically on [`metatools` (and quasiquoting)](https://github.com/Technologicat/mcpyrate/blob/master/doc/quasiquotes.md) and on [`compiler`](https://github.com/Technologicat/mcpyrate/blob/master/doc/compiler.md). The tests of `mcpyrate` itself provide some examples on how to use `compiler`. #### Overview -We provide the low-level syntactic constructs `test[]`, `test_raises[]` and `test_signals[]`, with the usual meanings. The last one is for testing code that uses the `signal` function and its sisters (related to conditions and restarts à la Common Lisp); see the module [`unpythonic.conditions`](../unpythonic/conditions.py), and the user manual section on [conditions and restarts](features.md#handlers-restarts-conditions-and-restarts). +All testing *macros* are provided in the module `unpythonic.syntax`. All regular functions related to testing are provided in the module `unpythonic.test.fixtures`. + +We provide the low-level syntactic constructs `test[]`, `test_raises[]` and `test_signals[]`, with the usual meanings. The last one is for testing code that uses `unpythonic.signal` and its sisters (related to conditions and restarts à la Common Lisp); see the module [`unpythonic.conditions`](../unpythonic/conditions.py), and the user manual section on [conditions and restarts](features.md#handlers-restarts-conditions-and-restarts). + +By default, the `test[expr]` macro asserts that the value of `expr` is truthy. If you want to assert only that `expr` runs to completion normally, use `test[returns_normally(expr)]`. Here `returns_normally` is a regular function, which is available in the module `unpythonic.test.fixtures`. -By default, the `test[expr]` macro asserts that the value of `expr` is truthy. If you want to assert only that `expr` runs to completion normally, use `test[returns_normally(expr)]`. +All three testing constructs also come in block variants, `with test`, `with test_raises[exctype]`, `with test_signals[exctype]`. -The test macros also come in block variants, `with test`, `with test_raises[exctype]`, `with test_signals[exctype]`. +As usual in test frameworks, the testing constructs behave somewhat like `assert`, with the difference that a failure or error will not abort the whole unit, unless explicitly asked to do so. There is no return value; upon success, the testing constructs return `None`. Upon failure (test assertion not satisfied) or error (unexpected exception or signal), the failure or error is reported, and further tests continue running. -As usual in test frameworks, the test constructs behave somewhat like `assert`, with the difference that a failure or error will not abort the whole unit (unless explicitly asked to do so). There is no return value; upon success, the test constructs return `None`. Upon failure (test assertion not satisfied) or error (unexpected exception or signal), the failure or error is reported, and further tests continue running. +All the variants of the testing constructs catch any uncaught exceptions and signals from inside the test expression or block. Any unexpected uncaught exception or signal is considered an error. -All the test variants catch any uncaught exceptions and signals from inside the test expression or block. Any unexpected uncaught exception or signal is considered an error. +Because `unpythonic.test.fixtures` is, by design, a minimalistic *no-framework* (cf. "NoSQL"), it is up to you to define - in your custom test runner - whether having any failures, errors or warnings should lead to the whole test suite failing. Whether the program's exit code is zero, is important e.g. for GitHub's CI workflows. -Because `unpythonic.test.fixtures` is, by design, a minimalistic *no-framework* (cf. "NoSQL"), it is up to you to define - in your custom test runner - whether having any failures, errors or warnings should lead to the whole test suite failing (whether the program's exit code is zero is important e.g. for GitHub's CI workflows). For example, in `unpythonic`'s own tests (see the very short [`runtests.py`](../runtests.py)), warnings do not cause the test suite to fail, but errors and failures do. +For example, in `unpythonic`'s own tests, warnings do not cause the test suite to fail, but errors and failures do. The very short [`runtests.py`](../runtests.py) (just under 60 SLOC) is a complete test runner using `unpythonic.test.fixtures`. #### Testing syntax quick reference @@ -1997,6 +2003,9 @@ from unpythonic.test.fixtures import session, testset def runtests(): with testset("something 1"): + test[...] + test_raises[TypeError, ...] + test_raises[ValueError, ...] ... with testset("something 2"): ... @@ -2007,9 +2016,9 @@ if __name__ == '__main__': # pragma: no cover runtests() ``` -The if-main idiom allows running this test module individually, but it is tagged with `# pragma: no cover`, so that the coverage reporter won't yell about it when the module is run by the test runner as part of the complete test suite (which, incidentally, is also a good opportunity to measure coverage). +The if-main idiom allows running this test module individually, but it is tagged with `# pragma: no cover`, so that the coverage reporter will not yell about it when the module is run by the test runner as part of the complete test suite (which, incidentally, is also a good opportunity to [measure coverage](../measure_coverage.sh)). -If you want to ensure that testing macros expand before anything else - including your own code-walking block macros (when you have tests inside the body) - import the macro `expand_testing_macros_first`, and put a `with expand_testing_macros_first` around the affected code. (See [Expansion order](#expansion-order), below.) +If you want to ensure that testing macros expand before anything else - including your own code-walking block macros (when you have tests inside the body of a `with` block that invokes a code-walking block macro) - import the macro `expand_testing_macros_first`, and put a `with expand_testing_macros_first` around the affected code. (See [Expansion order](#expansion-order), below.) **Sessions and testsets**: @@ -2020,7 +2029,7 @@ with session(name): with testset(name): ... - with testset(name): + with testset(name): # nested testset ... with testset(name): @@ -2028,11 +2037,11 @@ with session(name): ... ``` -Each `name` above is human-readable and optional. The purpose of the naming feature is to improve [scannability](https://www.teachingenglish.org.uk/article/scanning) of the testing report for the human reader. +Each `name` above is human-readable and optional. The purpose of the naming feature is to improve [scannability](https://www.teachingenglish.org.uk/article/scanning) of the testing report, and of the unit test source code, for the human reader. Note that even if `name` is omitted, the parentheses are still mandatory, because `session` and `testset` are just garden variety context managers that must be instantiated in order for them to perform their jobs. -A session implicitly introduces a top-level testset, for convenience. +A session implicitly introduces a top-level testset, for convenience - so if you only a have a few tests and don't want to group them, you do not need to use `with testset` at all. Testsets can be nested arbitrarily deep. @@ -2042,13 +2051,13 @@ Additional tools for code using **conditions and restarts**: The `catch_signals` context manager controls the signal barrier of `with testset` and the `test` family of syntactic constructs. It is provided for writing tests for code that uses conditions and restarts. -Used as `with catch_signals(False)`, it disables the signal barrier for the dynamic extent of the block. When the barrier is disabled, an uncaught signal (in the sense of `unpythonic.signal` and its sisters) is not considered as an errored test. This can be useful, because sometimes leaving a signal uncaught is the right thing to do. See [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py) for examples. +Used as `with catch_signals(False)`, it disables the signal barrier for the dynamic extent of the block. When the barrier is disabled, an uncaught signal (in the sense of `unpythonic.signal` and its sisters) is not considered as an error. This can be useful, because sometimes leaving a signal uncaught is the right thing to do. See [`unpythonic.tests.test_conditions`](../unpythonic/tests/test_conditions.py) for examples. The `with catch_signals` construct can be nested. Used as `with catch_signals(True)`, it re-enables the barrier, if currently disabled, for the dynamic extent of that inner `with catch_signals` block. When a `with catch_signals` block exits, the previous state of the signal barrier is automatically restored. -**Expression** forms: +**Expression** forms - complete list: ```python test[expr] @@ -2064,24 +2073,25 @@ error[message] warn[message] ``` -Inside a `test`, the helper macro `the[]` is available to mark interesting subexpressions inside `expr`, for failure and error reporting. An `expr` may contain an arbitrary number of `the[]`. By default, if `expr` is a comparison, the leftmost term is automatically marked (so that e.g. `test[x < 3]` will automatically report the value of `x` if the test fails); otherwise nothing. The default is only used if there is no explicit `the[]` inside `expr`. +Inside a `test[]`, the helper macro `the[]` is available to mark one or more interesting subexpressions inside `expr`, for failure and error reporting. An `expr` may contain an arbitrary number of `the[]`. By default, if `expr` is a comparison, the leftmost term is implicitly marked (so that e.g. `test[x < 3]` will automatically report the value of `x` if the test fails); otherwise nothing. The default is only used when there is **no** explicit `the[]` inside `expr`. The constructs `test_raises`, `test_signals`, `fail`, `error` and `warn` do **not** support `the[]`. Tests can be nested; this is sometimes useful as an explicit signal barrier. -Note the macros `error[]` and `warn[]` have nothing to do with the functions with the same name in the module `unpythonic.conditions`. The macros are part of the test framework; the functions with the same name are signaling protocols of the conditions and restarts system. Following the usual naming conventions in both systems, this naming conflict is unfortunately what we get. +Note that the testing constructs `error[]` and `warn[]`, which are macros, have nothing to do with the functions with the same name in the module `unpythonic.conditions`. The macros are part of the test framework; the functions with the same name are signaling protocols of the conditions and restarts system. Following the usual naming conventions separately in both systems, this naming conflict is unfortunately what we get. -**Block** forms: +**Block** forms - complete list: ```python with test: body ... + # no `return`; assert just that the block completes normally with test: body ... - return expr + return expr # assert that `expr` is truthy with test[message]: body ... @@ -2122,7 +2132,7 @@ with yourblockmacro: # outside-in Here the `...` may be edited by `yourblockmacro` before `test[]` sees it. (It likely **will** be edited, since this pattern will commonly appear in the tests for `yourblockmacro`, where the whole point is to have the `...` depend on what `yourblockmacro` outputs.) -If you need testing macros to expand before anything else even in this scenario (so you can more clearly see where in the unexpanded source code a particular expression came from), you can do this: +If you need testing macros to expand before anything else even in this scenario (so you can more clearly see where in the unexpanded source code a particular expression in a failing/erroring test came from), you can do this: ```python from unpythonic.syntax import macros, expand_testing_macros_first @@ -2132,9 +2142,9 @@ with expand_testing_macros_first: test[...] ``` -The `expand_testing_macros_first` macro is itself a code-walking block macro that does as it says on the tin. The testing macros are identified by scanning the bindings of the current macro expander; names don't matter, so it respects as-imports. +The `expand_testing_macros_first` macro is itself a code-walking block macro that does as it says on the tin. The testing macros are identified by scanning the bindings of the current macro expander; names do not matter, so it respects as-imports. -This does imply that `your_block_macro` will then receive the expanded form of `test[...]` as input, but that's macros for you. You'll have to choose which is more important: seeing the unexpanded code in error messages, or receiving unexpanded `test[]` expressions in `yourblockmacro`. +This does imply that `yourblockmacro` will then receive the expanded form of `test[...]` as input, but that's macros for you. You will have to choose which is more important: seeing the unexpanded code in error messages, or receiving unexpanded `test[]` expressions in `yourblockmacro`. #### `with test`: test blocks @@ -2144,13 +2154,13 @@ In `unpythonic.test.fixtures`, **a test block is implicitly lifted into a functi By default, a `with test` block asserts just that it completes normally. If you instead want to assert that an expression is truthy, use `return expr` to terminate the implicit function and return the value of the desired `expr`. The return value is passed to the test asserter for checking that it is truthy. -(Another way to view the default behavior is that the `with test` macro injects a `return True` at the end of the block, if there is no `return`. This is actually how the default behavior is implemented.) +Another way to view the default behavior is that the `with test` macro injects a `return True` at the end of the block to terminate the implicit function, if there is no explicit `return`. This is actually how the default behavior is implemented. -The `with test_raises[exctype]` and `with test_signals[exctype]` blocks assert that the block raises (respectively, signals) the declared exception (condition) type. These blocks are implicitly lifted into functions, too, but they do not check the return value. For them, **not** raising/signaling the declared exception/condition type is considered a test failure. Raising/signaling some other (hence unexpected) exception/condition type is considered an error. +The `with test_raises[exctype]` and `with test_signals[exctype]` blocks assert that the block raises (respectively, signals) the declared exception type. These blocks are implicitly lifted into functions, too, but they do not check the return value. For them, **not** raising/signaling the declared exception type is considered a test failure. Raising/signaling some other (hence unexpected) exception type is considered an error. #### `the`: capture the value of interesting subexpressions -The point of `unpythonic.test.fixtures` is to make testing macro-enabled Python as frictionless as reasonably possible. +The point of `unpythonic.test.fixtures` is to make testing macro-enabled Python as frictionless as reasonably possible. Thus we provide this convenience feature. Inside a `test[]` expression, or anywhere within the code in a `with test` block, the `the[]` macro can be used to declare any number of subexpressions as interesting, for capturing the source code and value into the test failure message, which is shown if the test fails. Each `the[]` captures one subexpression (as many times as it is evaluated, in the order evaluated). @@ -2158,7 +2168,7 @@ Because test macros expand outside-in, the source code is captured before any ne By default (if no explicit `the[]` is present), `test[]` implicitly inserts a `the[]` for the leftmost term if the top-level expression is a comparison (common use case), and otherwise does not capture anything. -When nothing is captured, if the test fails, the value of the whole expression is shown. Of course, you'll then already know the value is falsey, but there's still the possibly useful distinction of whether it's, say, `False`, `None`, `0` or `[]`. +When nothing is captured, if the test fails, the value of the whole expression is shown. Of course, you will then already know the value is falsey, but there is still the possibly useful distinction of whether it is, say, `False`, `None`, `0` or `[]`. A `test[]` or `with test` can have any number of subexpressions marked as `the[]`. It is possible to even nest a `the[]` inside another `the[]`, if you need the value of some subexpression as well as one of *its* subexpressions. The captured values are gathered, in the order they were evaluated (by Python's standard evaluation rules), into a list that is shown upon test failure. @@ -2168,25 +2178,25 @@ In case of nested `test[]` or nested `with test`, each `the[...]` is understood The `the[]` mechanism is smart enough to skip reporting trivialities for literals, such as `(1, 2, 3) = (1, 2, 3)` in `test[4 in the[(1, 2, 3)]]`, or `4 = 4` in `test[4 in (1, 2, 3)]`. In the second case, note the implicit `the[]` on the LHS, because `in` is a comparison operator. -If nothing but such trivialities were captured, the failure message will instead report the value of the whole expression. (The captures still remain inspectable in the exception instance.) +If nothing but such trivialities were captured, the failure message will instead report the value of the whole expression. The captures still remain inspectable in the exception instance. -To make testing/debugging macro code more convenient, the `the[]` mechanism automatically unparses an AST value into its source code representation for display in the test failure message. This is meant for debugging macro utilities, to which a test case hands some quoted code (i.e. code lifted into its AST representation using mcpyrate's `q[]` macro). See [`unpythonic.syntax.tests.test_letdoutil`](unpythonic/syntax/tests/test_letdoutil.py) for some examples. (Note the unparsing is done for display only; the raw value remains inspectable in the exception instance.) +To make testing/debugging macro code more convenient, the `the[]` mechanism automatically unparses an AST value into its source code representation for display in the test failure message. This is meant for debugging macro utilities, to which a test case hands some quoted code (i.e. code lifted into its AST representation using mcpyrate's `q[]` macro). See [`unpythonic.syntax.tests.test_letdoutil`](unpythonic/syntax/tests/test_letdoutil.py) for some examples. Note the unparsing is done for display only; the raw value remains inspectable in the exception instance. **CAUTION**: The source code is back-converted from the AST representation; hence its surface syntax may look slightly different to the original (e.g. extra parentheses). See `mcpyrate.unparse`. -**CAUTION**: The name of the `the[]` construct was inspired by Common Lisp, but the semantics are completely different. Common Lisp's `THE` is a return-type declaration (pythonistas would say *return-type annotation*), meant as a hint for the compiler to produce performance-optimized compiled code (see [chapter 32 of Peter Seibel's Practical Common Lisp](http://www.gigamonkeys.com/book/conclusion-whats-next.html)), whereas our `the[]` captures a value for test reporting. The only common factors are the name, and that neither construct changes the semantics of the marked code, much. In `unpythonic.test.fixtures`, the reason behind picking this name was that it doesn't change the flow of the source code as English that much, specifically to suggest, between the lines, that it doesn't change the semantics much. The reasoning behind CL's `THE` may be similar, but I have not researched its etymology. +**CAUTION**: The name of the `the[]` construct was inspired by Common Lisp, but that is where the similarities end. The `THE` construct of Common Lisp is a return-type declaration (pythonistas would say *return-type annotation*), meant as a hint for the compiler to produce performance-optimized compiled code. See [chapter 32 in Practical Common Lisp by Peter Seibel](http://www.gigamonkeys.com/book/conclusion-whats-next.html). In contrast, our `the[]` captures a value for test reporting. The only common factors are the name, and that neither construct changes the semantics of the marked code, much. In `unpythonic.test.fixtures`, the reason behind picking this name was that it does not change the flow of the source code as English that much, specifically to suggest, between the lines, that it does not change the semantics much. The reasoning behind CL's `THE` may be similar, but I have not researched its etymology. #### Test sessions and testsets The `with session()` in the example session above is optional. The human-readable session name is also optional, used for display purposes only. The session serves two roles: it provides an exit point for `terminate`, and defines an implicit top-level `testset`. -Tests can optionally be grouped into testsets. Each `testset` tallies passed, failed and errored tests within it, and displays the totals when it exits. Testsets can be named and nested. +Tests can optionally be grouped into testsets. Each `testset` tallies passed, failed and errored tests within it, and displays the totals when the context exits. Testsets can be named and nested. -It is useful to have at least one `testset` (the implicit top-level one established by `with session` is sufficient), because the `testset` mechanism forms one half of the test framework. It is possible to use the test macros without a `testset`, but that is only intended for building alternative test frameworks. +It is useful to have at least one `testset` (the implicit top-level one established by `with session` is fine), because the `testset` mechanism forms fully one half of the test framework. It is technically possible to use the testing macros without a `testset`, but that is only intended for building alternative test frameworks. Testsets also provide an option to locally install a `postproc` handler that gets a copy of each failure or error in that testset (and by default, any of its inner testsets), after the failure or error has been printed. In nested testsets, the dynamically innermost `postproc` wins. A failure is an instance of `unpythonic.test.fixtures.TestFailure`, an error is an instance of `unpythonic.test.fixtures.TestError`, and a warning is an instance of `unpythonic.test.fixtures.TestWarning`. All three inherit from `unpythonic.test.fixtures.TestingException`. Beside the human-readable message, these exception types contain attributes with programmatically inspectable information about what happened. -If you want to set a default global `postproc`, which is used when no local `postproc` is in effect, this too is configured in the `TestConfig` bunch of constants in `unpythonic.test.fixtures`. +If you want to set a default global `postproc`, which is used when no local `postproc` is in effect, this is configured in the `TestConfig` bunch of constants in `unpythonic.test.fixtures`. The `with testset` construct comes with one other important feature. The nearest dynamically enclosing `with testset` **catches any stray exceptions or signals** that occur within its dynamic extent, but outside a test construct. @@ -2220,19 +2230,19 @@ Look at the implementation of `testset` as an example. Because `unpythonic` is effectively a language extension, the standard options were not applicable. -The standard library's [`unittest`](https://docs.python.org/3/library/unittest.html) fails with `unpythonic` due to technical reasons related to `unpythonic`'s unfortunate choice of module names. The `unittest` framework chokes if a module in a library exports anything that has the same name as the module itself, and the library's top-level init then `from`-imports that construct into its namespace, causing the *module reference*, that was [implicitly brought in](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-submodules-are-added-to-the-package-namespace-trap) by the `from`-import itself, to be overwritten with what was explicitly imported: a reference to the construct that has the same name as the module. (Bad naming on my part, yes, but as of v0.15.0, I see no reason to cross that particular bridge yet.) +The standard library's [`unittest`](https://docs.python.org/3/library/unittest.html) fails with `unpythonic` due to technical reasons related to `unpythonic`'s unfortunate choice of module names. The `unittest` framework crashes if a module in a library exports anything that has the same name as the module itself, and the library's top-level init then `from`-imports that construct into its namespace, causing the *module reference*, that was [implicitly brought in](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-submodules-are-added-to-the-package-namespace-trap) by the `from`-import itself, to be overwritten with what was explicitly imported: a reference to the construct that has the same name as the module. This is bad naming on my part, yes, but as of v0.15.0, I see no reason to cross that particular bridge yet. -Also, in my opinion, `unittest` is overly verbose to use; automated tests are already a particularly verbose kind of program, even if the testing syntax is minimal. +Also, in my opinion, `unittest` is overly verbose to use; automated tests are already a particularly verbose kind of program, even if the testing syntax is minimal. Eliminating extra verbosity encourages writing more tests. -[Pytest](https://docs.pytest.org/en/latest/), on the other hand, provides compact syntax by hijacking the assert statement, but its import hook (to provide that syntax) can't coexist with a macro expander, which also needs to install a different import hook. It's also fairly complex. +[Pytest](https://docs.pytest.org/en/latest/), on the other hand, provides compact syntax by hijacking the assert statement, but its import hook (to provide that syntax) cannot coexist with a macro expander, which also needs to install a (different) import hook. Pytest is also fairly complex. -The central functional requirement for whatever would be used for testing `unpythonic` was to be able to easily deal with macro-enabled Python. No hoops to jump through, compared to testing regular Python, in order to be able to test all of `unpythonic` (including `unpythonic.syntax`) in a uniform way. +The central functional requirement for whatever would be used for testing `unpythonic` was to be able to *easily* deal with macro-enabled Python. No hoops to jump through, compared to testing regular Python, in order to be able to test all of `unpythonic` (including `unpythonic.syntax`) in a uniform way. Simple and minimalistic would be a bonus. As of v0.14.3, the whole test framework is about 1.8k SLOC, counting docstrings, comments and blanks; under 700 SLOC if counting only active code lines. Add another 800 SLOC (all) / 200 SLOC (active code lines) for the machinery that implements conditions and restarts. -The framework will likely still evolve a bit as I find more holes in the [UX](https://en.wikipedia.org/wiki/User_experience) - which so far has led to features such as `the[]` and AST value auto-unparsing - but most of the desired functionality is already there. For example, I consider pytest-style implicit fixtures and a central test discovery system as outside the scope of this system. +The framework will likely still evolve a bit as I find more holes in the [UX](https://en.wikipedia.org/wiki/User_experience) - which so far has led to features such as `the[]` and AST value auto-unparsing - but most of the desired functionality is already present and working fine. For example, I consider pytest-style implicit fixtures and a central test discovery system as outside the scope of this framework. It does make the code shorter, but is perhaps slightly too much magic. -It's clear that `unpythonic.test.fixtures` is not going to replace `pytest`, nor does it aim to do so - [any more than Chuck Moore's Forth-based VLSI tools](https://yosefk.com/blog/my-history-with-forth-stack-machines.html) were intended to replace the commercial [VLSI](https://en.wikipedia.org/wiki/Very_Large_Scale_Integration) offerings. +It is clear that `unpythonic.test.fixtures` is not going to replace `pytest`, nor does it aim to do so - [any more than Chuck Moore's Forth-based VLSI tools](https://yosefk.com/blog/my-history-with-forth-stack-machines.html) were intended to replace the commercial [VLSI](https://en.wikipedia.org/wiki/Very_Large_Scale_Integration) offerings. What we have is small, simple, custom-built for its purpose (works well with macro-enabled Python; integrates with conditions and restarts), arguably somewhat pedagogic (demonstrates how to build a test framework in under 700 active SLOC), and importantly, works just fine. From 6e5665ec5cabe576b5dd027483eb9092aca9b8aa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 21 Jun 2021 20:04:26 +0300 Subject: [PATCH 600/832] 0.15.0: update dbg macro doc --- doc/macros.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index d670d7d3..49948bcd 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2257,7 +2257,11 @@ Inspired by [Julia](https://julialang.org/)'s standard-library [`Test` package]( ### `dbg`: debug-print expressions with source code -**Changed in 0.14.2.** The `dbg[]` macro now works in the REPL, too. You can use `mcpyrate.repl.console` (a.k.a. `macropython -i` in the shell) or the IPython extension `mcpyrate.repl.iconsole`. +**Changed in v0.15.0.** *We now use the [`mcpyrate`](https://github.com/Technologicat/mcpyrate/) macro expander instead of `macropy`. Updated the REPL note below.* + +*Also, `dbgprint_expr` is now a dynvar.* + +**Changed in 0.14.2.** *The `dbg[]` macro now works in the REPL, too. You can use `mcpyrate.repl.console` (a.k.a. `macropython -i` in the shell) or the IPython extension `mcpyrate.repl.iconsole`.* [DRY](https://en.wikipedia.org/wiki/Don't_repeat_yourself) out your [qnd](https://en.wiktionary.org/wiki/quick-and-dirty) debug printing code. Both block and expression variants are provided: From a46ca8678f1fa750dec88bf1db2bc320b723f8cf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:11:40 +0300 Subject: [PATCH 601/832] 0.15.0: make `nb` work together with `autoreturn` --- CHANGELOG.md | 1 + doc/design-notes.md | 5 +++++ unpythonic/syntax/nb.py | 20 +++++++++++++------- 3 files changed, 19 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1870d201..a7409125 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -123,6 +123,7 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - The generator instances created by the gfuncs returned by `gmemoize`, `imemoize`, and `fimemoize`, now support the `__len__` and `__getitem__` methods to access the already-yielded, memoized part. Asking for the `len` returns the current length of the memo. For subscripting, both a single `int` index and a slice are accepted. Note that memoized generators do **not** support all of the [`collections.abc.Sequence`](https://docs.python.org/3/library/collections.abc.html) API, because e.g. `__contains__` and `__reversed__` are missing, on purpose. - `fup`/`fupdate`/`ShadowedSequence` can now walk the start of a memoized infinite replacement backwards. (Use `imemoize` on the original iterable, instantiate the generator, and use that generator instance as the replacement.) - When using the `autoreturn` macro, if the item in tail position is a function definition or class definition, return the thing that was defined. + - The `nb` macro now works together with `autoreturn`. - `unpythonic.conditions.signal`, when the signal goes unhandled, now returns the canonized input `condition`, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of `signal`; `error` already uses it to produce a nice-looking error report. - The internal exception types `unpythonic.conditions.InvokeRestart` and `unpythonic.ec.Escape` now inherit from `BaseException`, so that they are not inadvertently caught by `except Exception` handlers. - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. diff --git a/doc/design-notes.md b/doc/design-notes.md index 06370ab8..bef7cc19 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -282,6 +282,11 @@ More on type systems: - `envify` needs to see the output of `lazify` in order to shunt function args into an unpythonic `env` without triggering the implicit forcing. + - `nb` needs to determine whether an expression should be printed. + - It needs to see invocations of testing macros, because those are akin to asserts - while they are technically implemented as expr macros, they expand into function calls into test asserter functions that have no meaningful return value. Thus, just in case the user has requested testing macros to expand first, `nb` needs to expand before anything that may edit function calls, such as `tco` and `autocurry`. + - It needs to see bare expressions (technically, in the AST, an *expression statements* `ast.Expr`). Thus `nb` should expand before `autoreturn`, to treat also expressions that appear in tail position. + - `nb` performs the printing using a passthrough helper function, so that the value that was printed is available as the return value of the print helper, so that `return theprint(value)` works, for co-operation with `autoreturn`. + - With MacroPy, it used to be so that some of the block macros could be comboed as multiple context managers in the same `with` statement (expansion order is then *left-to-right*), whereas some (notably `autocurry` and `namedlambda`) required their own `with` statement. In `mcpyrate`, block macros can be comboed in the same `with` statement (and expansion order is *left-to-right*). - See the relevant [issue report](https://github.com/azazel75/macropy/issues/21) and [PR](https://github.com/azazel75/macropy/pull/22). - When in doubt, you can use a separate `with` statement for each block macro that applies to the same section of code, and nest the blocks. In `mcpyrate`, this is almost equivalent to having the macros invoked in a single `with` statement, in the same order. diff --git a/unpythonic/syntax/nb.py b/unpythonic/syntax/nb.py index 3a0e0863..39ab6c13 100644 --- a/unpythonic/syntax/nb.py +++ b/unpythonic/syntax/nb.py @@ -47,20 +47,26 @@ def nb(tree, *, args, syntax, **kw): def _nb(body, args): p = args[0] if args else q[h[print]] # custom print function hook - with q as newbody: # pragma: no cover, quoted only. + with q as newbody: _ = None - theprint = a[p] + theprint = lambda value: h[_print_and_passthrough](a[p], value) for stmt in body: - # We ignore statements (because no return value), and, - # test[] and related expressions from our test framework. - # Those don't return a value either, and play a role - # similar to the `assert` statement. + # We ignore statements (because no return value), and, test[] and related + # expressions from our test framework. Those have no meaningful return value + # either, and play a role similar to the `assert` statement. if type(stmt) is not Expr or istestmacro(stmt.value): newbody.append(stmt) continue - with q as newstmts: # pragma: no cover, quoted only. + with q as newstmts: _ = a[stmt.value] if _ is not None: theprint(_) newbody.extend(newstmts) return newbody + +# Work together with `autoreturn`. If the implicit print appears in tail position, +# the passthrough will return the value that was printed, so that when `autoreturn` +# transforms the code into `return theprint(_)`, it still works fine. +def _print_and_passthrough(printer, value): + printer(value) + return value From 7ce170ff4b78e9934f586d6e78e14bf5a6a15575 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:12:00 +0300 Subject: [PATCH 602/832] 0.15.0: add nb macro placement into xmas tree combo --- doc/macros.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index 49948bcd..f74a3c50 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2377,7 +2377,7 @@ As an example of a specific technical reason, the `tco` macro skips already expa The **AST edits** performed by the block macros are designed to run in the following order (leftmost first): ``` -prefix > autoreturn, quicklambda > multilambda > continuations or tco > ... +prefix > nb > autoreturn, quicklambda > multilambda > continuations or tco > ... ... > autocurry > namedlambda, autoref > lazify > envify ``` From 4563276372b3ea6d7e3024e02ef3dab1bccbcf27 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:12:21 +0300 Subject: [PATCH 603/832] update stats --- doc/macros.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index f74a3c50..9b86763c 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2238,7 +2238,9 @@ Also, in my opinion, `unittest` is overly verbose to use; automated tests are al The central functional requirement for whatever would be used for testing `unpythonic` was to be able to *easily* deal with macro-enabled Python. No hoops to jump through, compared to testing regular Python, in order to be able to test all of `unpythonic` (including `unpythonic.syntax`) in a uniform way. -Simple and minimalistic would be a bonus. As of v0.14.3, the whole test framework is about 1.8k SLOC, counting docstrings, comments and blanks; under 700 SLOC if counting only active code lines. Add another 800 SLOC (all) / 200 SLOC (active code lines) for the machinery that implements conditions and restarts. +Also, if I was going to build my own framework, it would be nice for it to work seamlessly with code that uses conditions and restarts - since those are part of `unpythonic`, but not standard Python. + +Simple and minimalistic would be a bonus. As of v0.15.0, the whole test framework is about 1.8k SLOC, counting docstrings, comments and blanks; under 700 SLOC if counting only active code lines. Add another 1k SLOC (all) / 200 SLOC (active code lines) for the machinery that implements conditions and restarts. The framework will likely still evolve a bit as I find more holes in the [UX](https://en.wikipedia.org/wiki/User_experience) - which so far has led to features such as `the[]` and AST value auto-unparsing - but most of the desired functionality is already present and working fine. For example, I consider pytest-style implicit fixtures and a central test discovery system as outside the scope of this framework. It does make the code shorter, but is perhaps slightly too much magic. From 2c30a6981aa1e24f5ab423b26d3d869f6905fd4b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:12:41 +0300 Subject: [PATCH 604/832] wording fixes for macro docs --- doc/macros.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 9b86763c..1b6147c6 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2369,7 +2369,7 @@ We have taken into account that: [The dialect examples](dialects.md) use this ordering. -For simplicity, **the block macros make no attempt to prevent invalid combos**, unless there is a specific technical reason to do that for some particular combination. Be careful; e.g. don't nest several `with tco` blocks (lexically), that won't work. +For simplicity, **the block macros make no attempt to prevent invalid combos**, unless there is a specific technical reason to do that for some particular combination. Be careful; e.g. do not nest several `with tco` blocks (lexically), that will not work. As an example of a specific technical reason, the `tco` macro skips already expanded `with continuations` blocks lexically contained within the `with tco`. This allows the [Lispython dialect](dialects/lispython.md) to support `continuations`. @@ -2470,7 +2470,7 @@ Tested with `anaconda-mode`. #### How to use (for Emacs beginners) -If you use the [Spacemacs](http://spacemacs.org/) kit, the right place to insert the snippet is into the function `dotspacemacs/user-config`. Here's [my spacemacs.d](https://github.com/Technologicat/spacemacs.d/) for reference; the snippet is in `prettify-symbols-config.el`, and it's invoked from `dotspacemacs/user-config` in `init.el`. +If you use the [Spacemacs](http://spacemacs.org/) kit, the right place to insert the snippet is into the function `dotspacemacs/user-config`. Here's [my spacemacs.d](https://github.com/Technologicat/spacemacs.d/) for reference; the snippet is in `prettify-symbols-config.el`, and it is invoked from `dotspacemacs/user-config` in `init.el`. In a basic Emacs setup, the snippet goes into the `~/.emacs` startup file, or if you have an `.emacs.d/` directory, then into `~/.emacs.d/init.el`. From 5c5b682140a82f73156735158711e5db48452a88 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:13:33 +0300 Subject: [PATCH 605/832] 0.15.0 release: remove "coming soon" notice --- README.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/README.md b/README.md index 07b5ad64..57d5ef54 100644 --- a/README.md +++ b/README.md @@ -9,17 +9,6 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f *Some hypertext features of this README, such as local links to detailed documentation, and expandable example highlights, are not supported when viewed on PyPI; [view on GitHub](https://github.com/Technologicat/unpythonic) to have those work properly.* -### New version soon! - -**As of May 2021, `unpythonic` 0.15 is Coming Soon™.** - -As of [3b5e5af](https://github.com/Technologicat/unpythonic/commit/3b5e5aff3ba3bd758151b7bf5aa5f2abb07cd82f), the code itself is in a releasable state, and it is already in `master`. All that remains is an extensive documentation review. The changelog is known to be up to date, but something may still need an update in all the other parts of documentation. - -The new version requires Python 3.6 or above, and optionally the [`mcpyrate`](https://github.com/Technologicat/mcpyrate) macro expander. Python 3.4 and 3.5, and the MacroPy macro expander, are no longer supported by `unpythonic`. - -The release will be numbered **0.15.0**, even though the codebase is mostly stable at this point, and we have already adhered to [semantic versioning](https://semver.org/) since 2019 (albeit with a leading zero). The reason is that the next major version has been known under this development version number for such a long time that it makes no sense to renumber it now. - - ### Dependencies None required. From 6eddb3c0e4b03e7349b05400f33e2f834e16f94a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:16:24 +0300 Subject: [PATCH 606/832] Oops, forgot to mention that docs have been updated. --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a7409125..e94e6986 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -129,6 +129,8 @@ The same applies if you need the macro parts of `unpythonic` (i.e. import anythi - The modules `unpythonic.dispatch` and `unpythonic.typecheck`, which provide the `@generic` and `@typed` decorators and the `isoftype` function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of `unpythonic`. - CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7). - CI: Test coverage improved to 94%. + - Full update pass for the user manual written in Markdown. + - Things added or changed in 0.14.2 and later are still mentioned as such, and have not necessarily been folded into the main text. But everything should be at least up to date now. **Breaking changes**: From df042c0c822a58a7a15f503460d3f191d82d09c9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:17:30 +0300 Subject: [PATCH 607/832] oops, mention release date in changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e94e6986..288d4a64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -**0.15.0** (in progress; updated 19 May 2021) - *"We say 'howdy' around these parts"* edition: +**0.15.0** (22 June 2021) - *"We say 'howdy' around these parts"* edition: Beside introducing **dialects** (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This introduces some breaking changes, so we have also taken the opportunity to apply any such that were previously scheduled. From 51429647421c2c474c23d7d6971ec166d864f443 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 11:19:39 +0300 Subject: [PATCH 608/832] pre-emptive version bump --- CHANGELOG.md | 7 +++++++ unpythonic/__init__.py | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 288d4a64..974f7911 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +**0.15.1** (in progress) + +*Pre-emptive version bump. No user-visible changes yet.* + +--- + + **0.15.0** (22 June 2021) - *"We say 'howdy' around these parts"* edition: Beside introducing **dialects** (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure `unpythonic` keeps working for the next few years. This introduces some breaking changes, so we have also taken the opportunity to apply any such that were previously scheduled. diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 5d4a8709..21f0920e 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.15.0' +__version__ = '0.15.1' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From 43792e2bf0e9a8eb102f7131fa58d83d8f6065a1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 12:42:02 +0300 Subject: [PATCH 609/832] add links to some useful concepts for programming language design --- doc/readings.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/readings.md b/doc/readings.md index 9be0e931..d22d4cd1 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -220,6 +220,10 @@ The common denominator is programming. Some relate to language design, some to c - [Matthew Might: First-class (run-time) macros and meta-circular evaluation](https://matt.might.net/articles/metacircular-evaluation-and-first-class-run-time-macros/) - *First-class macros are macros that can be bound to variables, passed as arguments and returned from functions. First-class macros expand and evaluate syntax at run-time.* +- Useful concepts for programming language design: + - [Cognitive dimensions of notations](https://en.wikipedia.org/wiki/Cognitive_dimensions_of_notations) + - [System quality attributes](https://en.wikipedia.org/wiki/List_of_system_quality_attributes) + # Python-related FP resources From 8268ba43c3468970e5c277418efd8fe15c3cdae4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 14:14:34 +0300 Subject: [PATCH 610/832] readings: add Shutt 2016: Interpreted programming languages --- doc/readings.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/readings.md b/doc/readings.md index d22d4cd1..75ec095d 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -167,6 +167,7 @@ The common denominator is programming. Some relate to language design, some to c - [Abstractive power (2013)](https://fexpr.blogspot.com/2013/12/abstractive-power.html). - [Where do types come from? (2011)](https://fexpr.blogspot.com/2011/11/where-do-types-come-from.html). - [Continuations and term-rewriting calculi (2014)](https://fexpr.blogspot.com/2014/03/continuations-and-term-rewriting-calculi.html). + - [Interpreted programming languages (2016)](https://fexpr.blogspot.com/2016/08/interpreted-programming-languages.html) - Discussion of Kernel on LtU: [Decomposing lambda - the Kernel language](http://lambda-the-ultimate.org/node/1680). - [Walid Taha 2003: A Gentle Introduction to Multi-stage Programming](https://www.researchgate.net/publication/221024597_A_Gentle_Introduction_to_Multi-stage_Programming) From 27245c152200f037f74d08da7b60f979e5d3b8cf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 14:29:08 +0300 Subject: [PATCH 611/832] fix borked blockquote --- doc/essays.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/essays.md b/doc/essays.md index 011a0a4f..dccafaa0 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -113,7 +113,7 @@ To summarize; as someone already put it, `hoon` offers a glimpse into an alterna I think the perfect place to end this piece is to quote a few lines from the language definition [`hoon.hoon`](https://github.com/cgyarvin/urbit/blob/master/urb/zod/arvo/hoon.hoon), to give a flavor: -`` +``` ++ doos :: sleep until |= hap=path ^- (unit ,@da) (doze:(wink:(vent bud (dink (dint hap))) now 0 (beck ~)) now [hap ~]) @@ -154,7 +154,7 @@ I think the perfect place to end this piece is to quote a few lines from the lan [p.i.mor t.i.q.i.mor t.q.i.mor r.i.mor] [p.yub [[p.i.naf ves:q.yub] t.naf]] -- -`` +``` The Lisp family (particularly the Common Lisp branch) has a reputation for silly terminology, but I think `hoon` deserves the crown. All control structures are punctuation-only ASCII digraphs, and almost every name is a monosyllabic nonsense word. Still, this Lewis-Carroll-esque naming convention of making words mean what you define them to mean makes at least as much sense as the standard naming convention in mathematics, naming theorems after their discoverers! (Or at least, [after someone else](https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy).) From baf80ba60678ba9e9db021a74958a016c726793c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 14:29:41 +0300 Subject: [PATCH 612/832] fix typo At least semantically, it's "phonemic", regardless of which word the original authors used. ;) --- doc/essays.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/essays.md b/doc/essays.md index dccafaa0..696af287 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -158,7 +158,7 @@ I think the perfect place to end this piece is to quote a few lines from the lan The Lisp family (particularly the Common Lisp branch) has a reputation for silly terminology, but I think `hoon` deserves the crown. All control structures are punctuation-only ASCII digraphs, and almost every name is a monosyllabic nonsense word. Still, this Lewis-Carroll-esque naming convention of making words mean what you define them to mean makes at least as much sense as the standard naming convention in mathematics, naming theorems after their discoverers! (Or at least, [after someone else](https://en.wikipedia.org/wiki/Stigler's_law_of_eponymy).) -I actually like the phonetic base, making numbers sound like [*sorreg-namtyv*](https://urbit.org/docs/hoon/hoon-school/nouns/); that is 5 702 400 for the rest of us. And I think I will, quite seriously, adopt the verb *bunt*, meaning *to take the default value of*. That is such a common operation in programming that I find it hard to believe there is no standard abbreviation. I wonder what other discoveries await. +I actually like the phonemic base, making numbers sound like [*sorreg-namtyv*](https://urbit.org/docs/hoon/hoon-school/nouns/); that is 5 702 400 for the rest of us. And I think I will, quite seriously, adopt the verb *bunt*, meaning *to take the default value of*. That is such a common operation in programming that I find it hard to believe there is no standard abbreviation. I wonder what other discoveries await. Finally, in some way I cannot quite put a finger on, to me the style has echoes of [Jorge Luis Borges](https://en.wikipedia.org/wiki/Jorge_Luis_Borges). Maybe it is that the `hoon` source code sounds like something out of [The Library of Babel](https://en.wikipedia.org/wiki/The_Library_of_Babel). The Borgesian flavor seems intentional, too; the company building the Urbit stack, which `hoon` is part of, is itself named *[Tlon](https://en.wikipedia.org/wiki/Tl%C3%B6n%2C_Uqbar%2C_Orbis_Tertius)*. Remaking the world by re-imagining it, indeed. From 370abcbb4f40d4f2ea963334e6c6e0dc98f9bf1e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 14:40:01 +0300 Subject: [PATCH 613/832] add note about as-importing the fn[] macro It's explained in the macro docs, but a local mention is better. --- doc/dialects/lispython.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dialects/lispython.md b/doc/dialects/lispython.md index 4cf6d909..d3ac6f39 100644 --- a/doc/dialects/lispython.md +++ b/doc/dialects/lispython.md @@ -107,7 +107,7 @@ In the `Lispy` variant, that's it - the dialect changes the semantics only. Noth This is the pythonic variant of Lispython, keeping in line with *explicit is better than implicit*. The rule is: *if a name appears in user code, it must be defined explicitly*, as is usual in Python. -Note this implies that you must **explicitly import** the `local[]` macro if you want to declare local variables in a multiple-expression lambda, and the `fn[]` macro if you want to take advantage of the implicit `quicklambda`. Both are available in `unpythonic.syntax`, as usual. +Note this implies that you must **explicitly import** the `local[]` macro if you want to declare local variables in a multiple-expression lambda, and the `fn[]` macro if you want to take advantage of the implicit `quicklambda`. Both are available in `unpythonic.syntax`, as usual. (Note that you can rename the `fn[]` macro with an as-import, and the implicit `quicklambda` will still work.) The point of the implicit `quicklambda` is that all invocations of `fn[]`, if there are any, will expand early, so that other macros that expect lambdas to be in standard Python notation will get exactly that. This includes other macros invoked by the dialect definition, namely `multilambda`, `namedlambda`, and `tco`. From f9e94b7e6811bb3f81edcf8d859cd1e134f8dcc0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 14:46:45 +0300 Subject: [PATCH 614/832] fix comment --- unpythonic/dialects/tests/test_listhell.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/dialects/tests/test_listhell.py b/unpythonic/dialects/tests/test_listhell.py index 6d1f8283..8959c9ad 100644 --- a/unpythonic/dialects/tests/test_listhell.py +++ b/unpythonic/dialects/tests/test_listhell.py @@ -57,7 +57,7 @@ def f(*, a, b): # in case of duplicate name across kws, rightmost wins test[(f, kw(a="hi there"), kw(b="foo"), kw(b="bar")) == (q, "hi there", "bar")] # noqa: F821 - # give *args with unpythonic.fun.apply, like in Lisps: + # give *args with unpythonic.apply, like in Lisps: with testset("starargs with apply()"): lst = [1, 2, 3] def g(*args, **kwargs): From 73917ef57d8888ba1f64900772ed0e043a1b26ec Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 14:51:47 +0300 Subject: [PATCH 615/832] add note that README examples are not the most general ones --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 57d5ef54..a416bd60 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ This depends on the purpose of each feature, as well as ease-of-use consideratio ### Examples -Small, limited-space overview of the overall flavor. There's a lot more that doesn't fit here, especially in the pure-Python feature set. See the [full documentation](doc/features.md) and [unit tests](unpythonic/tests/) for more examples. +Small, limited-space overview of the overall flavor. There is a lot more that does not fit here, especially in the pure-Python feature set. We give here simple examples that are **not** necessarily of the most general form supported by the constructs. See the [full documentation](doc/features.md) and [unit tests](unpythonic/tests/) for more examples. #### Unpythonic in 30 seconds: Pure Python From c8790ff7b1915579fa5cc8df6195ad3cea402558 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 15:04:33 +0300 Subject: [PATCH 616/832] update dialect examples in README Make Pytkell and Listhell examples more easily comparable. --- README.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a416bd60..3397840f 100644 --- a/README.md +++ b/README.md @@ -778,7 +778,8 @@ my_prod = foldl(mul, 1) my_map = lambda f: foldr(compose(cons, f), nil) assert my_sum(range(1, 5)) == 10 assert my_prod(range(1, 5)) == 24 -assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) +double = lambda x: 2 * x +assert my_map(double, (1, 2, 3)) == ll(2, 4, 6) ```
Listhell: Prefix syntax for function calls, and automatic currying. @@ -788,12 +789,17 @@ assert tuple(my_map((lambda x: 2 * x), (1, 2, 3))) == (2, 4, 6) ```python from unpythonic.dialects import dialects, Listhell # noqa: F401 -from unpythonic import foldr, cons, nil, ll +from operator import add, mul +from unpythonic import foldl, foldr, cons, nil, ll (print, "hello from Listhell") -double = lambda x: 2 * x +my_sum = (foldl, add, 0) +my_prod = (foldl, mul, 1) my_map = lambda f: (foldr, (compose, cons, f), nil) +assert (my_sum, range(1, 5)) == 10 +assert (my_prod, range(1, 5)) == 24 +double = lambda x: 2 * x assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ```
From 0022e51aae4b239da4b4abb1846746185caa6c5d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 22 Jun 2021 15:05:40 +0300 Subject: [PATCH 617/832] update Listhell dialect example in README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3397840f..aed27634 100644 --- a/README.md +++ b/README.md @@ -797,8 +797,8 @@ from unpythonic import foldl, foldr, cons, nil, ll my_sum = (foldl, add, 0) my_prod = (foldl, mul, 1) my_map = lambda f: (foldr, (compose, cons, f), nil) -assert (my_sum, range(1, 5)) == 10 -assert (my_prod, range(1, 5)) == 24 +assert (my_sum, (range, 1, 5)) == 10 +assert (my_prod, (range, 1, 5)) == 24 double = lambda x: 2 * x assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ``` From 7d99d50c5d1c3151fb822491167a7008024d97a5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 17 Jul 2021 19:03:19 +0300 Subject: [PATCH 618/832] fix: `triangular` should be public --- unpythonic/mathseq.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/mathseq.py b/unpythonic/mathseq.py index d92aca13..03250e80 100644 --- a/unpythonic/mathseq.py +++ b/unpythonic/mathseq.py @@ -24,7 +24,7 @@ "sround", "strunc", "sfloor", "sceil", "slshift", "srshift", "sand", "sxor", "sor", "cauchyprod", "diagonal_reduce", - "fibonacci", "primes"] + "fibonacci", "triangular", "primes"] from itertools import repeat, takewhile, count from functools import wraps From b85c2a1f65acc4b5121249ba7ea510888dede617 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 1 Dec 2021 13:43:33 +0200 Subject: [PATCH 619/832] fix typo --- doc/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/features.md b/doc/features.md index d3131f76..af688656 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1409,7 +1409,7 @@ assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) #### `memoize` -**Changed in v0.15.0.** *Fix bug: `memoize` is now thread-safe. Even when the same memoized function instance is called concurrently from multiple threads. Exactly one thread will compute the result. If `f` is recursive, the thread that acquired the lock is the one that is allowed to recurse into the memoized `f`.* +**Changed in v0.15.0.** *Fix bug: `memoize` is now thread-safe. Even when the same memoized function instance is called concurrently from multiple threads, exactly one thread will compute the result. If `f` is recursive, the thread that acquired the lock is the one that is allowed to recurse into the memoized `f`.* [*Memoization*](https://en.wikipedia.org/wiki/Memoization) is a functional programming technique, meant to be used with [pure functions](https://en.wikipedia.org/wiki/Pure_function). It caches the return value, so that *for each unique set of arguments*, the original function will be evaluated only once. All arguments must be hashable. From 781b19a279780a35a2adab8842eee5c1fa203605 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 1 Dec 2021 13:43:48 +0200 Subject: [PATCH 620/832] add recipes to quickly list the whole public API only --- doc/troubleshooting.md | 67 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index 4dc742b2..c39a8ad7 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -21,6 +21,7 @@ - [But I did run my program with `macropython`?](#but-i-did-run-my-program-with-macropython) - [I'm hacking a macro inside a module in `unpythonic.syntax`, and my changes don't take?](#im-hacking-a-macro-inside-a-module-in-unpythonicsyntax-and-my-changes-dont-take) - [Both `unpythonic` and library `x` provide language-extension feature `y`. Which is better?](#both-unpythonic-and-library-x-provide-language-extension-feature-y-which-is-better) + - [How to list the whole public API, and only the public API?](#how-to-list-the-whole-public-api-and-only-the-public-api) @@ -92,3 +93,69 @@ The point of having these features in `unpythonic` is integration, and a consist In some cases (e.g. the condition system), our implementation may offer extra features not present in the original library that inspired it. In other cases (e.g. multiple dispatch), the *other* implementation may be better (e.g. runs much faster). + + +### How to list the whole public API, and only the public API? + +In short, use Python's introspection capabilities. There are some subtleties here; below are some ready-made recipes. + +To view **the public API of a given submodule**: + +```python +import sys +print(sys.modules["unpythonic.collections"].__all__) # for example +``` + +If the `__all__` attribute for some submodule is missing, that submodule has no public API. + +For most submodules, you could just + +```python +print(unpythonic.collections.__all__) # for example +``` + +but there are some public API symbols in `unpythonic` that have the same name as a submodule. In these cases, the object overrides the submodule in the top-level namespace of `unpythonic`. So, for example, for `unpythonic.llist`, the second approach fails because `unpythonic.llist` points to a function, not to a module. Therefore, the first approach is preferable, as it always works. + +To view **the whole public API**, grouped by submodule: + +```python +import sys + +import unpythonic + +submodules = [name for name in dir(unpythonic) + if f"unpythonic.{name}" in sys.modules] + +for name in submodules: + module = sys.modules[f"unpythonic.{name}"] + if hasattr(module, "__all__"): # has a public API? + print("=" * 79) + print(f"Public API of 'unpythonic.{name}':") + print(module.__all__) +``` + +Note that even if you examine the API grouped by submodule, `unpythonic` guarantees all of its public API symbols to be present in the top-level namespace, too, so when you actually import the symbols, you can import them from the top-level namespace. (Actually, the macros expect you to do so, to recognize uses of various `unpythonic` constructs when analyzing code.) + +**Do not*** do this to retrieve the submodules: + +```python +import types +submodules_wrong = [name for name in dir(unpythonic) + if issubclass(type(getattr(unpythonic, name)), types.ModuleType)] +``` + +for the same reason as above; in this variant, any submodules that have the same name as an object will be missing from the list. + +To view **the whole public API** available in the top-level namespace: + +```python +import types + +import unpythonic + +non_module_names = [name for name in dir(unpythonic) + if not issubclass(type(getattr(unpythonic, name)), types.ModuleType)] +print(non_module_names) +``` + +Now be very very careful: for the same reason as above, for the correct semantics we must use `issubclass(..., types.ModuleType)`, not `... in sys.modules`. Here we want to list each symbol in the top-level namespace of `unpythonic` that does not point to a module; **including** any objects that override a module in the top-level namespace. From 47570044276dbe83c8664f52ee791816f7c11f5f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 1 Dec 2021 13:47:45 +0200 Subject: [PATCH 621/832] fix markdown typo in public API recipes --- doc/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index c39a8ad7..46688eef 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -136,7 +136,7 @@ for name in submodules: Note that even if you examine the API grouped by submodule, `unpythonic` guarantees all of its public API symbols to be present in the top-level namespace, too, so when you actually import the symbols, you can import them from the top-level namespace. (Actually, the macros expect you to do so, to recognize uses of various `unpythonic` constructs when analyzing code.) -**Do not*** do this to retrieve the submodules: +**Do not** do this to retrieve the submodules: ```python import types From ae7b69bd83e03fdc40e83fc07b9b8cff9177483a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 1 Dec 2021 13:48:24 +0200 Subject: [PATCH 622/832] fix indentation in public API recipes example --- doc/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/troubleshooting.md b/doc/troubleshooting.md index 46688eef..fb0d027e 100644 --- a/doc/troubleshooting.md +++ b/doc/troubleshooting.md @@ -141,7 +141,7 @@ Note that even if you examine the API grouped by submodule, `unpythonic` guarant ```python import types submodules_wrong = [name for name in dir(unpythonic) - if issubclass(type(getattr(unpythonic, name)), types.ModuleType)] + if issubclass(type(getattr(unpythonic, name)), types.ModuleType)] ``` for the same reason as above; in this variant, any submodules that have the same name as an object will be missing from the list. From d19761bc7fbba2bd58de3fab5a530dafc1fef550 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 10:36:15 +0200 Subject: [PATCH 623/832] fix borked import in unpythonic.net.server --- unpythonic/net/server.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpythonic/net/server.py b/unpythonic/net/server.py index e1cd14ec..a62a3a05 100644 --- a/unpythonic/net/server.py +++ b/unpythonic/net/server.py @@ -133,7 +133,8 @@ from code import InteractiveConsole as Console from ..collections import ThreadLocalBox, Shim -from ..misc import async_raise, namelambda +from ..excutil import async_raise +from ..misc import namelambda from ..symbol import sym from .util import ReuseAddrThreadingTCPServer, socketsource From 76af51735554f3e0674e3513042384cc0f9edf86 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 10:36:34 +0200 Subject: [PATCH 624/832] install all intended subpackages --- setup.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 573ce4ff..9fa0e116 100644 --- a/setup.py +++ b/setup.py @@ -66,7 +66,9 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa setup( name="unpythonic", version=version, - packages=["unpythonic", "unpythonic.syntax"], + # `unpythonic.test` is the macro-enabled testing framework, intended for public consumption; + # the unit tests of `unpythonic` itself in `unpythonic.tests` are NOT deployed. + packages=["unpythonic", "unpythonic.syntax", "unpythonic.test", "unpythonic.net"], provides=["unpythonic"], keywords=["functional-programming", "language-extension", "syntactic-macros", "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", From 4af2a70cce1a8ddfa17c00a08d218b766b6cdb14 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 10:42:30 +0200 Subject: [PATCH 625/832] update changelog --- CHANGELOG.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 974f7911..592370cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,11 @@ -**0.15.1** (in progress) +**0.15.1** (in progress, last updated 8 December 2021) + +**Fixed**: + +- The test framework `unpythonic.test.fixtures` is now correctly installed when installing `unpythonic`. See [#81](https://github.com/Technologicat/unpythonic/issues/81). +- The subpackage for live REPL functionality, `unpythonic.net`, is now correctly installed when installing `unpythonic`. +- Fix a broken import that prevented the REPL server `unpythonic.net.server` from starting. This was broken by the move of `async_raise` into `unpythonic.excutil` in 0.15.0. -*Pre-emptive version bump. No user-visible changes yet.* --- From 33823e8d0d5f3cb7f71b65b235cf2eaf65192c6e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 11:00:14 +0200 Subject: [PATCH 626/832] fix wrong macro name in error message --- unpythonic/syntax/prefix.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 7f9a31f0..0f611bea 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -122,8 +122,8 @@ def q(tree, *, syntax, **kw): # noqa: F811 def u(tree, *, syntax, **kw): # noqa: F811 """[syntax, name] Unquote operator. Only meaningful in a tuple inside a prefix block.""" if syntax != "name": - raise SyntaxError("q (unpythonic.syntax.prefix.q) is a name macro only") # pragma: no cover - raise SyntaxError("q (unpythonic.syntax.prefix.q) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander + raise SyntaxError("u (unpythonic.syntax.prefix.u) is a name macro only") # pragma: no cover + raise SyntaxError("u (unpythonic.syntax.prefix.u) is only valid in a tuple inside a `with prefix` block") # pragma: no cover, not meant to hit the expander # TODO: This isn't a perfect solution, because there is no "call" macro kind. # TODO: We currently trigger the error on any appearance of the name `kw` outside a valid context. From ad744eae808e2ee477cd1f732c9df3549dfcf36c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 11:07:04 +0200 Subject: [PATCH 627/832] document limitation --- unpythonic/syntax/prefix.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 0f611bea..54dcc347 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -86,6 +86,9 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 Current limitations: + - The `q`, `u` and `kw` macros cannot be renamed by as-importing; + `with prefix` expects them to have their original names. + - passing ``*args`` and ``**kwargs`` not supported. Workarounds: ``call(...)``; Python's usual function call syntax. From b903c4e8af2b61755158f80e152c44068dd5a979 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 11:07:13 +0200 Subject: [PATCH 628/832] update comment --- unpythonic/syntax/prefix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 54dcc347..cd65bfbf 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -111,7 +111,7 @@ def prefix(tree, *, syntax, **kw): # noqa: F811 # operators compiled away by `prefix`), but the "q[]" we use as a macro in # this module is the quasiquote operator from `mcpyrate.quotes`. # -# This `def` doesn't overwrite the macro `q`, because the `def` runs at run time. +# This `def` doesn't overwrite the `mcpyrate` quasiquote macro `q`, because the `def` runs at run time. # The expander does not try to expand this `q` as a macro, because `def q(...)` # is not a valid macro invocation even when the name `q` has been imported as a macro. @namemacro From 633cb95f5cd98495beac0e416a97630306d39239 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 11:07:17 +0200 Subject: [PATCH 629/832] mark TODO --- unpythonic/syntax/prefix.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index cd65bfbf..671358f0 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -147,6 +147,8 @@ def kw(tree, *, syntax, **kw): # noqa: F811 # -------------------------------------------------------------------------------- def _prefix(block_body): + # TODO: Should change these to query the expander to allow renaming by as-imports. + # TODO: How to do that can be found in the implementation of `quicklambda`. isquote = lambda tree: getname(tree, accept_attr=False) == "q" isunquote = lambda tree: getname(tree, accept_attr=False) == "u" iskwargs = lambda tree: type(tree) is Call and getname(tree.func, accept_attr=False) == "kw" From d76cb4aca6bf1b283a3bc67a951d3d7020386f29 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 8 Dec 2021 11:08:19 +0200 Subject: [PATCH 630/832] update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 592370cc..dbee799d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ - The test framework `unpythonic.test.fixtures` is now correctly installed when installing `unpythonic`. See [#81](https://github.com/Technologicat/unpythonic/issues/81). - The subpackage for live REPL functionality, `unpythonic.net`, is now correctly installed when installing `unpythonic`. - Fix a broken import that prevented the REPL server `unpythonic.net.server` from starting. This was broken by the move of `async_raise` into `unpythonic.excutil` in 0.15.0. +- `unpythonic.syntax.prefix`: Fix wrong macro name in error message of `unpythonic.syntax.prefix.u`. Document in the docstring that the magic operators `q`, `u`, and `kw` (of the `prefix` macro) cannot be renamed by as-importing. --- From 84e44ee4e345f79dfc82d5b8b29685f8b4915874 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 19 Jan 2022 11:15:22 +0200 Subject: [PATCH 631/832] add timeutil module --- CHANGELOG.md | 6 +- unpythonic/__init__.py | 1 + unpythonic/tests/test_timeutil.py | 37 +++++++++ unpythonic/timeutil.py | 125 ++++++++++++++++++++++++++++++ 4 files changed, 168 insertions(+), 1 deletion(-) create mode 100644 unpythonic/tests/test_timeutil.py create mode 100644 unpythonic/timeutil.py diff --git a/CHANGELOG.md b/CHANGELOG.md index dbee799d..bcda69d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,8 @@ -**0.15.1** (in progress, last updated 8 December 2021) +**0.15.1** (in progress, last updated 19 January 2022) + +**New**: + +- New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. **Fixed**: diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 21f0920e..ab14dc60 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -41,6 +41,7 @@ from .slicing import * # noqa: F401, F403 from .symbol import * # noqa: F401, F403 from .tco import * # noqa: F401, F403 +from .timeutil import * # noqa: F401, F403 from .typecheck import * # noqa: F401, F403 # -------------------------------------------------------------------------------- diff --git a/unpythonic/tests/test_timeutil.py b/unpythonic/tests/test_timeutil.py new file mode 100644 index 00000000..ba7574b0 --- /dev/null +++ b/unpythonic/tests/test_timeutil.py @@ -0,0 +1,37 @@ +# -*- coding: utf-8 -*- + +from ..syntax import macros, test # noqa: F401 +from ..test.fixtures import session, testset, returns_normally + +from ..timeutil import seconds_to_human, format_human_time, ETAEstimator + +def runtests(): + with testset("seconds_to_human"): + test[seconds_to_human(30) == (0, 0, 0, 30)] + test[seconds_to_human(30.0) == (0, 0, 0, 30.0)] + test[seconds_to_human(90) == (0, 0, 1, 30)] + test[seconds_to_human(3690) == (0, 1, 1, 30)] + test[seconds_to_human(86400 + 3690) == (1, 1, 1, 30)] + test[seconds_to_human(2 * 86400 + 3690) == (2, 1, 1, 30)] + + with testset("format_human_time"): + test[format_human_time(30) == "30 seconds"] + test[format_human_time(90) == "01:30"] # mm:ss + test[format_human_time(3690) == "01:01:30"] # hh:mm:ss + test[format_human_time(86400 + 3690) == "1 day 01:01:30"] + test[format_human_time(2 * 86400 + 3690) == "2 days 01:01:30"] + + # This is a UI thing so we can't test functionality reliably. Let's just check it doesn't crash. + with testset("ETAEstimator"): + e = ETAEstimator(total=5) + test[returns_normally(e.estimate)] # before the first tick + test[returns_normally(e.elapsed)] + test[returns_normally(e.formatted_eta)] + test[returns_normally(e.tick())] + test[returns_normally(e.estimate)] # after the first tick + test[returns_normally(e.elapsed)] + test[returns_normally(e.formatted_eta)] + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() diff --git a/unpythonic/timeutil.py b/unpythonic/timeutil.py new file mode 100644 index 00000000..abec77cb --- /dev/null +++ b/unpythonic/timeutil.py @@ -0,0 +1,125 @@ +# -*- coding: utf-8 -*- +"""Some additional batteries for time handling.""" + +__all__ = ["seconds_to_human", "format_human_time", + "ETAEstimator"] + +from collections import deque +import time +import typing + +def seconds_to_human(s: typing.Union[float, int]) -> typing.Tuple[int, int, int, float]: + """Convert a number of seconds into (days, hours, minutes, seconds).""" + d = int(s // 86400) + s -= d * 86400 + h = int(s // 3600) + s -= h * 3600 + m = int(s // 60) + s -= m * 60 + return d, h, m, s + + +def format_human_time(s: typing.Union[float, int]) -> str: + """Convert a number of seconds to a human-readable string. + + The representation format switches automatically depending on + how large `s` is. Examples: + + assert format_human_time(30) == "30 seconds" + assert format_human_time(90) == "01:30" # mm:ss + assert format_human_time(3690) == "01:01:30" # hh:mm:ss + assert format_human_time(86400 + 3690) == "1 day 01:01:30" + assert format_human_time(2 * 86400 + 3690) == "2 days 01:01:30" + """ + d, h, m, s = seconds_to_human(s) + + if all(x == 0 for x in (d, h, m)): # under one minute + plural = "s" if int(s) != 1.0 else "" + return f"{int(s):d} second{plural}" + + if d > 0: + plural = "s" if d > 1 else "" + days = f"{d:d} day{plural} " + else: + days = "" + hours = f"{h:02d}:" if (d > 0 or h > 0) else "" + minutes = f"{m:02d}:" + seconds = f"{int(s):02d}" + return f"{days}{hours}{minutes}{seconds}" + + +class ETAEstimator: + """Estimate the time of completion. + + `total`: number of tasks in the whole job, used for estimating + how much work is still needed. + + Stored in `self.total`, which is writable; but note that + if you move the goalposts, the ETA cannot be accurate. + Changing `self.total` is mostly useful if you suddenly + discover that the workload is actually larger or smaller + than what was initially expected, and want the estimate + to reflect this sudden new information. + + `keep_last`: use the timings from at most this many most recently + completed tasks when computing the estimate. + + If not given, keep all. + + If you need it, the number of tasks that have been marked completed + is available in `self.completed`. + """ + def __init__(self, total: int, keep_last: typing.Optional[int] = None): + self.t1 = time.monotonic() # time since last tick + self.t0 = self.t1 # time since beginning + self.total = total # total number of work items + self.completed = 0 # number of completed work items + self.que = deque([], maxlen=keep_last) + + def tick(self) -> None: + """Mark one more task as completed, automatically updating the internal timings cache.""" + self.completed += 1 + t = time.monotonic() + dt = t - self.t1 + self.t1 = t + self.que.append(dt) + + def _estimate(self) -> typing.Optional[float]: + if self.completed == 0: + return None + # TODO: Smoother ETA? + # + # Let us consider the ETA estimation process as downsampling the data + # vector (deque) into an extremely low-resolution version that has just + # one sample. + # + # As we know from signal processing, as a downsampling filter, the + # running average has an abysmal frequency response; so we should + # expect the ETA to fluctuate wildly depending on the smoothness of + # the input data (i.e. the time taken by each task)... which actually + # matches observation. + # + # Maybe we could use a Lanczos downsampling filter to make the ETA + # behave more smoothly? + remaining = self.total - self.completed + dt_avg = sum(self.que) / len(self.que) + return remaining * dt_avg + estimate = property(fget=_estimate, doc="Estimate of time remaining, in seconds. Computed when read; read-only. If no tasks have been marked completed yet, the estimate is `None`.") + + def _elapsed(self) -> float: + return time.monotonic() - self.t0 + elapsed = property(fget=_elapsed, doc="Total elapsed time, in seconds. Computed when read; read-only.") + + def _formatted_eta(self) -> str: + elapsed = self.elapsed + estimate = self.estimate + if estimate: + total = elapsed + estimate + formatted_estimate = format_human_time(estimate) + formatted_total = format_human_time(total) + else: + formatted_estimate = "unknown" + formatted_total = "unknown" + formatted_elapsed = format_human_time(elapsed) + return f"elapsed {formatted_elapsed}, ETA {formatted_estimate}, total {formatted_total}" + formatted_eta = property(fget=_formatted_eta, doc="Human-readable estimate, with elapsed, ETA and remaining time. See `format_human_time` for details of the format used.") From 806dc9020393e9ac3d2fc4381cc89533f3277e51 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 15:47:07 +0200 Subject: [PATCH 632/832] silly comment --- unpythonic/syntax/tailtools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 7f2951f2..47daf1ff 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -787,7 +787,7 @@ class CallCcMarker(ContinuationsMarker): """AST marker denoting a `call_cc[]` invocation.""" -def _continuations(block_body): +def _continuations(block_body): # here be dragons. # This is a very loose pythonification of Paul Graham's continuation-passing # macros in On Lisp, chapter 20. # From 32756b4bdb17584614365d7d9e58eb42b7826a60 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 16:21:31 +0200 Subject: [PATCH 633/832] add get_cc --- CHANGELOG.md | 1 + unpythonic/syntax/tailtools.py | 186 ++++++++++++++++++++++++++++++++- 2 files changed, 186 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bcda69d5..89fce574 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ **New**: - New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. +- Add `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses continuations.) The two work together. See docstring. **Fixed**: diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 47daf1ff..3eb5d48d 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -5,7 +5,7 @@ __all__ = ["autoreturn", "tco", - "continuations", "call_cc"] + "continuations", "call_cc", "get_cc"] from functools import partial @@ -1156,6 +1156,190 @@ def transform(self, tree): # (needed to support continuations in the Lispython dialect, since it applies tco globally.) return ExpandedContinuationsMarker(body=new_block_body) +# TODO: Do we need to account for `_pcc` here? Probably not, since this is defined at the +# TODO: top level of a module, not as a closure inside another function. +@trampolined +def get_cc(*, cc): + """When used together with `call_cc[]`, capture and get the current continuation. + + This convenience function covers the common use case when working with + continuations, when you just want to snapshot the control state into a + local variable. + + In other words, this is what you want 99% of the time when you need `call_cc`. + + Or in yet other words, `get_cc` is the less antisocial little sister of `call_cc` + from an alternate timeline, and in this adventure the two work as a team. + + Usage:: + + with continuations: + ... + def dostuff(): + ... + + k = call_cc[get_cc()] + + # Now `k` is the continuation from this point on. + # You can do whatever you want with it! + # + # To invoke it, `k(k)` to always preserve the meaning + # of `k` in this part of the code. (See below.) + + ... + return k # maybe our caller wants to replay part of us later + + As for how this works, you may have seen the following helper function + in Matthew Might's article on continuations by example: + + (define (current-continuation) + (call/cc (lambda (cc) (cc cc)))) + + The lambda is pretty much `get_cc`. We cannot factor away the `call/cc`, + because our `call_cc` is a macro that arranges for the actual capture to + happen at its use site (and it cannot affect any outer levels of the call + stack). + + + **CAUTION**: + + In `k = call_cc[get_cc()]`, the continuation is automatically assigned to + `k` only during the first run, i.e. (in the example) whenever `dostuff` is + called normally. + + By the rules of `unpythonic.syntax.call_cc`, the continuation function will + have parameters for whatever is on the left-hand side of the assignment; in + this case, there will be one parameter, `k`. + + When you invoke the continuation later, the name `k` inside the continuation + (i.e. in the code below the `call_cc` line) will point to whatever value you + sent into the continuation as its argument. + + To achieve least surprise, in 99% of cases, one should arrange things so that + in the continuation, the name `k` always actually points to the continuation, + no matter whether the code runs normally or via continuation invocation. + + Thus, unless there is a specific reason to do otherwise, the recommended way + to invoke the continuation is `k(k)` (giving it itself as the argument). + + Note this caution applies to any continuation that expects to take itself + as an argument; the `k = call_cc[get_cc()]` pattern is just a convenient + way to create such continuations. + + + **Comparison to Lisps**: + + The `k = call_cc[get_cc()]` pattern was inspired by The One True Way to use + `call/cc` in Lisp dialects that have multi-shot continuations, as well as the + `let/cc` construct in Racket. + + The One True Way is to use a one-argument lambda that is invoked immediately + by the `call/cc`: + + (define dostuff () + ... + (call/cc (lambda (k) + ;; ...now k is the continuation... + ... + k))) ;; return it just for the lulz + + The name `call/cc` (`call-with-current-continuation`) is a misnomer; the + purpose of the construct is not really to call a reusable function defined + somewhere else; used that way, it may seem an esoteric feature primarily + intended to confuse programmers. Instead, when combined with a lexical closure + as above, it exposes the continuation as a local variable - which is a + clean and useful technique for a variety of purposes (custom escapes, + generators, backtracking, ...). + + Racket abstracts this pattern into `let/cc`, which communicates the intent + more clearly: + + (define dostuff () + ... + (let/cc k + ;; ...now k is the continuation... + ... + k)) ;; return it just for the lulz + + (Racket has no `return` keyword - it does not need one, since you can + create one using `(let/cc return ...)`, scoping it to whichever block + you want.) + + In the Lisp examples above, `k` is the continuation starting with the next + expression after the `call/cc` or `let/cc` block (expression). + + In our `k = call_cc[get_cc()]` pattern, `k` is the rest of the function body + after the statement `k = call_cc[get_cc()]`. + + So in Lisps, invoking `k` inside the block performs an exit (think of a Python + `return` from that block), whereas in our implementation, doing so loops back + to the next statement just after the `call_cc`. + + There is a similarity between our `get_cc` and something that is possible + in Lisps: our continuation starts from the next statement that runs after + `k = call_cc[get_cc()]`. This is exactly how the `(current-continuation)` + function, mentioned at the beginning, works. + + + **Why `get_cc`?**: + + In Python, a function using all the features of the language cannot be + defined in an expression, so in most cases the (un)pythonic `call_cc` + must indeed call a function defined somewhere else. + + The question becomes, what should this function be? + + 1. To be useful at all, it should make it easier to program with continuations, + over arbitrary use of `call_cc`. + + 2. To promote a standard usage pattern, the function should be as general as + possible, so that we only ever need one. + + 3. For least surprise, the function should do as little as possible; + particularly, no side effects. + + 4. For familiarity, we should stay as close to The One True Way pattern as + possible. In the pattern, the lambda converts the call into a let-like + construct, which pythonifies into an assignment, `k = call_cc[...]`. + + 5. The only reason to use `call_cc` is when you want to get the continuation. + + The obvious solution is a function that just passes the continuation as an + argument into that very same continuation, without any side effects; this is + exactly what `get_cc` does. Thus we get the pattern `k = call_cc[get_cc()]`, + which arguably does exactly what it says on the tin. + """ + # If `get_cc` was defined inside a `with continuations` block, the definition + # could be just: + # + # def get_cc(*, cc): + # return cc + # + # because that means "send the value `cc` into the current continuation" + # (i.e. "escape into the current continuation with the value `cc`"), and + # `cc` is the current continuation. For a more detailed analysis in Scheme: + # + # https://stackoverflow.com/questions/57663699/returning-continuations-from-call-cc + # + # Since `get_cc` is not defined inside a `with continuations` block (so that + # we can easily provide it in the same module that defines the continuation + # machinery, without using multiphase compilation), we make the actual definition + # essentially as a handcrafted macro expansion. + # + # So when returning, we are expected to tail-call (i.e. TCO-jump into) the + # continuation function that was given to us, with our return value(s) becoming + # its argument(s). + # + # Below the first `cc` is the continuation function, and the second `cc` + # is the return value that we are sending into it. + # + # One often sees the pattern `(cc cc)` also in Lisps; for example, see + # the function `(current-continuation)` in Matthew Might's article on + # continuations by example: + # http://matt.might.net/articles/programming-with-continuations--exceptions-backtracking-search-threads-generators-coroutines/ + # + return jump(cc, cc) + # ----------------------------------------------------------------------------- def _tco_transform_def(tree, *, preproc_cb): From c9355aabc5a11ce057b7c289bcb2efb038eb38be Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 16:29:13 +0200 Subject: [PATCH 634/832] add test for get_cc --- unpythonic/syntax/tests/test_conts.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 34238797..6f7161f9 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -5,6 +5,7 @@ from ...test.fixtures import session, testset, returns_normally from ...syntax import macros, continuations, call_cc, multilambda, autoreturn, autocurry, let # noqa: F401, F811 +from ...syntax import get_cc from ...ec import call_ec from ...fploop import looped @@ -652,6 +653,29 @@ def s(loop, acc=0): test[tuple(out) == 2 * tuple(range(11))] test[s == 10] + # As of 0.15.1, the preferred way of working with continuations is as follows. + # + # The pattern `k = call_cc[get_cc()]` covers the 99% common case where you + # just want to snapshot and save the control state into a local variable. + # + # See docstring of `unpythonic.syntax.get_cc` for more. It's a regular function + # that works together with the `call_cc` macro. + with testset("get_cc, the less antisocial little sister of call_cc"): + with continuations: + def append_stuff_to(lst): + lst.append("one") + k = call_cc[get_cc()] + print(k) + lst.append("two") + return k + + lst = [] + k = append_stuff_to(lst) + test[lst == ["one", "two"]] + # invoke the continuation + k(k) # send `k` back in as argument so it the continuation sees it as its local `k` + test[lst == ["one", "two", "two"]] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 51d7591e9e45d1d89cd468aaada8e3a318de0a24 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 16:57:37 +0200 Subject: [PATCH 635/832] get_cc now makes also parametric continuations --- unpythonic/syntax/tailtools.py | 29 +++++++++++++++++++++++++-- unpythonic/syntax/tests/test_conts.py | 22 ++++++++++++++++++++ 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 3eb5d48d..5262cc21 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -1159,7 +1159,7 @@ def transform(self, tree): # TODO: Do we need to account for `_pcc` here? Probably not, since this is defined at the # TODO: top level of a module, not as a closure inside another function. @trampolined -def get_cc(*, cc): +def get_cc(*args, cc): """When used together with `call_cc[]`, capture and get the current continuation. This convenience function covers the common use case when working with @@ -1171,6 +1171,8 @@ def get_cc(*, cc): Or in yet other words, `get_cc` is the less antisocial little sister of `call_cc` from an alternate timeline, and in this adventure the two work as a team. + The `*args`, if any, are passed through. + Usage:: with continuations: @@ -1189,6 +1191,26 @@ def dostuff(): ... return k # maybe our caller wants to replay part of us later + Any positional `*args` are passed through, so that you can also make a + continuation that takes additional arguments:: + + def domorestuff(): + ... + + k, x1, x2 = call_cc[get_cc(1, 2)] # -> k=cc, x1=1, x2=2 + + print(x1, x2) + return k + + k = domorestuff() + k(3, 4) + k(x1=3, x2=4) # same thing + + Important: in the `get_cc` call, the initial values for the additional + arguments, if any, must be passed positionally, due to `call_cc` syntax + limitations. However, when invoking the continuation, they can be passed + any way you want. + As for how this works, you may have seen the following helper function in Matthew Might's article on continuations by example: @@ -1333,12 +1355,15 @@ def dostuff(): # Below the first `cc` is the continuation function, and the second `cc` # is the return value that we are sending into it. # + # The `*args` are a passthrough so that e.g. `k, a, b = call_cc[get_cc(1, 2)]`; + # allows you to pass parameters into the continuation later. + # # One often sees the pattern `(cc cc)` also in Lisps; for example, see # the function `(current-continuation)` in Matthew Might's article on # continuations by example: # http://matt.might.net/articles/programming-with-continuations--exceptions-backtracking-search-threads-generators-coroutines/ # - return jump(cc, cc) + return jump(cc, cc, *args) # ----------------------------------------------------------------------------- diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 6f7161f9..e32ef358 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -676,6 +676,28 @@ def append_stuff_to(lst): k(k) # send `k` back in as argument so it the continuation sees it as its local `k` test[lst == ["one", "two", "two"]] + # If your continuation needs to take arguments, `get_cc` can also make a parametric continuation: + with testset("get_cc with parametric continuation"): + with continuations: + def append_stuff_to(lst): + # Important: in the `get_cc` call, the initial values for + # the additional arguments, if any, must be passed positionally, + # due to `call_cc` syntax limitations. + k, x1, x2 = call_cc[get_cc(1, 2)] + lst.extend([x1, x2]) + return k + + lst = [] + k = append_stuff_to(lst) + test[lst == [1, 2]] + # invoke the continuation, sending both `k` and our additional arguments. + k(k, 3, 4) + test[lst == [1, 2, 3, 4]] + # When invoking the continuation, the additional arguments can be passed + # in any way allowed by Python. + k(k, x1=5, x2=6) + test[lst == [1, 2, 3, 4, 5, 6]] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 12dde26ff8cdd89a87206b14ac59a5c6584d7477 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 17:27:45 +0200 Subject: [PATCH 636/832] tag continuation functions --- CHANGELOG.md | 1 + unpythonic/syntax/tailtools.py | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 89fce574..2e733931 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ - New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. - Add `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses continuations.) The two work together. See docstring. +- Tag continuation closures (generated by the `with continuations`), for introspection. A continuation closure now has the attribute `is_continuation` (and redundantly, it is set to `True`). **Fixed**: diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 5262cc21..602fa8b9 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -1035,8 +1035,12 @@ def prepare_call(tree): decorator_list=[], # patched later by transform_def returns=None) # return annotation not used here - # in the output stmts, define the continuation function... - newstmts = [funcdef] + # 0.15.1: tag the continuation function as a continuation, for introspection. + setcontflag = Assign(targets=[q[n[f"{contname}.is_continuation"]]], + value=q[True]) + + # in the output stmts, define the continuation function, set its is-continuation flag, ... + newstmts = [funcdef, setcontflag] if owner: # ...and tail-call it (if currently inside a def) def jumpify(tree): tree.args = [tree.func] + tree.args From f6fc0ebf3a9ab449d1afe1b955ac351e24083304 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 17:27:55 +0200 Subject: [PATCH 637/832] add remark --- unpythonic/collections.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpythonic/collections.py b/unpythonic/collections.py index 748ab5cf..280f8704 100644 --- a/unpythonic/collections.py +++ b/unpythonic/collections.py @@ -290,6 +290,9 @@ class Some: In a way, `Some` is a relative of `box`: it's an **immutable** single-item container. It supports `.get` and `unbox`, but no `<<` or `.set`. + + It is also the logical opposite of a bare `None`, also syntactically: + `Some(...) is not None`. """ def __init__(self, x=None): self.x = x From a3b4e24fe8e2b674e107c36e002d653562a82703 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Jan 2022 17:30:41 +0200 Subject: [PATCH 638/832] add lispy example of passing continuation arguments --- unpythonic/syntax/tests/test_conts.py | 42 +++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index e32ef358..b8326002 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -698,6 +698,48 @@ def append_stuff_to(lst): k(k, x1=5, x2=6) test[lst == [1, 2, 3, 4, 5, 6]] + # On the other hand, if inside the continuation, you don't need a reference + # to the continuation itself, you can abuse `k` to pass an arbitrary object. + # + # Then in the continuation, you can ask `k` whether it is a continuation + # (first run, return value of `get_cc()`), or something else (second and + # further runs, a value sent in via the continuation). + # + # This is the lispy solution. Whether this or the previous example is more pythonic + # is left as an exercise to the reader. + # + # Just, for simplicity, don't send in a continuation function (without at least + # wrapping it in a box), to avoid the need to detect whether `k` is *the* + # continuation that should have been returned by *this* `get_cc`. You could + # look at the function name, but there is no 100% reliable way. If you need to + # send in a continuation function, it is much simpler to just to box it (in a + # read-only `Some` container, even), to make it explicit that it's intended as data. + # + with testset("get_cc lispy style"): + with continuations: + def append_stuff_to(lst): + ... # could do something useful here (otherwise, why make a continuation?) + k = call_cc[get_cc()] + + # <-- the resume point is here, with `k` set to "the return value of `call_cc`" + + # in 0.15.1+, continuation functions created by the macro are tagged as `is_continuation`. + # TODO: add an interface function to query it + if hasattr(k, "is_continuation"): # got the continuation; just return it + return k + + # invoked via continuation, now `k` is input for us instead of a continuation + x1, x2 = k + lst.extend([x1, x2]) + return None # k is not the continuation now + + lst = [] + k = append_stuff_to(lst) + k([1, 2]) # whatever we send in becomes the local `k` in the continuation. + test[lst == [1, 2]] + k([3, 4]) + test[lst == [1, 2, 3, 4]] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 5925184608f5025e1a53e6fae80a6bd15500290d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 Jan 2022 01:01:40 +0200 Subject: [PATCH 639/832] add `iscontinuation` to go with the tagging --- CHANGELOG.md | 2 +- unpythonic/syntax/tailtools.py | 11 ++++++++++- unpythonic/syntax/tests/test_conts.py | 8 ++++---- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e733931..541fd94e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ - New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. - Add `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses continuations.) The two work together. See docstring. -- Tag continuation closures (generated by the `with continuations`), for introspection. A continuation closure now has the attribute `is_continuation` (and redundantly, it is set to `True`). +- Tag continuation closures (generated by the `with continuations`), for introspection. To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. (The information is stored as an attribute on the function object; so be careful if applying decorators manually to the continuation function.) **Fixed**: diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 602fa8b9..e0bf61f5 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -5,7 +5,7 @@ __all__ = ["autoreturn", "tco", - "continuations", "call_cc", "get_cc"] + "continuations", "call_cc", "get_cc", "iscontinuation"] from functools import partial @@ -1160,6 +1160,15 @@ def transform(self, tree): # (needed to support continuations in the Lispython dialect, since it applies tco globally.) return ExpandedContinuationsMarker(body=new_block_body) +def iscontinuation(x): + """Return whether the object `x` is a continuation function. + + This function can be used for inspection at run time. + + Continuation functions are created by `call_cc[...]` in a `with continuations` block. + """ + return callable(x) and hasattr(x, "is_continuation") and x.is_continuation + # TODO: Do we need to account for `_pcc` here? Probably not, since this is defined at the # TODO: top level of a module, not as a closure inside another function. @trampolined diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index b8326002..05deb54c 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -5,7 +5,7 @@ from ...test.fixtures import session, testset, returns_normally from ...syntax import macros, continuations, call_cc, multilambda, autoreturn, autocurry, let # noqa: F401, F811 -from ...syntax import get_cc +from ...syntax import get_cc, iscontinuation from ...ec import call_ec from ...fploop import looped @@ -723,9 +723,9 @@ def append_stuff_to(lst): # <-- the resume point is here, with `k` set to "the return value of `call_cc`" - # in 0.15.1+, continuation functions created by the macro are tagged as `is_continuation`. - # TODO: add an interface function to query it - if hasattr(k, "is_continuation"): # got the continuation; just return it + # in 0.15.1+, continuation functions created by the macro are tagged. + # TODO: multi-shot generator example using get_cc + if iscontinuation(k): # got the continuation; just return it return k # invoked via continuation, now `k` is input for us instead of a continuation From 02f8ce413d965f61006e0a1ab7c392b563e664ad Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 Jan 2022 01:01:58 +0200 Subject: [PATCH 640/832] remove silly print from test --- unpythonic/syntax/tests/test_conts.py | 1 - 1 file changed, 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 05deb54c..bc108000 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -665,7 +665,6 @@ def s(loop, acc=0): def append_stuff_to(lst): lst.append("one") k = call_cc[get_cc()] - print(k) lst.append("two") return k From 1d8d848ba3ae8ca0de0de74e53a6affeae112e7a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 Jan 2022 01:12:26 +0200 Subject: [PATCH 641/832] update comments in lispy example --- unpythonic/syntax/tests/test_conts.py | 47 ++++++++++++++++----------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index bc108000..62ef5dd3 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -697,44 +697,53 @@ def append_stuff_to(lst): k(k, x1=5, x2=6) test[lst == [1, 2, 3, 4, 5, 6]] - # On the other hand, if inside the continuation, you don't need a reference - # to the continuation itself, you can abuse `k` to pass an arbitrary object. + # You can also abuse `k` to pass an arbitrary object, if inside the + # continuation, you don't need a reference to the continuation itself. + # This is the lispy solution. # - # Then in the continuation, you can ask `k` whether it is a continuation + # Then you can `iscontinuation(k)` to check whether it is a continuation # (first run, return value of `get_cc()`), or something else (second and # further runs, a value sent in via the continuation). # - # This is the lispy solution. Whether this or the previous example is more pythonic - # is left as an exercise to the reader. - # - # Just, for simplicity, don't send in a continuation function (without at least - # wrapping it in a box), to avoid the need to detect whether `k` is *the* - # continuation that should have been returned by *this* `get_cc`. You could - # look at the function name, but there is no 100% reliable way. If you need to - # send in a continuation function, it is much simpler to just to box it (in a - # read-only `Some` container, even), to make it explicit that it's intended as data. + # Whether this or the previous example is more pythonic is left as an + # exercise to the reader. # + # In this solution, be careful, if you need to send in a continuation + # function for some reason. It is impossible to be 100% sure whether `k` + # is *the* continuation that should have been returned by *this* `get_cc`. + # If you need to send in a continuation function, box it (in a read-only + # `Some` box, even), to make it explicit that it's intended as data. with testset("get_cc lispy style"): with continuations: + # The pattern + # + # k = call_cc[get_cc()] + # if iscontinuation(k): + # return k + # + # creates a multi-shot resume point: def append_stuff_to(lst): ... # could do something useful here (otherwise, why make a continuation?) + k = call_cc[get_cc()] - # <-- the resume point is here, with `k` set to "the return value of `call_cc`" + # <-- the resume point is here, with `k` set to "the return value of the `call_cc`", + # i.e. the continuation during the first run, and whatever was sent in during later runs. - # in 0.15.1+, continuation functions created by the macro are tagged. - # TODO: multi-shot generator example using get_cc - if iscontinuation(k): # got the continuation; just return it + # In 0.15.1+, continuation functions created by the `call_cc[...]` macro are + # tagged, and can be detected using `unpythonic.syntax.iscontinuation`, which + # is a regular function: + if iscontinuation(k): # first run; just return the continuation return k - # invoked via continuation, now `k` is input for us instead of a continuation + # invoked via continuation, now `k` is input data instead of a continuation x1, x2 = k lst.extend([x1, x2]) - return None # k is not the continuation now + return None lst = [] k = append_stuff_to(lst) - k([1, 2]) # whatever we send in becomes the local `k` in the continuation. + k([1, 2]) # whatever object we send in becomes the local `k` in the continuation. test[lst == [1, 2]] k([3, 4]) test[lst == [1, 2, 3, 4]] From ed42f8612770de173e64c7d041adfd358ff1a07a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 Jan 2022 08:59:08 +0200 Subject: [PATCH 642/832] wording --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 541fd94e..446848a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,9 @@ - New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. - Add `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses continuations.) The two work together. See docstring. -- Tag continuation closures (generated by the `with continuations`), for introspection. To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. (The information is stored as an attribute on the function object; so be careful if applying decorators manually to the continuation function.) +- Tag continuation closures (generated by the `with continuations` macro), for introspection. + - To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. + - The information is stored as an attribute on the function object; so be careful if applying decorators manually to the continuation function. **Fixed**: From 6ba830916d4637ff295a68e69b479e19274f519c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 26 Jan 2022 13:37:22 +0200 Subject: [PATCH 643/832] update TODO comment --- unpythonic/syntax/tailtools.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index e0bf61f5..dd31b964 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -901,8 +901,9 @@ def split_at_callcc(body): # TODO: To support Python's scoping properly in assignments after the `call_cc`, # TODO: we have to scan `before` for assignments to local variables (stopping at # TODO: scope boundaries; use `unpythonic.syntax.scoping.get_names_in_store_context`, - # TODO: and declare those variables `nonlocal` in `after`. This way the binding - # TODO: will be shared between the original context and the continuation. + # TODO: and declare those variables (plus any variables already declared as `nonlocal` + # TODO: in `before`) as `nonlocal` in `after`. This way the binding will be shared + # TODO: between the original context and the continuation. Also, propagate `global`. # See Politz et al 2013 (the "full monty" paper), section 4.2. return before, stmt, after before.append(stmt) From d370a89dcc282bd57405ca4437e372a22f1cf210 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 Jan 2022 11:37:20 +0200 Subject: [PATCH 644/832] preserve source location info of dialect-import in example dialects Any code coming in from the template will be marked as if it came from the line containing the dialect-import statement. (Invoke the `mcpyrate.debug.StepExpansion` dialect before any other dialects to see the line numbers.) Preserving the source location info requires `mcpyrate` 3.6.0 or later. This will also run on earlier versions, without preserving the source location info, just like before; then it will look like all dialect template code came from the beginning of the unexpanded user code. --- unpythonic/dialects/lispython.py | 20 ++++++++++++++++++-- unpythonic/dialects/listhell.py | 10 +++++++++- unpythonic/dialects/pytkell.py | 10 +++++++++- 3 files changed, 36 insertions(+), 4 deletions(-) diff --git a/unpythonic/dialects/lispython.py b/unpythonic/dialects/lispython.py index 13dbc869..28265c5a 100644 --- a/unpythonic/dialects/lispython.py +++ b/unpythonic/dialects/lispython.py @@ -39,7 +39,15 @@ def transform_ast(self, tree): # tree is an ast.Module from unpythonic import cons, car, cdr, ll, llist, nil, prod, dyn, Values # noqa: F401, F811 with autoreturn, quicklambda, multilambda, namedlambda, tco: __paste_here__ # noqa: F821, just a splicing marker. - tree.body = splice_dialect(tree.body, template, "__paste_here__") + + # Beginning with 3.6.0, `mcpyrate` makes available the source location info + # of the dialect-import that imported this dialect. + if hasattr(self, "lineno"): # mcpyrate 3.6.0+ + tree.body = splice_dialect(tree.body, template, "__paste_here__", + lineno=self.lineno, col_offset=self.col_offset) + else: + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree @@ -68,5 +76,13 @@ def transform_ast(self, tree): # tree is an ast.Module # of the template. with autoreturn, quicklambda, multilambda, namedlambda, tco: __paste_here__ # noqa: F821, just a splicing marker. - tree.body = splice_dialect(tree.body, template, "__paste_here__") + + # Beginning with 3.6.0, `mcpyrate` makes available the source location info + # of the dialect-import that imported this dialect. + if hasattr(self, "lineno"): # mcpyrate 3.6.0+ + tree.body = splice_dialect(tree.body, template, "__paste_here__", + lineno=self.lineno, col_offset=self.col_offset) + else: + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree diff --git a/unpythonic/dialects/listhell.py b/unpythonic/dialects/listhell.py index 35ece7d4..9d1defb6 100644 --- a/unpythonic/dialects/listhell.py +++ b/unpythonic/dialects/listhell.py @@ -23,5 +23,13 @@ def transform_ast(self, tree): # tree is an ast.Module from unpythonic import composerc as compose # compose from Right, Currying # noqa: F401 with prefix, autocurry: __paste_here__ # noqa: F821, just a splicing marker. - tree.body = splice_dialect(tree.body, template, "__paste_here__") + + # Beginning with 3.6.0, `mcpyrate` makes available the source location info + # of the dialect-import that imported this dialect. + if hasattr(self, "lineno"): # mcpyrate 3.6.0+ + tree.body = splice_dialect(tree.body, template, "__paste_here__", + lineno=self.lineno, col_offset=self.col_offset) + else: + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree diff --git a/unpythonic/dialects/pytkell.py b/unpythonic/dialects/pytkell.py index d676388a..f3780794 100644 --- a/unpythonic/dialects/pytkell.py +++ b/unpythonic/dialects/pytkell.py @@ -39,5 +39,13 @@ def transform_ast(self, tree): # tree is an ast.Module from unpythonic import cons, car, cdr, ll, llist, nil # noqa: F401 with lazify, autocurry: __paste_here__ # noqa: F821, just a splicing marker. - tree.body = splice_dialect(tree.body, template, "__paste_here__") + + # Beginning with 3.6.0, `mcpyrate` makes available the source location info + # of the dialect-import that imported this dialect. + if hasattr(self, "lineno"): # mcpyrate 3.6.0+ + tree.body = splice_dialect(tree.body, template, "__paste_here__", + lineno=self.lineno, col_offset=self.col_offset) + else: + tree.body = splice_dialect(tree.body, template, "__paste_here__") + return tree From e51076c815ae2c5b0a27f11d6bc0fe4a8d8558b3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 Jan 2022 14:19:00 +0200 Subject: [PATCH 645/832] update CHANGELOG --- CHANGELOG.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 446848a6..ac3f21e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,9 @@ -**0.15.1** (in progress, last updated 19 January 2022) +**0.15.1** (in progress, last updated 27 January 2022) **New**: - New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. -- Add `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses continuations.) The two work together. See docstring. +- Add function `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses multi-shot continuations.) The two work together. See docstring. - Tag continuation closures (generated by the `with continuations` macro), for introspection. - To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. - The information is stored as an attribute on the function object; so be careful if applying decorators manually to the continuation function. @@ -14,6 +14,9 @@ - The subpackage for live REPL functionality, `unpythonic.net`, is now correctly installed when installing `unpythonic`. - Fix a broken import that prevented the REPL server `unpythonic.net.server` from starting. This was broken by the move of `async_raise` into `unpythonic.excutil` in 0.15.0. - `unpythonic.syntax.prefix`: Fix wrong macro name in error message of `unpythonic.syntax.prefix.u`. Document in the docstring that the magic operators `q`, `u`, and `kw` (of the `prefix` macro) cannot be renamed by as-importing. +- Preserve the source location info of the dialect-import statement in the example dialects in [`unpythonic.dialects`](unpythonic/dialects/). In the output, the lines of expanded source code that originate in a particular dialect template are marked as coming from the unexpanded source line that contains the corresponding dialect-import. + - If you want to see the line numbers before and after dialect expansion, use the `StepExpansion` dialect from `mcpyrate.debug`. + - This fix requires `mcpyrate` 3.6.0 or later. The code will run also on earlier versions of `mcpyrate`; then, just like before, it will look as if all lines that originate in any dialect template came from the beginning of the user source code. --- From 86ea7d45c4c7dff069c697400e98b990e1dc6cd5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 Jan 2022 14:20:14 +0200 Subject: [PATCH 646/832] wording --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac3f21e0..98a27a42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ - Add function `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses multi-shot continuations.) The two work together. See docstring. - Tag continuation closures (generated by the `with continuations` macro), for introspection. - To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. - - The information is stored as an attribute on the function object; so be careful if applying decorators manually to the continuation function. + - The information is stored as an attribute on the function object; keep this in mind if you intend to wrap the continuation function with another function. **Fixed**: From ee2420c310b9b7799b069198a3cd0ce72d1bfc70 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 Jan 2022 14:26:16 +0200 Subject: [PATCH 647/832] wording --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 98a27a42..1101284a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,8 @@ - Add function `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses multi-shot continuations.) The two work together. See docstring. - Tag continuation closures (generated by the `with continuations` macro), for introspection. - To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. - - The information is stored as an attribute on the function object; keep this in mind if you intend to wrap the continuation function with another function. + - This is purely an introspection feature; `unpythonic` itself does not use this information. For why you might want to query this, see `get_cc`, particularly the [examples in unit tests](unpythonic/syntax/tests/test_conts.py). + - The information is stored as an attribute on the function object; keep this in mind if you intend to wrap the continuation function with another function. (Strictly, this is the correct behavior, since the wrapper is not a continuation function generated by the `with continuations` macro.) **Fixed**: From faaa9c176fc342ddd6b6f8b2177b3265659fb7d0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 Jan 2022 14:26:51 +0200 Subject: [PATCH 648/832] wording again --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1101284a..141d51d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ - Tag continuation closures (generated by the `with continuations` macro), for introspection. - To detect at run time whether a given object is a continuation function, use the function `unpythonic.syntax.iscontinuation`. - This is purely an introspection feature; `unpythonic` itself does not use this information. For why you might want to query this, see `get_cc`, particularly the [examples in unit tests](unpythonic/syntax/tests/test_conts.py). - - The information is stored as an attribute on the function object; keep this in mind if you intend to wrap the continuation function with another function. (Strictly, this is the correct behavior, since the wrapper is not a continuation function generated by the `with continuations` macro.) + - The information is stored as an attribute on the function object; keep this in mind if you intend to wrap the continuation function with another function. (Strictly, this is the correct behavior, since a custom wrapper is not a continuation function generated by the `with continuations` macro.) **Fixed**: From 610617013d69791b38aa6b5229368493b8e53cc4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 27 Jan 2022 14:38:40 +0200 Subject: [PATCH 649/832] add Python 3.10 to CI --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 7ffd1add..efd6a054 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7] + python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy-3.6, pypy-3.7] steps: - uses: actions/checkout@v2 From 8b5e487ac933dffc2d9a1f29f79db092aa601b52 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 12:17:01 +0200 Subject: [PATCH 650/832] bump mcpyrate to 3.6.0 for Python 3.10 support --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 4fe57592..1840f0d6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -mcpyrate>=3.5.0 +mcpyrate>=3.6.0 sympy>=1.4 From 84027a8f0e023fcae0159686b783f429e7babb86 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 12:54:36 +0200 Subject: [PATCH 651/832] fix `namelambda` on Python 3.10 --- unpythonic/misc.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/unpythonic/misc.py b/unpythonic/misc.py index d13bbfd0..bd5c297a 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -107,13 +107,9 @@ def rename(f): # https://docs.python.org/3/library/types.html#types.CodeType # https://docs.python.org/3/library/inspect.html#types-and-members if version_info >= (3, 8, 0): # Python 3.8+: positional-only parameters - f.__code__ = CodeType(co.co_argcount, co.co_posonlyargcount, co.co_kwonlyargcount, - co.co_nlocals, co.co_stacksize, co.co_flags, - co.co_code, co.co_consts, co.co_names, - co.co_varnames, co.co_filename, - name, - co.co_firstlineno, co.co_lnotab, co.co_freevars, - co.co_cellvars) + # In Python 3.8+, `CodeType` has the convenient `replace()` method to functionally update it. + # In Python 3.10, we must actually use it to avoid losing the line number info. + f.__code__ = f.__code__.replace(co_name=name) else: f.__code__ = CodeType(co.co_argcount, co.co_kwonlyargcount, co.co_nlocals, co.co_stacksize, co.co_flags, From ef7698f2e4cec76a70aac414cef75ee629b8dfbf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 12:55:47 +0200 Subject: [PATCH 652/832] update comment --- unpythonic/misc.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpythonic/misc.py b/unpythonic/misc.py index bd5c297a..2e765a77 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -108,7 +108,8 @@ def rename(f): # https://docs.python.org/3/library/inspect.html#types-and-members if version_info >= (3, 8, 0): # Python 3.8+: positional-only parameters # In Python 3.8+, `CodeType` has the convenient `replace()` method to functionally update it. - # In Python 3.10, we must actually use it to avoid losing the line number info. + # In Python 3.10, we must actually use it to avoid losing the line number info, + # or `inspect.stack()` will crash in the unit tests for `callsite_filename()`. f.__code__ = f.__code__.replace(co_name=name) else: f.__code__ = CodeType(co.co_argcount, co.co_kwonlyargcount, From 43fac1ec40ce034b3992ccac85a321ba2f54054b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 13:15:10 +0200 Subject: [PATCH 653/832] fix detection of `typing.NewType` on Python 3.10 --- unpythonic/typecheck.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 1e5758bc..1ab400bf 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -15,6 +15,7 @@ """ import collections +import sys import typing try: @@ -93,7 +94,7 @@ def isoftype(value, T): # there's not much we can do. # TODO: Right now we're accessing internal fields to get what we need. - # TODO: Would be nice to update this if Python, at some point, adds an + # TODO: Would be nice to rewrite this if Python, at some point, adds an # TODO: official API to access the static type information at run time. if T is typing.Any: @@ -185,8 +186,14 @@ def get_origin(tp): if T is typing.Union: # isinstance(T, typing._SpecialForm) and T._name == "Union": return False # pragma: no cover, Python 3.7+ only. - # TODO: in Python 3.7+, what is the mysterious callable that doesn't have `__qualname__`? - if callable(T) and hasattr(T, "__qualname__") and T.__qualname__ == "NewType..new_type": + def isNewType(T): + # In Python 3.10, an instance of `typing.NewType` is now actually such and not just a function. Nice! + if sys.version_info >= (3, 10, 0): + return isinstance(T, typing.NewType) + # Python 3.6 through Python 3.9 + # TODO: in Python 3.7+, what is the mysterious callable that doesn't have a `__qualname__`? + return callable(T) and hasattr(T, "__qualname__") and T.__qualname__ == "NewType..new_type" + if isNewType(T): # This is the best we can do, because the static types created by `typing.NewType` # have a constructor that discards the type information at runtime: # UserId = typing.NewType("UserId", int) From 2514f56ec02367dd7b225cc7b13a64c9adcac06f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 13:17:25 +0200 Subject: [PATCH 654/832] advertise Python 3.10 support --- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 9fa0e116..75575e8a 100644 --- a/setup.py +++ b/setup.py @@ -74,7 +74,7 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", "dynamic-variable", "macros", "lisp", "scheme", "racket", "haskell"], install_requires=[], # mcpyrate is optional for us, so we can't really put it here even though we recommend it. - python_requires=">=3.6,<3.10", + python_requires=">=3.6,<3.11", author="Juha Jeronen", author_email="juha.m.jeronen@gmail.com", url="https://github.com/Technologicat/unpythonic", @@ -94,6 +94,7 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Software Development :: Libraries", From 8e790ee2cee7e7f31f1647a3ed37b6a2da54e4a2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 13:19:43 +0200 Subject: [PATCH 655/832] update changelog --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 141d51d9..dfde3193 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,8 @@ -**0.15.1** (in progress, last updated 27 January 2022) +**0.15.1** (in progress, last updated 28 January 2022) **New**: +- **Python 3.10 support**. Running on Python 3.10 requires `mcpyrate` 3.6.0. - New module `unpythonic.timeutil`, with utilities for converting a number of seconds into human-understood formats (`seconds_to_human`, `format_human_time`), and a simple running-average `ETAEstimator` that takes advantage of these. As usual, these are available at the top level of `unpythonic`. - Add function `unpythonic.syntax.get_cc`, the less antisocial little sister of `call_cc` from an alternate timeline, to make programming with continuations slightly more convenient. (Alternate timelines happen a lot when one uses multi-shot continuations.) The two work together. See docstring. - Tag continuation closures (generated by the `with continuations` macro), for introspection. From f9034e8d139ea03d6c397d8e9bdcec13e72fea9e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 13:24:52 +0200 Subject: [PATCH 656/832] finalize 0.15.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dfde3193..04d3908f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -**0.15.1** (in progress, last updated 28 January 2022) +**0.15.1** (28 January 2022) - *New Year's edition*: **New**: From f95cbbeefd799cee353b1a5c5f4a1ee5fc5173bf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 13:26:49 +0200 Subject: [PATCH 657/832] pre-emptive version bump --- CHANGELOG.md | 7 +++++++ unpythonic/__init__.py | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 04d3908f..39b80951 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +**0.15.2** (in progress, last updated 28 January 2022) + +*No user-visible changes yet.* + + +--- + **0.15.1** (28 January 2022) - *New Year's edition*: **New**: diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index ab14dc60..2b0f5ad4 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.15.1' +__version__ = '0.15.2' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From efd7d51178343c4d865c78fc99e0641165d8656d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 14:33:12 +0200 Subject: [PATCH 658/832] update language version support mention --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index aed27634..c4007083 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ None required. - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +The 0.15.x series should run on CPython 3.6, 3.7, 3.8, 3.9 and 3.10, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation From f18a34b5c0ed970e0bd6f5ad4858b09c9ce4ac1e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 28 Jan 2022 18:09:10 +0200 Subject: [PATCH 659/832] example of multishot generators using pattern `k = call_cc[get_cc()]` See #80. This still needs to be moved into its own module. --- unpythonic/syntax/tests/test_conts_gen.py | 245 +++++++++++++++++++++- 1 file changed, 242 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index 2b432a13..47319d35 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -18,7 +18,9 @@ https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt """ -from ...syntax import macros, test, test_raises # noqa: F401 +from mcpyrate.multiphase import macros, phase + +from ...syntax import macros, test, test_raises # noqa: F401, F811 from ...test.fixtures import session, testset from ...syntax import macros, continuations, call_cc, dlet, abbrev, let_syntax, block # noqa: F401, F811 @@ -26,7 +28,214 @@ from ...fploop import looped from ...fun import identity -#from mcpyrate.debug import macros, step_expansion # noqa: F811, F401 +from mcpyrate.debug import macros, step_expansion # noqa: F811, F401 + +# TODO: pretty long, move into its own module +# Multishot generators can also be implemented using the pattern `k = call_cc[get_cc()]`. +# +# Because `with continuations` is a two-pass macro, it will first expand any +# `@multishot` inside the block before performing its own processing, which is +# exactly what we want. +# +# We could force the ordering with the metatool `mcpyrate.metatools.expand_first` +# added in `mcpyrate` 3.6.0, but we don't need to do that. +# +# To make these multi-shot generators support the most basic parts +# of the API of Python's native generators, make a wrapper object: +# +# - `__iter__` on the original function should create the wrapper object +# and initialize it. Maybe always inject a bare `myield` at the beginning +# of a multishot function before other processing, and run the function +# until it returns the initial continuation? This continuation can then +# be stashed just like with any resume point. +# - `__next__` needs a stash for the most recent continuation +# per activation of the multi-shot generator. It should run +# the most recent continuation (with no arguments) until the next `myield`, +# stash the new continuation, and return the yielded value, if any. +# - `send` should send a value into the most recent continuation +# (thus resuming). +# - When the function returns normally, without returning any further continuation, +# the wrapper should `raise StopIteration`, providing the return value as argument +# to the exception. +# +# Note that a full implementation of the generator API requires much +# more. We should at least support `close` and `throw`, and think hard +# about how to handle exceptions. Particularly, a `yield` inside a +# `finally` is a classic catch. This sketch also has no support for +# `yield from`; we would likely need our own `myield_from`. +with phase[1]: + # TODO: relative imports + # TODO: mcpyrate does not recognize current package in phases higher than 0? (parent missing) + + import ast + from functools import partial + import sys + + from mcpyrate.quotes import macros, q, a, h # noqa: F811 + from unpythonic.syntax import macros, call_cc # noqa: F811 + + from mcpyrate import namemacro, gensym + from mcpyrate.quotes import is_captured_value + from mcpyrate.utils import extract_bindings + from mcpyrate.walkers import ASTTransformer + + from unpythonic.syntax import get_cc, iscontinuation + + def myield_function(tree, syntax, **kw): + if syntax not in ("name", "expr"): + raise SyntaxError("myield is a name and expr macro only") + + # Accept `myield` in any non-load context, so that we can below define the macro `it`. + # + # This is only an issue, because this example uses multi-phase compilation. + # The phase-1 `myield` is in the macro expander - preventing us from referring to + # the name `myield` - when the lifted phase-0 definition is being run. During phase 0, + # that makes the line `myield = namemacro(...)` below into a macro-expansion-time + # syntax error, because that `myield` is not inside a `@multishot`. + # + # We hack around it, by allowing `myield` anywhere as long as the context is not a `Load`. + if hasattr(tree, "ctx") and type(tree.ctx) is not ast.Load: + return tree + + raise SyntaxError("myield may only appear inside a multishot function") + myield = namemacro(myield_function) + + def multishot(tree, syntax, expander, **kw): + """[syntax, block] Multi-shot generators based on the pattern `k = call_cc[get_cc()]`.""" + if syntax != "decorator": + raise SyntaxError("multishot is a decorator macro only") # pragma: no cover + if type(tree) is not ast.FunctionDef: + raise SyntaxError("@multishot supports `def` only") + + # Detect the name(s) of `myield` at the use site (this accounts for as-imports) + macro_bindings = extract_bindings(expander.bindings, myield_function) + if not macro_bindings: + raise SyntaxError("The use site of `multishot` must macro-import `myield`, too.") + names_of_myield = list(macro_bindings.keys()) + + def is_myield_name(node): + return type(node) is ast.Name and node.id in names_of_myield + def is_myield_expr(node): + return type(node) is ast.Subscript and is_myield_name(node.value) + def getslice(subscript_node): + if sys.version_info >= (3, 9, 0): # Python 3.9+: no ast.Index wrapper + return subscript_node.slice + return subscript_node.slice.value + # We can work with variations of the pattern + # + # k = call_cc[get_cc()] + # if iscontinuation(k): + # return k + # # here `k` is the data sent in via the continuation + # + # to create a multi-shot resume point. The details will depend on whether our + # user wants each particular resume point to return and/or take in a value. + # + # Note that `myield`, beside optionally yielding a value, always returns the + # continuation that resumes execution just after that `myield`. The caller + # is free to stash the continuations and invoke earlier ones again, as needed. + class MultishotYieldTransformer(ASTTransformer): + def transform(self, tree): + if is_captured_value(tree): # do not recurse into hygienic captures + return tree + # respect scope boundaries + if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, + ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): + return tree + + # `k = myield[value]` + if type(tree) is ast.Assign and is_myield_expr(tree.value): + if len(tree.targets) != 1: + raise SyntaxError("expected exactly one assignment target in k = myield[expr]") + var = tree.targets[0] + value = getslice(tree.value) + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return a[var], a[value] + return quoted + + # `k = myield` + elif type(tree) is ast.Assign and is_myield_name(tree.value): + if len(tree.targets) != 1: + raise SyntaxError("expected exactly one assignment target in k = myield[expr]") + var = tree.targets[0] + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return a[var] + return quoted + + # `myield[value]` + elif type(tree) is ast.Expr and is_myield_expr(tree.value): + var = ast.Name(id=gensym("myield_cont")) + value = getslice(tree.value) + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return h[partial](a[var], None), a[value] + return quoted + + # `myield` + elif type(tree) is ast.Expr and is_myield_name(tree.value): + var = ast.Name(id=gensym("myield_cont")) + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return h[partial](a[var], None) + return quoted + + return self.generic_visit(tree) + + class ReturnToStopIterationTransformer(ASTTransformer): + def transform(self, tree): + if is_captured_value(tree): # do not recurse into hygienic captures + return tree + # respect scope boundaries + if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, + ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): + return tree + + if type(tree) is ast.Return: + # `return` + if tree.value is None: + with q as quoted: + raise h[StopIteration] + return quoted + # `return value` + with q as quoted: + raise h[StopIteration](a[tree.value]) + return quoted + + return self.generic_visit(tree) + + # ------------------------------------------------------------ + # main processing logic + + # Make the multishot generator raise `StopIteration` when it finishes + # via any `return`. First make the implicit bare `return` explicit. + # + # We must do this before we transform the `myield` statements, + # to avoid breaking tail-calling the continuations. + if type(tree.body[-1]) is not ast.Return: + with q as quoted: + return + tree.body.extend(quoted) + tree.body = ReturnToStopIterationTransformer().visit(tree.body) + + # Inject a bare `myield` resume point at the beginning of the function body. + # This makes the resulting function work somewhat like a Python generator. + # When initially called, the arguments are bound, and you get a continuation; + # then resuming that continuation starts the actual computation. + tree.body.insert(0, ast.Expr(value=ast.Name(id=names_of_myield[0]))) + + # Transform multishot yields (`myield`) into `call_cc`. + tree.body = MultishotYieldTransformer().visit(tree.body) + + return tree + +from __self__ import macros, multishot, myield # noqa: F811, F401 + def runtests(): with testset("a basic generator"): @@ -178,7 +387,7 @@ def result(loop, i=0): x = g2() # noqa: F821 test[out == list(range(10))] - with testset("multi-shot generators"): + with testset("multi-shot generators with call_cc[]"): with continuations: with let_syntax: with block[value] as my_yield: # noqa: F821 @@ -242,6 +451,36 @@ def my_yieldf(value=None, *, cc): # outside any make_generator are caught at compile time. The actual template the # make_generator macro needs to splice in is already here in the final example.) + with testset("multi-shot generators with the pattern call_cc[get_cc()]"): + with continuations: + @multishot + def g(): + myield[1] + myield[2] + myield[3] + + try: + out = [] + k = g() # instantiate the multishot generator + while True: + k, x = k() + out.append(x) + except StopIteration: + pass + test[out == [1, 2, 3]] + + k0 = g() # instantiate the multishot generator + k1, x1 = k0() + k2, x2 = k1() + k3, x3 = k2() + k, x = k1() # multi-shot generator can resume from an earlier point + test[x1 == 1] + test[x2 == x == 2] + test[x3 == 3] + test[k.func.__qualname__ == k2.func.__qualname__] # same bookmarked position... + test[k.func is not k2.func] # ...but different function object instance + test_raises[StopIteration, k3()] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 00fef0ab6bbb2d66cdb45a453ffb5561bb923ae4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 20:50:52 +0200 Subject: [PATCH 660/832] fix missing macro import --- unpythonic/syntax/tests/test_conts.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 62ef5dd3..47c18a1f 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- """Continuations (call/cc for Python).""" -from ...syntax import macros, test, test_raises, error # noqa: F401 +from ...syntax import macros, test, test_raises, error, fail # noqa: F401 from ...test.fixtures import session, testset, returns_normally from ...syntax import macros, continuations, call_cc, multilambda, autoreturn, autocurry, let # noqa: F401, F811 @@ -404,7 +404,7 @@ def amb(lst, cc): ourcc = cc stack.append(lambda: amb(rest, cc=ourcc)) return first - def fail(): + def fail(): # noqa: F811, not redefining, the first one is a macro. if stack: f = stack.pop() return f() From f0f6080b6a956d027443c424084668fe2e96c45d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 20:51:08 +0200 Subject: [PATCH 661/832] scopeanalyzer: add extract_args, collect_globals, collect_nonlocals --- unpythonic/syntax/scopeanalyzer.py | 68 ++++++++++++++++++++++++------ 1 file changed, 55 insertions(+), 13 deletions(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 50b16c2f..fd7e13dc 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -71,7 +71,10 @@ "scoped_transform", "get_lexical_variables", "get_names_in_store_context", - "get_names_in_del_context"] + "get_names_in_del_context", + "extract_args", + "collect_globals", + "collect_nonlocals"] from ast import (Name, Tuple, Lambda, FunctionDef, AsyncFunctionDef, ClassDef, Import, ImportFrom, Try, ListComp, SetComp, GeneratorExp, @@ -227,20 +230,9 @@ def get_lexical_variables(tree, collect_locals=True): nonlocals = [] if type(tree) in (FunctionDef, AsyncFunctionDef): fname = [tree.name] - if collect_locals: localvars = list(uniqify(get_names_in_store_context(tree.body))) - - class NonlocalsCollector(ASTVisitor): - def examine(self, tree): - if type(tree) in (Global, Nonlocal): - for x in tree.names: - self.collect(x) - if not isnewscope(tree): - self.generic_visit(tree) - nc = NonlocalsCollector() - nc.visit(tree.body) - nonlocals = nc.collected + nonlocals = collect_nonlocals(tree.body) + collect_globals(tree.body) return list(uniqify(fname + argnames + localvars)), list(uniqify(nonlocals)) @@ -396,3 +388,53 @@ def examine(self, tree): nc = DelNamesCollector() nc.visit(tree) return nc.collected + +def extract_args(tree): + """Extract the parameter names from a `Lambda`, `FunctionDef`, or `AsyncFunctionDef` node. + + Return a `list` of bare `str`. + """ + if type(tree) not in (Lambda, FunctionDef, AsyncFunctionDef): + raise ValueError(f"Expected a function definition AST node, got {tree}") + a = tree.args + allargs = a.args + a.kwonlyargs + if hasattr(a, "posonlyargs"): # Python 3.8+: positional-only arguments + allargs += a.posonlyargs + argnames = [x.arg for x in allargs] + if a.vararg: + argnames.append(a.vararg.arg) + if a.kwarg: + argnames.append(a.kwarg.arg) + return argnames + +def collect_globals(tree): + """Collect the names of all names declared `global` in `tree`, stopping at scope boundaries. + + Return a `list` of bare `str`. + """ + class GlobalsCollector(ASTVisitor): + def examine(self, tree): + if type(tree) is Global: + for name in tree.names: + self.collect(name) + if not isnewscope(tree): + self.generic_visit(tree) + collector = GlobalsCollector() + collector.visit(tree) + return collector.collected + +def collect_nonlocals(tree): + """Collect the names of all names declared `nonlocal` in `tree`, stopping at scope boundaries. + + Return a `list` of bare `str`. + """ + class NonlocalsCollector(ASTVisitor): + def examine(self, tree): + if type(tree) is Nonlocal: + for name in tree.names: + self.collect(name) + if not isnewscope(tree): + self.generic_visit(tree) + collector = NonlocalsCollector() + collector.visit(tree) + return collector.collected From 4d2e2709bef4d609dcf6e2465c835a71d556c3f4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 20:54:07 +0200 Subject: [PATCH 662/832] use extract_args --- unpythonic/syntax/scopeanalyzer.py | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index fd7e13dc..bfa13e3e 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -215,16 +215,7 @@ def get_lexical_variables(tree, collect_locals=True): raise TypeError(f"Expected a tree representing a lexical scope, got {type(tree)}") if type(tree) in (Lambda, FunctionDef, AsyncFunctionDef): - a = tree.args - allargs = a.args + a.kwonlyargs - if hasattr(a, "posonlyargs"): # Python 3.8+: positional-only arguments - allargs += a.posonlyargs - argnames = [x.arg for x in allargs] - if a.vararg: - argnames.append(a.vararg.arg) - if a.kwarg: - argnames.append(a.kwarg.arg) - + argnames = extract_args(tree) fname = [] localvars = [] nonlocals = [] From 2c7477c3d890adb997d5375946237aeb744239a3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 20:54:50 +0200 Subject: [PATCH 663/832] Experiment: scoping of locals in continuations (see #82) Maybe not worth it, after all; much simpler, and more robust, to just document that introducing a continuation introduces a scope boundary. --- unpythonic/syntax/tailtools.py | 96 +++++++++++++++++++++++++++++----- 1 file changed, 83 insertions(+), 13 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index dd31b964..348585e1 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -16,6 +16,7 @@ BoolOp, And, Or, With, AsyncWith, If, IfExp, Try, Assign, Return, Expr, Await, + Global, Nonlocal, copy_location) import sys @@ -34,6 +35,8 @@ has_tco, sort_lambda_decorators, suggest_decorator_index, UnpythonicASTMarker, ExpandedContinuationsMarker) +from .scopeanalyzer import (get_names_in_store_context, extract_args, + collect_globals, collect_nonlocals) from ..dynassign import dyn from ..fun import identity @@ -883,32 +886,97 @@ def data_cb(tree): # transform an inert-data return value into a tail-call to c # specified inside the body of the macro invocation like PG's solution does. # Instead, we capture as the continuation all remaining statements (i.e. # those that lexically appear after the ``call_cc[]``) in the current block. - def iscallcc(tree): + def iscallccstatement(tree): if type(tree) not in (Assign, Expr): return False return isinstance(tree.value, CallCcMarker) - def split_at_callcc(body): + # owner: FunctionDef node, or `None` if the use site of the `call_cc` is not inside a function + def split_at_callcc(owner, body): if not body: return [], None, [] before, after = [], body while True: stmt, *after = after - if iscallcc(stmt): + if iscallccstatement(stmt): # after is always non-empty here (has at least the explicitified "return") # ...unless we're at the top level of the "with continuations" block if not after: raise SyntaxError("call_cc[] cannot appear as the last statement of a 'with continuations' block (no continuation to capture)") # pragma: no cover - # TODO: To support Python's scoping properly in assignments after the `call_cc`, - # TODO: we have to scan `before` for assignments to local variables (stopping at - # TODO: scope boundaries; use `unpythonic.syntax.scoping.get_names_in_store_context`, - # TODO: and declare those variables (plus any variables already declared as `nonlocal` - # TODO: in `before`) as `nonlocal` in `after`. This way the binding will be shared - # TODO: between the original context and the continuation. Also, propagate `global`. - # See Politz et al 2013 (the "full monty" paper), section 4.2. + after = patch_scoping(owner, before, stmt, after) return before, stmt, after before.append(stmt) if not after: return before, None, [] + # Try to maintain an illusion of Python's standard scoping rules across the split + # into the parent context (`before`) and continuation closure (`after`). + # See Politz et al 2013 (the "full monty" paper), section 4.2. + # + # TODO: We are still missing the case where a new local is introduced in the continuation. + # TODO: Ideally, it should be made a nonlocal up to the top-level owner, where it should be defined; + # TODO: this would allow a continuation to declare a variable that is then read by the `before` part. + # TODO: (Right now that can be done, by simply declaring the variable and setting it to `None` (or + # TODO: any value, really, in the top-level owner; it will then propagate.)) + # TODO: But we still can't easily replicate the behavior that accessing the name before a value + # TODO: has been assigned to it should raise `UnboundLocalError`. + # + # TODO: Alternatively, we could declare `patch_scoping` a failed experiment, and just document + # TODO: that a continuation is a scope boundary, with all the usual implications. (This is the + # TODO: behavior up to 0.15.1, anyway, though it's not documented.) + # + # TODO: Then we can just forget about the whole thing and delete the `patch_scoping` function. :) + # + # owner: FunctionDef node, or `None` if the use site of the `call_cc` is not inside a function + def patch_scoping(owner, before, callcc, after): + # Determine the names of all variables that should be made local to the continuation function. + # In the unexpanded code, the continuation doesn't look like a new scope, so by appearances, + # these will effectively break the usual scoping rules. Thus this set should be kept minimal. + # To allow the machinery to actually work, at least the parameters of the continuation function + # *must* be allowed to shadow names from the parent scope. + targets, starget, ignored_condition, ignored_thecall, ignored_altcall = analyze_callcc(callcc) + if not targets and not starget: + targets = ["_ignored_arg"] # this must match what `make_continuation` does, below + # The assignment targets of the `call_cc` become parameters of the continuation function. + # Furthermore, a continuation function generated by `make_continuation` always takes + # the `cc` and `_pcc` parameters. + afterargs = targets + ([starget] or []) + ["cc", "_pcc"] + afterlocals = afterargs + + if owner: + # When `call_cc` is used inside a function, local variables of the + # parent function (including parameters) become nonlocals in the + # continuation. + # + # But only those that are not also locals of the continuation! + # In that case, the local variable of the continuation overrides. + # Locals of the continuation include its arguments, and any names in store context. + beforelocals = set(extract_args(owner) + get_names_in_store_context(before)) + afternonlocals = list(beforelocals.difference(afterlocals)) + if afternonlocals: # TODO: Python 3.8: walrus assignment + after.insert(0, Nonlocal(names=afternonlocals)) + else: + # When `call_cc` is used at the top level of `with continuations` block, + # the variables at that level become globals in the continuation. + # + # TODO: This **CANNOT** always work correctly, because we would need to know + # TODO: whether the `with continuations` block itself is inside a function or not. + # TODO: So we just assume it's outside any function. + beforelocals = set(get_names_in_store_context(before)) + afternonlocals = list(beforelocals.difference(afterlocals)) + if afternonlocals: # TODO: Python 3.8: walrus assignment + after.insert(0, Global(names=afternonlocals)) + + # Nonlocals of the parent function remain nonlocals in the continuation. + # When `owner is None`, `beforenonlocals` will be empty. + beforenonlocals = collect_nonlocals(before) + if beforenonlocals: # TODO: Python 3.8: walrus assignment + after.insert(0, Nonlocal(names=beforenonlocals)) + + # Globals of parent are also globals in the continuation. + beforeglobals = collect_globals(before) + if beforeglobals: # TODO: Python 3.8: walrus assignment + after.insert(0, Global(names=beforeglobals)) + + return after # we mutate; return it just for convenience # TODO: To support named return values (`kwrets` in a `Values` object) from the `call_cc`'d function, # TODO: we need to change the syntax to something that allows us to specify which names are meant to # TODO: capture the positional return values, and which ones the named return values. Doing so will @@ -947,7 +1015,7 @@ def maybe_starred(expr): # return [expr.id] or set starget raise SyntaxError(f"call_cc[]: expected an assignment or a bare expr, got {stmt}") # pragma: no cover # extract the function call(s) if not isinstance(stmt.value, CallCcMarker): # both Assign and Expr have a .value - assert False # we should get only valid call_cc[] invocations that pass the `iscallcc` test # pragma: no cover + assert False # we should get only valid call_cc[] invocations that pass the `iscallccstatement` test # pragma: no cover theexpr = stmt.value.body # discard the AST marker if not (type(theexpr) in (Call, IfExp) or (type(theexpr) in (Constant, NameConstant) and getconstant(theexpr) is None)): raise SyntaxError("the bracketed expression in call_cc[...] must be a function call, an if-expression, or None") # pragma: no cover @@ -966,6 +1034,7 @@ def extract_call(tree): condition = altcall = None thecall = extract_call(theexpr) return targets, starget, condition, thecall, altcall + # owner: FunctionDef node, or `None` if the use site of the `call_cc` is not inside a function def make_continuation(owner, callcc, contbody): targets, starget, condition, thecall, altcall = analyze_callcc(callcc) @@ -1069,19 +1138,20 @@ def transform(self, tree): if type(tree) in (FunctionDef, AsyncFunctionDef): tree.body = transform_callcc(tree, tree.body) return self.generic_visit(tree) + # owner: FunctionDef node, or `None` if the use site of the `call_cc` is not inside a function def transform_callcc(owner, body): # owner: FunctionDef or AsyncFunctionDef node, or None (top level of block) # body: list of stmts # we need to consider only one call_cc in the body, because each one # generates a new nested def for the walker to pick up. - before, callcc, after = split_at_callcc(body) + before, callcc, after = split_at_callcc(owner, body) if callcc: body = before + make_continuation(owner, callcc, contbody=after) return body # TODO: improve error reporting for stray call_cc[] invocations class StrayCallccChecker(ASTVisitor): def examine(self, tree): - if iscallcc(tree): + if iscallccstatement(tree): raise SyntaxError("call_cc[...] only allowed at the top level of a def, or at the top level of the block; must appear as an expr or an assignment RHS") # pragma: no cover if type(tree) in (Assign, Expr): v = tree.value From b94d79ab587b13877596fa6f02d9357586931a82 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 21:17:19 +0200 Subject: [PATCH 664/832] disable continuation scoping hack, add comment why --- unpythonic/syntax/tailtools.py | 140 ++++++++++++++++++--------------- 1 file changed, 77 insertions(+), 63 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 348585e1..abfb2a4c 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -902,7 +902,7 @@ def split_at_callcc(owner, body): # ...unless we're at the top level of the "with continuations" block if not after: raise SyntaxError("call_cc[] cannot appear as the last statement of a 'with continuations' block (no continuation to capture)") # pragma: no cover - after = patch_scoping(owner, before, stmt, after) + # after = patch_scoping(owner, before, stmt, after) # bad idea, DON'T DO THIS return before, stmt, after before.append(stmt) if not after: @@ -911,72 +911,86 @@ def split_at_callcc(owner, body): # into the parent context (`before`) and continuation closure (`after`). # See Politz et al 2013 (the "full monty" paper), section 4.2. # - # TODO: We are still missing the case where a new local is introduced in the continuation. - # TODO: Ideally, it should be made a nonlocal up to the top-level owner, where it should be defined; - # TODO: this would allow a continuation to declare a variable that is then read by the `before` part. - # TODO: (Right now that can be done, by simply declaring the variable and setting it to `None` (or - # TODO: any value, really, in the top-level owner; it will then propagate.)) - # TODO: But we still can't easily replicate the behavior that accessing the name before a value - # TODO: has been assigned to it should raise `UnboundLocalError`. + # TODO: On second thought, this is a bad idea, DON'T DO THIS. # - # TODO: Alternatively, we could declare `patch_scoping` a failed experiment, and just document - # TODO: that a continuation is a scope boundary, with all the usual implications. (This is the - # TODO: behavior up to 0.15.1, anyway, though it's not documented.) + # The function `patch_scoping` is an experiment that implements propagation + # of the scope of variable definitions from the parent scope into the continuation, + # recursively. But: # - # TODO: Then we can just forget about the whole thing and delete the `patch_scoping` function. :) + # - Due to how the continuation machinery works, the continuation's + # parameters (assignment targets of the `call_cc`) **must** shadow + # the same names from the parent scope, if they happen to exist there. + # + # - There is no propagation from the continuation up the parent scope + # chain. That is, if a continuation declares a new local variable, the + # name won't become available to any of the parent contexts, even if + # those are part of the same original function (to which the + # continuation splitting was applied). Implementing this would require + # a second pass. + # + # - Without looking at the source code of the full module, it is not even + # possible to determine whether the top level of the with continuations + # block is inside a function or not. This has implications to `call_cc` + # invoked from the top level of the block: should the variables from + # the parent scope be declared `nonlocal` or `global`? + # + # It is much simpler and much more robust to just document that introducing a + # continuation introduces a scope boundary - that is a simple, transparent rule + # that is easy to work with. The behavior is no worse than how, in standard Python, + # comprehensions and generator expressions introduce a scope boundary. # # owner: FunctionDef node, or `None` if the use site of the `call_cc` is not inside a function - def patch_scoping(owner, before, callcc, after): - # Determine the names of all variables that should be made local to the continuation function. - # In the unexpanded code, the continuation doesn't look like a new scope, so by appearances, - # these will effectively break the usual scoping rules. Thus this set should be kept minimal. - # To allow the machinery to actually work, at least the parameters of the continuation function - # *must* be allowed to shadow names from the parent scope. - targets, starget, ignored_condition, ignored_thecall, ignored_altcall = analyze_callcc(callcc) - if not targets and not starget: - targets = ["_ignored_arg"] # this must match what `make_continuation` does, below - # The assignment targets of the `call_cc` become parameters of the continuation function. - # Furthermore, a continuation function generated by `make_continuation` always takes - # the `cc` and `_pcc` parameters. - afterargs = targets + ([starget] or []) + ["cc", "_pcc"] - afterlocals = afterargs - - if owner: - # When `call_cc` is used inside a function, local variables of the - # parent function (including parameters) become nonlocals in the - # continuation. - # - # But only those that are not also locals of the continuation! - # In that case, the local variable of the continuation overrides. - # Locals of the continuation include its arguments, and any names in store context. - beforelocals = set(extract_args(owner) + get_names_in_store_context(before)) - afternonlocals = list(beforelocals.difference(afterlocals)) - if afternonlocals: # TODO: Python 3.8: walrus assignment - after.insert(0, Nonlocal(names=afternonlocals)) - else: - # When `call_cc` is used at the top level of `with continuations` block, - # the variables at that level become globals in the continuation. - # - # TODO: This **CANNOT** always work correctly, because we would need to know - # TODO: whether the `with continuations` block itself is inside a function or not. - # TODO: So we just assume it's outside any function. - beforelocals = set(get_names_in_store_context(before)) - afternonlocals = list(beforelocals.difference(afterlocals)) - if afternonlocals: # TODO: Python 3.8: walrus assignment - after.insert(0, Global(names=afternonlocals)) - - # Nonlocals of the parent function remain nonlocals in the continuation. - # When `owner is None`, `beforenonlocals` will be empty. - beforenonlocals = collect_nonlocals(before) - if beforenonlocals: # TODO: Python 3.8: walrus assignment - after.insert(0, Nonlocal(names=beforenonlocals)) - - # Globals of parent are also globals in the continuation. - beforeglobals = collect_globals(before) - if beforeglobals: # TODO: Python 3.8: walrus assignment - after.insert(0, Global(names=beforeglobals)) - - return after # we mutate; return it just for convenience + # def patch_scoping(owner, before, callcc, after): + # # Determine the names of all variables that should be made local to the continuation function. + # # In the unexpanded code, the continuation doesn't look like a new scope, so by appearances, + # # these will effectively break the usual scoping rules. Thus this set should be kept minimal. + # # To allow the machinery to actually work, at least the parameters of the continuation function + # # *must* be allowed to shadow names from the parent scope. + # targets, starget, ignored_condition, ignored_thecall, ignored_altcall = analyze_callcc(callcc) + # if not targets and not starget: + # targets = ["_ignored_arg"] # this must match what `make_continuation` does, below + # # The assignment targets of the `call_cc` become parameters of the continuation function. + # # Furthermore, a continuation function generated by `make_continuation` always takes + # # the `cc` and `_pcc` parameters. + # afterargs = targets + ([starget] or []) + ["cc", "_pcc"] + # afterlocals = afterargs + # + # if owner: + # # When `call_cc` is used inside a function, local variables of the + # # parent function (including parameters) become nonlocals in the + # # continuation. + # # + # # But only those that are not also locals of the continuation! + # # In that case, the local variable of the continuation overrides. + # # Locals of the continuation include its arguments, and any names in store context. + # beforelocals = set(extract_args(owner) + get_names_in_store_context(before)) + # afternonlocals = list(beforelocals.difference(afterlocals)) + # if afternonlocals: # TODO: Python 3.8: walrus assignment + # after.insert(0, Nonlocal(names=afternonlocals)) + # else: + # # When `call_cc` is used at the top level of `with continuations` block, + # # the variables at that level become globals in the continuation. + # # + # # TODO: This **CANNOT** always work correctly, because we would need to know + # # TODO: whether the `with continuations` block itself is inside a function or not. + # # TODO: So we just assume it's outside any function. + # beforelocals = set(get_names_in_store_context(before)) + # afternonlocals = list(beforelocals.difference(afterlocals)) + # if afternonlocals: # TODO: Python 3.8: walrus assignment + # after.insert(0, Global(names=afternonlocals)) + # + # # Nonlocals of the parent function remain nonlocals in the continuation. + # # When `owner is None`, `beforenonlocals` will be empty. + # beforenonlocals = collect_nonlocals(before) + # if beforenonlocals: # TODO: Python 3.8: walrus assignment + # after.insert(0, Nonlocal(names=beforenonlocals)) + # + # # Globals of parent are also globals in the continuation. + # beforeglobals = collect_globals(before) + # if beforeglobals: # TODO: Python 3.8: walrus assignment + # after.insert(0, Global(names=beforeglobals)) + # + # return after # we mutate; return it just for convenience # TODO: To support named return values (`kwrets` in a `Values` object) from the `call_cc`'d function, # TODO: we need to change the syntax to something that allows us to specify which names are meant to # TODO: capture the positional return values, and which ones the named return values. Doing so will From eb1b8b3bc421c537d8ab140b5ded00a7e30c7b2e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 21:33:49 +0200 Subject: [PATCH 665/832] `with continuations` docstring: add note about scoping --- unpythonic/syntax/tailtools.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index abfb2a4c..f961e4dd 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -351,6 +351,13 @@ def myfunc(a, b, cc): Inside a ``with continuations:`` block, the ``call_cc[]`` statement captures a continuation. (It is actually a macro, for technical reasons.) + Capturing a continuation introduces a scope boundary. The continuation + captured by `call_cc` (i.e. the rest of the function body after the + `call_cc` statement) is a new scope, and the assignment part of the + `call_cc` statement takes effect in that new scope. Under the hood, + the assignment from the `call_cc` is implemented as function parameters; + the continuation is a function. + For various possible program topologies that continuations may introduce, see the clarifying pictures under ``doc/`` in the source distribution. From 8e06c8d207e57cdca1057a73f4d98aaf6eaf5b9d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 29 Jan 2022 21:34:22 +0200 Subject: [PATCH 666/832] remove imports needed only by the scoping hack (now disabled) --- unpythonic/syntax/tailtools.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index f961e4dd..cbfbe071 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -16,7 +16,6 @@ BoolOp, And, Or, With, AsyncWith, If, IfExp, Try, Assign, Return, Expr, Await, - Global, Nonlocal, copy_location) import sys @@ -35,8 +34,6 @@ has_tco, sort_lambda_decorators, suggest_decorator_index, UnpythonicASTMarker, ExpandedContinuationsMarker) -from .scopeanalyzer import (get_names_in_store_context, extract_args, - collect_globals, collect_nonlocals) from ..dynassign import dyn from ..fun import identity From c97bf990016be38ec9d297f59a7e9a3c94d5e13f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:06:23 +0200 Subject: [PATCH 667/832] slightly cleaner --- unpythonic/syntax/tests/test_conts_gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index 47319d35..beb9393d 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -168,7 +168,7 @@ def transform(self, tree): # `myield[value]` elif type(tree) is ast.Expr and is_myield_expr(tree.value): - var = ast.Name(id=gensym("myield_cont")) + var = q[n[gensym("k")]] # kontinuation value = getslice(tree.value) with q as quoted: a[var] = h[call_cc][h[get_cc]()] @@ -178,7 +178,7 @@ def transform(self, tree): # `myield` elif type(tree) is ast.Expr and is_myield_name(tree.value): - var = ast.Name(id=gensym("myield_cont")) + var = q[n[gensym("k")]] with q as quoted: a[var] = h[call_cc][h[get_cc]()] if h[iscontinuation](a[var]): From bc2b2697864c22680b811c64acd555feef322ba0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:07:25 +0200 Subject: [PATCH 668/832] comment --- unpythonic/syntax/tests/test_conts_gen.py | 1 + 1 file changed, 1 insertion(+) diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index beb9393d..55877fd5 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -150,6 +150,7 @@ def transform(self, tree): var = tree.targets[0] value = getslice(tree.value) with q as quoted: + # Note in `mcpyrate` we can hygienically capture macros, too. a[var] = h[call_cc][h[get_cc]()] if h[iscontinuation](a[var]): return a[var], a[value] From 512ad61804dfa6e1cf19d887abe8fbc3754180d8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:07:41 +0200 Subject: [PATCH 669/832] improve explanation and comments --- unpythonic/syntax/tests/test_conts_gen.py | 216 ++++++++++++++++++++-- 1 file changed, 198 insertions(+), 18 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index 55877fd5..c6bd7474 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -71,7 +71,7 @@ from functools import partial import sys - from mcpyrate.quotes import macros, q, a, h # noqa: F811 + from mcpyrate.quotes import macros, q, n, a, h # noqa: F811 from unpythonic.syntax import macros, call_cc # noqa: F811 from mcpyrate import namemacro, gensym @@ -82,26 +82,130 @@ from unpythonic.syntax import get_cc, iscontinuation def myield_function(tree, syntax, **kw): + """[syntax, name/expr] Yield from a multi-shot generator. + + For details, see `multishot`. + """ if syntax not in ("name", "expr"): raise SyntaxError("myield is a name and expr macro only") - # Accept `myield` in any non-load context, so that we can below define the macro `it`. + # Accept `myield` in any non-load context, so that we can below define the macro `myield`. # # This is only an issue, because this example uses multi-phase compilation. # The phase-1 `myield` is in the macro expander - preventing us from referring to # the name `myield` - when the lifted phase-0 definition is being run. During phase 0, # that makes the line `myield = namemacro(...)` below into a macro-expansion-time - # syntax error, because that `myield` is not inside a `@multishot`. + # syntax error, because that `myield` is not inside a `@multishot` generator. # # We hack around it, by allowing `myield` anywhere as long as the context is not a `Load`. if hasattr(tree, "ctx") and type(tree.ctx) is not ast.Load: return tree - raise SyntaxError("myield may only appear inside a multishot function") + # `myield` is not really a macro, but a pattern that `multishot` looks for and compiles away. + # Hence if any `myield` is left over and reaches the macro expander, it was placed incorrectly, + # so we can raise an error at macro expansion time. + raise SyntaxError("myield may only appear at the top level of a `@multishot` generator") myield = namemacro(myield_function) def multishot(tree, syntax, expander, **kw): - """[syntax, block] Multi-shot generators based on the pattern `k = call_cc[get_cc()]`.""" + """[syntax, block] Make a function into a multi-shot generator. + + Multi-shot yield is spelled `myield`. When using `multishot`, be sure to + macro-import also `myield`, so that `multishot` knows which name you want + to use to refer to the `myield` construct (it is automatically queried + from the current expander's bindings). + + There are four variants:: + + Multi-shot yield Returns `k` expects Single-shot analog + + myield k no argument yield + myield[expr] (k, value) no argument yield expr + var = myield k one argument var = yield + var = myield[expr] (k, value) one argument var = yield expr + + To resume, call the function `k`. In cases where `k` expects an argument, + it is the value to send into `var`. + + Important differences: + + - A multi-shot generator may be resumed from any `myield` arbitrarily + many times, in any order. There is no concept of a single paused + activation. Each continuation is a function (technically a closure). + + When a multi-shot generator "myields", it returns just like a + normal function, technically terminating its execution. But it gives + you a continuation closure, that you can call to continue execution + just after that particular `myield`. + + The magic is in that the continuation closures are nested, so for + a given activation of the multi-shot generator, any local variables + in the already executed part remain alive as long as at least one + reference to any relevant closure instance exists. + + And yes, "nested" does imply that the execution will branch into + "alternate timelines" if you re-invoke an earlier continuation. + (Maybe you want to send a different value into some algorithm, + to alter what it will do from a certain point onward.) + + This works in exactly the same way as manually nested closures. + The parent cells (in the technical sense of "cell variable") + are shared, but the continuation that was re-invoked is separately + activated again (in the sense of "activation record"), so the + continuation gets fresh locals. Thus the "timelines" will diverge. + + - `myield` is a *statement*, and it may only appear at the top level + of a multishot function definition, due to limitations of our `call_cc` + implementation. + + Usage:: + + with continuations: + @multishot + def f(): + # Stop, and return a continuation `k` that resumes just after this `myield`. + myield + + # Stop, and return the tuple `(k, 42)`. + myield[42] + + # Stop, and return a continuation `k`. Upon resuming `k`, + # set the local `k` to the value that was sent in. + k = myield + + # Stop, and return the tuple `(k, 42)`. Upon resuming `k`, + # set the local `k` to the value that was sent in. + k = myield[42] + + # Instantiate the multi-shot generator (like calling a gfunc). + # There is always an implicit bare `myield` at the beginning. + k0 = f() + + # Start, run up to the explicit bare `myield` in the example, + # receive new continuation. + k1 = k0() + + # Continue to the `myield[42]`, receive new continuation and the `42`. + k2, x2 = k1() + test[x2 == 42] + + # Continue to the `k = myield`, receive new continuation. + k3 = k2() + + # Send `23` as the value of `k`, continue to the `k = myield[42]`. + k4, x4 = k3(23) + test[x4 == 42] + + # Send `17` as the value of `k`, continue to the end. + # As with a regular Python generator, reaching the end raises `StopIteration`. + # (As with generators, you can also trigger a `StopIteration` earlier via `return`, + # with an optional value.) + test_raises[StopIteration, k4(17)] + + # Re-invoke an earlier continuation: + k2, x2 = k1() + test[x2 == 42] + """ if syntax != "decorator": raise SyntaxError("multishot is a decorator macro only") # pragma: no cover if type(tree) is not ast.FunctionDef: @@ -121,19 +225,6 @@ def getslice(subscript_node): if sys.version_info >= (3, 9, 0): # Python 3.9+: no ast.Index wrapper return subscript_node.slice return subscript_node.slice.value - # We can work with variations of the pattern - # - # k = call_cc[get_cc()] - # if iscontinuation(k): - # return k - # # here `k` is the data sent in via the continuation - # - # to create a multi-shot resume point. The details will depend on whether our - # user wants each particular resume point to return and/or take in a value. - # - # Note that `myield`, beside optionally yielding a value, always returns the - # continuation that resumes execution just after that `myield`. The caller - # is free to stash the continuations and invoke earlier ones again, as needed. class MultishotYieldTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): # do not recurse into hygienic captures @@ -482,6 +573,95 @@ def g(): test[k.func is not k2.func] # ...but different function object instance test_raises[StopIteration, k3()] + with continuations: + def f(): + # original function scope + x = None + + # continuation 1 scope begins here + # (from the statement following `call_cc` onward, but including the `k1`) + k1 = call_cc[get_cc()] + nonlocal x + if iscontinuation(k1): + x = "cont 1 first time" + return k1, x + + # continuation 2 scope begins here + k2 = call_cc[get_cc()] + nonlocal x + if iscontinuation(k2): + x = "cont 2 first time" + return k2, x + + x = "cont 2 second time" + return None, x + + k1, x = f() + test[x == "cont 1 first time"] + k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 + test[x == "cont 2 first time"] + k3, x = k2(None) + test[k3 is None] + test[x == "cont 2 second time"] + + k2, x = k1(None) # multi-shotting from earlier resume point + test[x == "cont 2 first time"] + + with continuations: + def f(): + # original function scope + x = None + + # continuation 1 scope begins here + # (from the statement following `call_cc` onward, but including the `k1`) + k1 = call_cc[get_cc()] + if iscontinuation(k1): + x = "cont 1 first time" + return k1, x + + # continuation 2 scope begins here + k2 = call_cc[get_cc()] + if iscontinuation(k2): + x = "cont 2 first time" + return k2, x + + x = "cont 2 second time" + return None, x + + k1, x = f() + test[x == "cont 1 first time"] + k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 + test[x == "cont 2 first time"] + k3, x = k2(None) + test[k3 is None] + test[x == "cont 2 second time"] + + k2, x = k1(None) # multi-shotting from earlier resume point + test[x == "cont 2 first time"] + + with continuations: + @multishot + def f(): + myield + myield[42] + k = myield + test[k == 23] + k = myield[42] + test[k == 17] + + k0 = f() + k1 = k0() + k2, x2 = k1() + test[x2 == 42] + k3 = k2() + k4, x4 = k3(23) + test[x4 == 42] + test_raises[StopIteration, k4(17)] + + # multi-shot: re-invoke an earlier continuation + k2, x2 = k1() + test[x2 == 42] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From d4f4a4221a646d8278746e6463ccda3d4c6e1a4b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:28:46 +0200 Subject: [PATCH 670/832] scoping musings --- unpythonic/syntax/tests/test_conts.py | 72 +++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 47c18a1f..d59579cc 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -748,6 +748,78 @@ def append_stuff_to(lst): k([3, 4]) test[lst == [1, 2, 3, 4]] + with testset("scoping, locals only"): + with continuations: + def f(): + # original function scope + x = None + + # continuation 1 scope begins here + # (from the statement following `call_cc` onward, but including the `k1`) + k1 = call_cc[get_cc()] + if iscontinuation(k1): + x = "cont 1 first time" + return k1, x + + # continuation 2 scope begins here + k2 = call_cc[get_cc()] + if iscontinuation(k2): + x = "cont 2 first time" + return k2, x + + x = "cont 2 second time" + return None, x + + k1, x = f() + test[x == "cont 1 first time"] + k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 + test[x == "cont 2 first time"] + k3, x = k2(None) + test[k3 is None] + test[x == "cont 2 second time"] + + k2, x = k1(None) # multi-shotting from earlier resume point + test[x == "cont 2 first time"] + + with testset("scoping, in presence of nonlocal"): + # It shouldn't matter in this example whether we declare the `x` in the + # continuations `nonlocal`, because once the parent returns, the only + # places that can access its locals *from that activation* are the + # continuation closures *created by that activation*. + with continuations: + def f(): + # original function scope + x = None + + # continuation 1 scope begins here + # (from the statement following `call_cc` onward, but including the `k1`) + k1 = call_cc[get_cc()] + nonlocal x + if iscontinuation(k1): + x = "cont 1 first time" + return k1, x + + # continuation 2 scope begins here + k2 = call_cc[get_cc()] + nonlocal x + if iscontinuation(k2): + x = "cont 2 first time" + return k2, x + + x = "cont 2 second time" + return None, x + + k1, x = f() + test[x == "cont 1 first time"] + k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 + test[x == "cont 2 first time"] + k3, x = k2(None) + test[k3 is None] + test[x == "cont 2 second time"] + + k2, x = k1(None) # multi-shotting from earlier resume point + test[x == "cont 2 first time"] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 394953036dd8a60d38eb01dfec93f1c736714bcf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:29:01 +0200 Subject: [PATCH 671/832] move get_cc multi-shot generator example --- unpythonic/syntax/tests/test_conts_gen.py | 424 +----------------- .../syntax/tests/test_conts_multishot.py | 372 +++++++++++++++ 2 files changed, 377 insertions(+), 419 deletions(-) create mode 100644 unpythonic/syntax/tests/test_conts_multishot.py diff --git a/unpythonic/syntax/tests/test_conts_gen.py b/unpythonic/syntax/tests/test_conts_gen.py index c6bd7474..60964d26 100644 --- a/unpythonic/syntax/tests/test_conts_gen.py +++ b/unpythonic/syntax/tests/test_conts_gen.py @@ -16,9 +16,10 @@ See also the Racket version of this: https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt -""" -from mcpyrate.multiphase import macros, phase +And see the alternative approach using the pattern `k = call_cc[get_cc()]` +in `test_conts_multishot.py`. +""" from ...syntax import macros, test, test_raises # noqa: F401, F811 from ...test.fixtures import session, testset @@ -30,304 +31,6 @@ from mcpyrate.debug import macros, step_expansion # noqa: F811, F401 -# TODO: pretty long, move into its own module -# Multishot generators can also be implemented using the pattern `k = call_cc[get_cc()]`. -# -# Because `with continuations` is a two-pass macro, it will first expand any -# `@multishot` inside the block before performing its own processing, which is -# exactly what we want. -# -# We could force the ordering with the metatool `mcpyrate.metatools.expand_first` -# added in `mcpyrate` 3.6.0, but we don't need to do that. -# -# To make these multi-shot generators support the most basic parts -# of the API of Python's native generators, make a wrapper object: -# -# - `__iter__` on the original function should create the wrapper object -# and initialize it. Maybe always inject a bare `myield` at the beginning -# of a multishot function before other processing, and run the function -# until it returns the initial continuation? This continuation can then -# be stashed just like with any resume point. -# - `__next__` needs a stash for the most recent continuation -# per activation of the multi-shot generator. It should run -# the most recent continuation (with no arguments) until the next `myield`, -# stash the new continuation, and return the yielded value, if any. -# - `send` should send a value into the most recent continuation -# (thus resuming). -# - When the function returns normally, without returning any further continuation, -# the wrapper should `raise StopIteration`, providing the return value as argument -# to the exception. -# -# Note that a full implementation of the generator API requires much -# more. We should at least support `close` and `throw`, and think hard -# about how to handle exceptions. Particularly, a `yield` inside a -# `finally` is a classic catch. This sketch also has no support for -# `yield from`; we would likely need our own `myield_from`. -with phase[1]: - # TODO: relative imports - # TODO: mcpyrate does not recognize current package in phases higher than 0? (parent missing) - - import ast - from functools import partial - import sys - - from mcpyrate.quotes import macros, q, n, a, h # noqa: F811 - from unpythonic.syntax import macros, call_cc # noqa: F811 - - from mcpyrate import namemacro, gensym - from mcpyrate.quotes import is_captured_value - from mcpyrate.utils import extract_bindings - from mcpyrate.walkers import ASTTransformer - - from unpythonic.syntax import get_cc, iscontinuation - - def myield_function(tree, syntax, **kw): - """[syntax, name/expr] Yield from a multi-shot generator. - - For details, see `multishot`. - """ - if syntax not in ("name", "expr"): - raise SyntaxError("myield is a name and expr macro only") - - # Accept `myield` in any non-load context, so that we can below define the macro `myield`. - # - # This is only an issue, because this example uses multi-phase compilation. - # The phase-1 `myield` is in the macro expander - preventing us from referring to - # the name `myield` - when the lifted phase-0 definition is being run. During phase 0, - # that makes the line `myield = namemacro(...)` below into a macro-expansion-time - # syntax error, because that `myield` is not inside a `@multishot` generator. - # - # We hack around it, by allowing `myield` anywhere as long as the context is not a `Load`. - if hasattr(tree, "ctx") and type(tree.ctx) is not ast.Load: - return tree - - # `myield` is not really a macro, but a pattern that `multishot` looks for and compiles away. - # Hence if any `myield` is left over and reaches the macro expander, it was placed incorrectly, - # so we can raise an error at macro expansion time. - raise SyntaxError("myield may only appear at the top level of a `@multishot` generator") - myield = namemacro(myield_function) - - def multishot(tree, syntax, expander, **kw): - """[syntax, block] Make a function into a multi-shot generator. - - Multi-shot yield is spelled `myield`. When using `multishot`, be sure to - macro-import also `myield`, so that `multishot` knows which name you want - to use to refer to the `myield` construct (it is automatically queried - from the current expander's bindings). - - There are four variants:: - - Multi-shot yield Returns `k` expects Single-shot analog - - myield k no argument yield - myield[expr] (k, value) no argument yield expr - var = myield k one argument var = yield - var = myield[expr] (k, value) one argument var = yield expr - - To resume, call the function `k`. In cases where `k` expects an argument, - it is the value to send into `var`. - - Important differences: - - - A multi-shot generator may be resumed from any `myield` arbitrarily - many times, in any order. There is no concept of a single paused - activation. Each continuation is a function (technically a closure). - - When a multi-shot generator "myields", it returns just like a - normal function, technically terminating its execution. But it gives - you a continuation closure, that you can call to continue execution - just after that particular `myield`. - - The magic is in that the continuation closures are nested, so for - a given activation of the multi-shot generator, any local variables - in the already executed part remain alive as long as at least one - reference to any relevant closure instance exists. - - And yes, "nested" does imply that the execution will branch into - "alternate timelines" if you re-invoke an earlier continuation. - (Maybe you want to send a different value into some algorithm, - to alter what it will do from a certain point onward.) - - This works in exactly the same way as manually nested closures. - The parent cells (in the technical sense of "cell variable") - are shared, but the continuation that was re-invoked is separately - activated again (in the sense of "activation record"), so the - continuation gets fresh locals. Thus the "timelines" will diverge. - - - `myield` is a *statement*, and it may only appear at the top level - of a multishot function definition, due to limitations of our `call_cc` - implementation. - - Usage:: - - with continuations: - @multishot - def f(): - # Stop, and return a continuation `k` that resumes just after this `myield`. - myield - - # Stop, and return the tuple `(k, 42)`. - myield[42] - - # Stop, and return a continuation `k`. Upon resuming `k`, - # set the local `k` to the value that was sent in. - k = myield - - # Stop, and return the tuple `(k, 42)`. Upon resuming `k`, - # set the local `k` to the value that was sent in. - k = myield[42] - - # Instantiate the multi-shot generator (like calling a gfunc). - # There is always an implicit bare `myield` at the beginning. - k0 = f() - - # Start, run up to the explicit bare `myield` in the example, - # receive new continuation. - k1 = k0() - - # Continue to the `myield[42]`, receive new continuation and the `42`. - k2, x2 = k1() - test[x2 == 42] - - # Continue to the `k = myield`, receive new continuation. - k3 = k2() - - # Send `23` as the value of `k`, continue to the `k = myield[42]`. - k4, x4 = k3(23) - test[x4 == 42] - - # Send `17` as the value of `k`, continue to the end. - # As with a regular Python generator, reaching the end raises `StopIteration`. - # (As with generators, you can also trigger a `StopIteration` earlier via `return`, - # with an optional value.) - test_raises[StopIteration, k4(17)] - - # Re-invoke an earlier continuation: - k2, x2 = k1() - test[x2 == 42] - """ - if syntax != "decorator": - raise SyntaxError("multishot is a decorator macro only") # pragma: no cover - if type(tree) is not ast.FunctionDef: - raise SyntaxError("@multishot supports `def` only") - - # Detect the name(s) of `myield` at the use site (this accounts for as-imports) - macro_bindings = extract_bindings(expander.bindings, myield_function) - if not macro_bindings: - raise SyntaxError("The use site of `multishot` must macro-import `myield`, too.") - names_of_myield = list(macro_bindings.keys()) - - def is_myield_name(node): - return type(node) is ast.Name and node.id in names_of_myield - def is_myield_expr(node): - return type(node) is ast.Subscript and is_myield_name(node.value) - def getslice(subscript_node): - if sys.version_info >= (3, 9, 0): # Python 3.9+: no ast.Index wrapper - return subscript_node.slice - return subscript_node.slice.value - class MultishotYieldTransformer(ASTTransformer): - def transform(self, tree): - if is_captured_value(tree): # do not recurse into hygienic captures - return tree - # respect scope boundaries - if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, - ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): - return tree - - # `k = myield[value]` - if type(tree) is ast.Assign and is_myield_expr(tree.value): - if len(tree.targets) != 1: - raise SyntaxError("expected exactly one assignment target in k = myield[expr]") - var = tree.targets[0] - value = getslice(tree.value) - with q as quoted: - # Note in `mcpyrate` we can hygienically capture macros, too. - a[var] = h[call_cc][h[get_cc]()] - if h[iscontinuation](a[var]): - return a[var], a[value] - return quoted - - # `k = myield` - elif type(tree) is ast.Assign and is_myield_name(tree.value): - if len(tree.targets) != 1: - raise SyntaxError("expected exactly one assignment target in k = myield[expr]") - var = tree.targets[0] - with q as quoted: - a[var] = h[call_cc][h[get_cc]()] - if h[iscontinuation](a[var]): - return a[var] - return quoted - - # `myield[value]` - elif type(tree) is ast.Expr and is_myield_expr(tree.value): - var = q[n[gensym("k")]] # kontinuation - value = getslice(tree.value) - with q as quoted: - a[var] = h[call_cc][h[get_cc]()] - if h[iscontinuation](a[var]): - return h[partial](a[var], None), a[value] - return quoted - - # `myield` - elif type(tree) is ast.Expr and is_myield_name(tree.value): - var = q[n[gensym("k")]] - with q as quoted: - a[var] = h[call_cc][h[get_cc]()] - if h[iscontinuation](a[var]): - return h[partial](a[var], None) - return quoted - - return self.generic_visit(tree) - - class ReturnToStopIterationTransformer(ASTTransformer): - def transform(self, tree): - if is_captured_value(tree): # do not recurse into hygienic captures - return tree - # respect scope boundaries - if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, - ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): - return tree - - if type(tree) is ast.Return: - # `return` - if tree.value is None: - with q as quoted: - raise h[StopIteration] - return quoted - # `return value` - with q as quoted: - raise h[StopIteration](a[tree.value]) - return quoted - - return self.generic_visit(tree) - - # ------------------------------------------------------------ - # main processing logic - - # Make the multishot generator raise `StopIteration` when it finishes - # via any `return`. First make the implicit bare `return` explicit. - # - # We must do this before we transform the `myield` statements, - # to avoid breaking tail-calling the continuations. - if type(tree.body[-1]) is not ast.Return: - with q as quoted: - return - tree.body.extend(quoted) - tree.body = ReturnToStopIterationTransformer().visit(tree.body) - - # Inject a bare `myield` resume point at the beginning of the function body. - # This makes the resulting function work somewhat like a Python generator. - # When initially called, the arguments are bound, and you get a continuation; - # then resuming that continuation starts the actual computation. - tree.body.insert(0, ast.Expr(value=ast.Name(id=names_of_myield[0]))) - - # Transform multishot yields (`myield`) into `call_cc`. - tree.body = MultishotYieldTransformer().visit(tree.body) - - return tree - -from __self__ import macros, multishot, myield # noqa: F811, F401 - def runtests(): with testset("a basic generator"): @@ -542,125 +245,8 @@ def my_yieldf(value=None, *, cc): # module level, define my_yield as a magic variable so that accidental uses # outside any make_generator are caught at compile time. The actual template the # make_generator macro needs to splice in is already here in the final example.) - - with testset("multi-shot generators with the pattern call_cc[get_cc()]"): - with continuations: - @multishot - def g(): - myield[1] - myield[2] - myield[3] - - try: - out = [] - k = g() # instantiate the multishot generator - while True: - k, x = k() - out.append(x) - except StopIteration: - pass - test[out == [1, 2, 3]] - - k0 = g() # instantiate the multishot generator - k1, x1 = k0() - k2, x2 = k1() - k3, x3 = k2() - k, x = k1() # multi-shot generator can resume from an earlier point - test[x1 == 1] - test[x2 == x == 2] - test[x3 == 3] - test[k.func.__qualname__ == k2.func.__qualname__] # same bookmarked position... - test[k.func is not k2.func] # ...but different function object instance - test_raises[StopIteration, k3()] - - with continuations: - def f(): - # original function scope - x = None - - # continuation 1 scope begins here - # (from the statement following `call_cc` onward, but including the `k1`) - k1 = call_cc[get_cc()] - nonlocal x - if iscontinuation(k1): - x = "cont 1 first time" - return k1, x - - # continuation 2 scope begins here - k2 = call_cc[get_cc()] - nonlocal x - if iscontinuation(k2): - x = "cont 2 first time" - return k2, x - - x = "cont 2 second time" - return None, x - - k1, x = f() - test[x == "cont 1 first time"] - k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 - test[x == "cont 2 first time"] - k3, x = k2(None) - test[k3 is None] - test[x == "cont 2 second time"] - - k2, x = k1(None) # multi-shotting from earlier resume point - test[x == "cont 2 first time"] - - with continuations: - def f(): - # original function scope - x = None - - # continuation 1 scope begins here - # (from the statement following `call_cc` onward, but including the `k1`) - k1 = call_cc[get_cc()] - if iscontinuation(k1): - x = "cont 1 first time" - return k1, x - - # continuation 2 scope begins here - k2 = call_cc[get_cc()] - if iscontinuation(k2): - x = "cont 2 first time" - return k2, x - - x = "cont 2 second time" - return None, x - - k1, x = f() - test[x == "cont 1 first time"] - k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 - test[x == "cont 2 first time"] - k3, x = k2(None) - test[k3 is None] - test[x == "cont 2 second time"] - - k2, x = k1(None) # multi-shotting from earlier resume point - test[x == "cont 2 first time"] - - with continuations: - @multishot - def f(): - myield - myield[42] - k = myield - test[k == 23] - k = myield[42] - test[k == 17] - - k0 = f() - k1 = k0() - k2, x2 = k1() - test[x2 == 42] - k3 = k2() - k4, x4 = k3(23) - test[x4 == 42] - test_raises[StopIteration, k4(17)] - - # multi-shot: re-invoke an earlier continuation - k2, x2 = k1() - test[x2 == 42] + # + # See `test_conts_multishot.py`, where we do librarify this a bit further. if __name__ == '__main__': # pragma: no cover with session(__file__): diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py new file mode 100644 index 00000000..7f3359be --- /dev/null +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -0,0 +1,372 @@ +# -*- coding: utf-8 -*- +"""Multi-shot generator demo using the pattern `k = call_cc[get_cc()]`. + +This is a barebones implementation, which does not even conform to Python's +generator API. + +We provide everything in one file, so we use `mcpyrate`'s multi-phase compilation. + +Because `with continuations` is a two-pass macro, it will first expand any +`@multishot` inside the block before performing its own processing, which is +exactly what we want. + +We could force the ordering with the metatool `mcpyrate.metatools.expand_first` +added in `mcpyrate` 3.6.0, but we don't need to do that. + +Exercise to the reader: + +To make these multi-shot generators support the most basic parts +of the API of Python's native generators, make a wrapper object: + + - `__iter__` on the original function should create the wrapper object + and initialize it. Stash the continuation from the implicit initial + resume point. + + - `__next__` needs a stash for the most recent continuation + per activation of the multi-shot generator. It should run + the most recent continuation (with no arguments) until the next `myield`, + stash the new continuation, and return the myielded value, if any. + + - `send` should send a value into the most recent continuation + (thus resuming). + +A full implementation of the generator API requires much more: + + - `close` + - `throw` + - Think hard on how to handle exceptions. + - Particularly, a `yield` inside a `finally` block is a classic catch. + - `yield from` (delegation); needs a custom `myield_from`. +""" + +from mcpyrate.multiphase import macros, phase + +from ...syntax import macros, test, test_raises # noqa: F401, F811 +from ...test.fixtures import session, testset + +from ...syntax import macros, continuations # noqa: F811 + +with phase[1]: + # TODO: relative imports + # TODO: mcpyrate does not recognize current package in phases higher than 0? (parent package missing) + + import ast + from functools import partial + import sys + + from mcpyrate.quotes import macros, q, n, a, h # noqa: F811 + from unpythonic.syntax import macros, call_cc # noqa: F811 + + from mcpyrate import namemacro, gensym + from mcpyrate.quotes import is_captured_value + from mcpyrate.utils import extract_bindings + from mcpyrate.walkers import ASTTransformer + + from unpythonic.syntax import get_cc, iscontinuation + + def myield_function(tree, syntax, **kw): + """[syntax, name/expr] Yield from a multi-shot generator. + + For details, see `multishot`. + """ + if syntax not in ("name", "expr"): + raise SyntaxError("myield is a name and expr macro only") + + # Accept `myield` in any non-load context, so that we can below define the macro `myield`. + # + # This is only an issue, because this example uses multi-phase compilation. + # The phase-1 `myield` is in the macro expander - preventing us from referring to + # the name `myield` - when the lifted phase-0 definition is being run. During phase 0, + # that makes the line `myield = namemacro(...)` below into a macro-expansion-time + # syntax error, because that `myield` is not inside a `@multishot` generator. + # + # We hack around it, by allowing `myield` anywhere as long as the context is not a `Load`. + if hasattr(tree, "ctx") and type(tree.ctx) is not ast.Load: + return tree + + # `myield` is not really a macro, but a pattern that `multishot` looks for and compiles away. + # Hence if any `myield` is left over and reaches the macro expander, it was placed incorrectly, + # so we can raise an error at macro expansion time. + raise SyntaxError("myield may only appear at the top level of a `@multishot` generator") + myield = namemacro(myield_function) + + def multishot(tree, syntax, expander, **kw): + """[syntax, block] Make a function into a multi-shot generator. + + Multi-shot yield is spelled `myield`. When using `multishot`, be sure to + macro-import also `myield`, so that `multishot` knows which name you want + to use to refer to the `myield` construct (it is automatically queried + from the current expander's bindings). + + There are four variants:: + + Multi-shot yield Returns `k` expects Single-shot analog + + myield k no argument yield + myield[expr] (k, value) no argument yield expr + var = myield k one argument var = yield + var = myield[expr] (k, value) one argument var = yield expr + + To resume, call the function `k`. In cases where `k` expects an argument, + it is the value to send into `var`. + + Important differences: + + - A multi-shot generator may be resumed from any `myield` arbitrarily + many times, in any order. There is no concept of a single paused + activation. Each continuation is a function (technically a closure). + + When a multi-shot generator "myields", it returns just like a + normal function, technically terminating its execution. But it gives + you a continuation closure, that you can call to continue execution + just after that particular `myield`. + + The magic is in that the continuation closures are nested, so for + a given activation of the multi-shot generator, any local variables + in the already executed part remain alive as long as at least one + reference to any relevant closure instance exists. + + And yes, "nested" does imply that the execution will branch into + "alternate timelines" if you re-invoke an earlier continuation. + (Maybe you want to send a different value into some algorithm, + to alter what it will do from a certain point onward.) + + This works in exactly the same way as manually nested closures. + The parent cells (in the technical sense of "cell variable") + are shared, but the continuation that was re-invoked is separately + activated again (in the sense of "activation record"), so the + continuation gets fresh locals. Thus the "timelines" will diverge. + + - `myield` is a *statement*, and it may only appear at the top level + of a multishot function definition, due to limitations of our `call_cc` + implementation. + + Usage:: + + with continuations: + @multishot + def f(): + # Stop, and return a continuation `k` that resumes just after this `myield`. + myield + + # Stop, and return the tuple `(k, 42)`. + myield[42] + + # Stop, and return a continuation `k`. Upon resuming `k`, + # set the local `k` to the value that was sent in. + k = myield + + # Stop, and return the tuple `(k, 42)`. Upon resuming `k`, + # set the local `k` to the value that was sent in. + k = myield[42] + + # Instantiate the multi-shot generator (like calling a gfunc). + # There is always an implicit bare `myield` at the beginning. + k0 = f() + + # Start, run up to the explicit bare `myield` in the example, + # receive new continuation. + k1 = k0() + + # Continue to the `myield[42]`, receive new continuation and the `42`. + k2, x2 = k1() + test[x2 == 42] + + # Continue to the `k = myield`, receive new continuation. + k3 = k2() + + # Send `23` as the value of `k`, continue to the `k = myield[42]`. + k4, x4 = k3(23) + test[x4 == 42] + + # Send `17` as the value of `k`, continue to the end. + # As with a regular Python generator, reaching the end raises `StopIteration`. + # (As with generators, you can also trigger a `StopIteration` earlier via `return`, + # with an optional value.) + test_raises[StopIteration, k4(17)] + + # Re-invoke an earlier continuation: + k2, x2 = k1() + test[x2 == 42] + """ + if syntax != "decorator": + raise SyntaxError("multishot is a decorator macro only") # pragma: no cover + if type(tree) is not ast.FunctionDef: + raise SyntaxError("@multishot supports `def` only") + + # Detect the name(s) of `myield` at the use site (this accounts for as-imports) + macro_bindings = extract_bindings(expander.bindings, myield_function) + if not macro_bindings: + raise SyntaxError("The use site of `multishot` must macro-import `myield`, too.") + names_of_myield = list(macro_bindings.keys()) + + def is_myield_name(node): + return type(node) is ast.Name and node.id in names_of_myield + def is_myield_expr(node): + return type(node) is ast.Subscript and is_myield_name(node.value) + def getslice(subscript_node): + if sys.version_info >= (3, 9, 0): # Python 3.9+: no ast.Index wrapper + return subscript_node.slice + return subscript_node.slice.value + class MultishotYieldTransformer(ASTTransformer): + def transform(self, tree): + if is_captured_value(tree): # do not recurse into hygienic captures + return tree + # respect scope boundaries + if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, + ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): + return tree + + # `k = myield[value]` + if type(tree) is ast.Assign and is_myield_expr(tree.value): + if len(tree.targets) != 1: + raise SyntaxError("expected exactly one assignment target in k = myield[expr]") + var = tree.targets[0] + value = getslice(tree.value) + with q as quoted: + # Note in `mcpyrate` we can hygienically capture macros, too. + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return a[var], a[value] + return quoted + + # `k = myield` + elif type(tree) is ast.Assign and is_myield_name(tree.value): + if len(tree.targets) != 1: + raise SyntaxError("expected exactly one assignment target in k = myield[expr]") + var = tree.targets[0] + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return a[var] + return quoted + + # `myield[value]` + elif type(tree) is ast.Expr and is_myield_expr(tree.value): + var = q[n[gensym("k")]] # kontinuation + value = getslice(tree.value) + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return h[partial](a[var], None), a[value] + return quoted + + # `myield` + elif type(tree) is ast.Expr and is_myield_name(tree.value): + var = q[n[gensym("k")]] + with q as quoted: + a[var] = h[call_cc][h[get_cc]()] + if h[iscontinuation](a[var]): + return h[partial](a[var], None) + return quoted + + return self.generic_visit(tree) + + class ReturnToStopIterationTransformer(ASTTransformer): + def transform(self, tree): + if is_captured_value(tree): # do not recurse into hygienic captures + return tree + # respect scope boundaries + if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, + ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): + return tree + + if type(tree) is ast.Return: + # `return` + if tree.value is None: + with q as quoted: + raise h[StopIteration] + return quoted + # `return value` + with q as quoted: + raise h[StopIteration](a[tree.value]) + return quoted + + return self.generic_visit(tree) + + # ------------------------------------------------------------ + # main processing logic + + # Make the multishot generator raise `StopIteration` when it finishes + # via any `return`. First make the implicit bare `return` explicit. + # + # We must do this before we transform the `myield` statements, + # to avoid breaking tail-calling the continuations. + if type(tree.body[-1]) is not ast.Return: + with q as quoted: + return + tree.body.extend(quoted) + tree.body = ReturnToStopIterationTransformer().visit(tree.body) + + # Inject a bare `myield` resume point at the beginning of the function body. + # This makes the resulting function work somewhat like a Python generator. + # When initially called, the arguments are bound, and you get a continuation; + # then resuming that continuation starts the actual computation. + tree.body.insert(0, ast.Expr(value=ast.Name(id=names_of_myield[0]))) + + # Transform multishot yields (`myield`) into `call_cc`. + tree.body = MultishotYieldTransformer().visit(tree.body) + + return tree + + +# macro-import from higher phase; we're now in phase 0 +from __self__ import macros, multishot, myield # noqa: F811, F401 + +def runtests(): + with testset("multi-shot generators with the pattern call_cc[get_cc()]"): + with continuations: + @multishot + def g(): + myield[1] + myield[2] + myield[3] + + try: + out = [] + k = g() # instantiate the multishot generator + while True: + k, x = k() + out.append(x) + except StopIteration: + pass + test[out == [1, 2, 3]] + + k0 = g() # instantiate the multishot generator + k1, x1 = k0() + k2, x2 = k1() + k3, x3 = k2() + k, x = k1() # multi-shot generator can resume from an earlier point + test[x1 == 1] + test[x2 == x == 2] + test[x3 == 3] + test[k.func.__qualname__ == k2.func.__qualname__] # same bookmarked position... + test[k.func is not k2.func] # ...but different function object instance + test_raises[StopIteration, k3()] + + with continuations: + @multishot + def f(): + myield + myield[42] + k = myield + test[k == 23] + k = myield[42] + test[k == 17] + + k0 = f() + k1 = k0() + k2, x2 = k1() + test[x2 == 42] + k3 = k2() + k4, x4 = k3(23) + test[x4 == 42] + test_raises[StopIteration, k4(17)] + + # multi-shot: re-invoke an earlier continuation + k2, x2 = k1() + test[x2 == 42] + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() From 0bc9a710096eeb77980e37964da22d39128c98be Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:38:06 +0200 Subject: [PATCH 672/832] update comment --- unpythonic/syntax/tests/test_conts.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index d59579cc..e7f1dc29 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -721,7 +721,7 @@ def append_stuff_to(lst): # if iscontinuation(k): # return k # - # creates a multi-shot resume point: + # creates a multi-shot resume point. See also `test_conts_multishot.py`. def append_stuff_to(lst): ... # could do something useful here (otherwise, why make a continuation?) From 9a992ca6552d9f7462d060d2abe4eb2056873c6e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 02:45:27 +0200 Subject: [PATCH 673/832] disable test that breaks coverage analyzer --- unpythonic/syntax/tests/test_conts.py | 78 ++++++++++++++------------- 1 file changed, 40 insertions(+), 38 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index e7f1dc29..c07c6711 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -781,44 +781,46 @@ def f(): k2, x = k1(None) # multi-shotting from earlier resume point test[x == "cont 2 first time"] - with testset("scoping, in presence of nonlocal"): - # It shouldn't matter in this example whether we declare the `x` in the - # continuations `nonlocal`, because once the parent returns, the only - # places that can access its locals *from that activation* are the - # continuation closures *created by that activation*. - with continuations: - def f(): - # original function scope - x = None - - # continuation 1 scope begins here - # (from the statement following `call_cc` onward, but including the `k1`) - k1 = call_cc[get_cc()] - nonlocal x - if iscontinuation(k1): - x = "cont 1 first time" - return k1, x - - # continuation 2 scope begins here - k2 = call_cc[get_cc()] - nonlocal x - if iscontinuation(k2): - x = "cont 2 first time" - return k2, x - - x = "cont 2 second time" - return None, x - - k1, x = f() - test[x == "cont 1 first time"] - k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 - test[x == "cont 2 first time"] - k3, x = k2(None) - test[k3 is None] - test[x == "cont 2 second time"] - - k2, x = k1(None) # multi-shotting from earlier resume point - test[x == "cont 2 first time"] + # TODO: This breaks the coverage analyzer, because 'name 'x' is assigned to before nonlocal declaration'. + # TODO: Fair enough, that's not standard Python. So let's just disable this for now. + # with testset("scoping, in presence of nonlocal"): + # # It shouldn't matter in this example whether we declare the `x` in the + # # continuations `nonlocal`, because once the parent returns, the only + # # places that can access its locals *from that activation* are the + # # continuation closures *created by that activation*. + # with continuations: + # def f(): + # # original function scope + # x = None + # + # # continuation 1 scope begins here + # # (from the statement following `call_cc` onward, but including the `k1`) + # k1 = call_cc[get_cc()] + # nonlocal x + # if iscontinuation(k1): + # x = "cont 1 first time" + # return k1, x + # + # # continuation 2 scope begins here + # k2 = call_cc[get_cc()] + # nonlocal x + # if iscontinuation(k2): + # x = "cont 2 first time" + # return k2, x + # + # x = "cont 2 second time" + # return None, x + # + # k1, x = f() + # test[x == "cont 1 first time"] + # k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 + # test[x == "cont 2 first time"] + # k3, x = k2(None) + # test[k3 is None] + # test[x == "cont 2 second time"] + # + # k2, x = k1(None) # multi-shotting from earlier resume point + # test[x == "cont 2 first time"] if __name__ == '__main__': # pragma: no cover with session(__file__): From 0f0a7d107b35ae8c3d139e90b0f3c4d4264d76fb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 03:02:28 +0200 Subject: [PATCH 674/832] add pypy-3.8 to CI --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index efd6a054..8be890d7 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy-3.6, pypy-3.7] + python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy-3.6, pypy-3.7, pypy-3.8] steps: - uses: actions/checkout@v2 From e2709cf8e60e4df254418315cd44a8346046dad5 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 03:08:19 +0200 Subject: [PATCH 675/832] update language version support mention --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c4007083..8b5414c8 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ None required. - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -The 0.15.x series should run on CPython 3.6, 3.7, 3.8, 3.9 and 3.10, and PyPy3 (language versions 3.6 and 3.7); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +The 0.15.x series should run on CPython 3.6, 3.7, 3.8, 3.9 and 3.10, and PyPy3 (language versions 3.6, 3.7 and 3.8); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation From 347fc46e5670e6dc4e906b3aca48af3a222c6d42 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 12:47:29 +0200 Subject: [PATCH 676/832] improve multi-shot generator example --- .../syntax/tests/test_conts_multishot.py | 109 ++++++++++++++---- 1 file changed, 84 insertions(+), 25 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index 7f3359be..4ff0c605 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -314,7 +314,89 @@ def transform(self, tree): from __self__ import macros, multishot, myield # noqa: F811, F401 def runtests(): + # To start with, here's a sketch of what we want to do. with testset("multi-shot generators with the pattern call_cc[get_cc()]"): + with continuations: + def g(): + # The resume point at the beginning (just after parameters of `g` have + # been bound to the given arguments; though here we don't have any). + k = call_cc[get_cc()] + if iscontinuation(k): + # The `partial` makes it so `k` doesn't expect an argument; + # otherwise it would expect a value to set the local variable `k` to + # when the continuation is resumed. + # + # Since this example doesn't use that `k` if it's not the continuation + # (i.e. the initial return value of the `call_cc[get_cc()]`), + # we can just set the argument to `None` here. + return partial(k, None) + + # yield 1 + k = call_cc[get_cc()] + if iscontinuation(k): + return partial(k, None), 1 + + # yield 2 + k = call_cc[get_cc()] + if iscontinuation(k): + return partial(k, None), 2 + + # yield 3 + k = call_cc[get_cc()] + if iscontinuation(k): + return partial(k, None), 3 + + raise StopIteration + + try: + out = [] + k = g() # instantiate the multi-shot generator + while True: + k, x = k() + out.append(x) + except StopIteration: + pass + test[out == [1, 2, 3]] + + k0 = g() # instantiate the multi-shot generator + k1, x1 = k0() + k2, x2 = k1() + k3, x3 = k2() + k, x = k1() # multi-shot generator can resume from an earlier point + test[x1 == 1] + test[x2 == x == 2] + test[x3 == 3] + test[k.func.__qualname__ == k2.func.__qualname__] # same bookmarked position... + test[k.func is not k2.func] # ...but different function object instance + test_raises[StopIteration, k3()] + + # Now, let's automate this. Testing all four kinds of multi-shot yield: + with testset("@multishot macro"): + with continuations: + @multishot + def f(): + myield + myield[42] + k = myield + test[k == 23] + k = myield[42] + test[k == 17] + + k0 = f() # instantiate the multi-shot generator + k1 = k0() + k2, x2 = k1() + test[x2 == 42] + k3 = k2() + k4, x4 = k3(23) + test[x4 == 42] + test_raises[StopIteration, k4(17)] + + # multi-shot: re-invoke an earlier continuation + k2, x2 = k1() + test[x2 == 42] + + # The first example rewritten to use the macro. + with testset("multi-shot generators with @multishot"): with continuations: @multishot def g(): @@ -324,7 +406,7 @@ def g(): try: out = [] - k = g() # instantiate the multishot generator + k = g() # instantiate the multi-shot generator while True: k, x = k() out.append(x) @@ -332,7 +414,7 @@ def g(): pass test[out == [1, 2, 3]] - k0 = g() # instantiate the multishot generator + k0 = g() # instantiate the multi-shot generator k1, x1 = k0() k2, x2 = k1() k3, x3 = k2() @@ -344,29 +426,6 @@ def g(): test[k.func is not k2.func] # ...but different function object instance test_raises[StopIteration, k3()] - with continuations: - @multishot - def f(): - myield - myield[42] - k = myield - test[k == 23] - k = myield[42] - test[k == 17] - - k0 = f() - k1 = k0() - k2, x2 = k1() - test[x2 == 42] - k3 = k2() - k4, x4 = k3(23) - test[x4 == 42] - test_raises[StopIteration, k4(17)] - - # multi-shot: re-invoke an earlier continuation - k2, x2 = k1() - test[x2 == 42] - if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 5896c924e026540d885c23078bac72519f260641 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 12:54:56 +0200 Subject: [PATCH 677/832] improve example --- unpythonic/syntax/tests/test_conts.py | 60 +++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 8 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index c07c6711..4a5ec5f8 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -7,6 +7,7 @@ from ...syntax import macros, continuations, call_cc, multilambda, autoreturn, autocurry, let # noqa: F401, F811 from ...syntax import get_cc, iscontinuation +from ...collections import box, unbox from ...ec import call_ec from ...fploop import looped from ...fun import withself @@ -751,22 +752,25 @@ def append_stuff_to(lst): with testset("scoping, locals only"): with continuations: def f(): - # original function scope + # Original function scope x = None - # continuation 1 scope begins here + # Continuation 1 scope begins here # (from the statement following `call_cc` onward, but including the `k1`) k1 = call_cc[get_cc()] if iscontinuation(k1): + # This `x` is local to continuation 1. x = "cont 1 first time" return k1, x - # continuation 2 scope begins here + # Continuation 2 scope begins here k2 = call_cc[get_cc()] if iscontinuation(k2): + # This `x` is local to continuation 2. x = "cont 2 first time" return k2, x + # Still in continuation 2, so this is the `x` of continuation 2. x = "cont 2 second time" return None, x @@ -784,30 +788,34 @@ def f(): # TODO: This breaks the coverage analyzer, because 'name 'x' is assigned to before nonlocal declaration'. # TODO: Fair enough, that's not standard Python. So let's just disable this for now. # with testset("scoping, in presence of nonlocal"): + # # TODO: better example # # It shouldn't matter in this example whether we declare the `x` in the # # continuations `nonlocal`, because once the parent returns, the only # # places that can access its locals *from that activation* are the # # continuation closures *created by that activation*. # with continuations: # def f(): - # # original function scope + # # Original function scope # x = None # - # # continuation 1 scope begins here + # # Continuation 1 scope begins here # # (from the statement following `call_cc` onward, but including the `k1`) # k1 = call_cc[get_cc()] - # nonlocal x + # nonlocal x # <-- IMPORTANT # if iscontinuation(k1): + # # This is now the original `x`. # x = "cont 1 first time" # return k1, x # - # # continuation 2 scope begins here + # # Continuation 2 scope begins here # k2 = call_cc[get_cc()] - # nonlocal x + # nonlocal x # <-- IMPORTANT # if iscontinuation(k2): + # # This too is the original `x`. # x = "cont 2 first time" # return k2, x # + # # Still the original `x`. # x = "cont 2 second time" # return None, x # @@ -822,6 +830,42 @@ def f(): # k2, x = k1(None) # multi-shotting from earlier resume point # test[x == "cont 2 first time"] + # If you want to scope like `nonlocal`, use a box to avoid the need to overwrite the name. + with testset("scoping, using a box"): + # TODO: better example + with continuations: + def f(): + # original function scope + x = box(None) + + # continuation 1 scope begins here + # (from the statement following `call_cc` onward, but including the `k1`) + k1 = call_cc[get_cc()] + if iscontinuation(k1): + # Now there is just one `x`, which is the box; we just update the contents. + x << "cont 1 first time" + return k1, unbox(x) + + # continuation 2 scope begins here + k2 = call_cc[get_cc()] + if iscontinuation(k2): + x << "cont 2 first time" + return k2, unbox(x) + + x << "cont 2 second time" + return None, unbox(x) + + k1, x = f() + test[x == "cont 1 first time"] + k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 + test[x == "cont 2 first time"] + k3, x = k2(None) + test[k3 is None] + test[x == "cont 2 second time"] + + k2, x = k1(None) # multi-shotting from earlier resume point + test[x == "cont 2 first time"] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From 4171ee2a3bf255b7d205e67dce372506d569e947 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 21:48:54 +0200 Subject: [PATCH 678/832] update comment --- unpythonic/syntax/tests/test_conts.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 4a5ec5f8..a7e20a74 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -830,7 +830,11 @@ def f(): # k2, x = k1(None) # multi-shotting from earlier resume point # test[x == "cont 2 first time"] - # If you want to scope like `nonlocal`, use a box to avoid the need to overwrite the name. + # If you need to scope like `nonlocal`, use the classic solution: box the value + # to avoid the need to overwrite the name. + # + # (Classic from before `nonlocal` declarations were a thing; it was added in 3.0, + # see https://www.python.org/dev/peps/pep-3104/ ) with testset("scoping, using a box"): # TODO: better example with continuations: From 46927eb58cb7bdfbe938dc28d075d65f81d69bb1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 21:54:31 +0200 Subject: [PATCH 679/832] wording --- unpythonic/syntax/tests/test_conts.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index a7e20a74..4439ef6c 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -833,8 +833,8 @@ def f(): # If you need to scope like `nonlocal`, use the classic solution: box the value # to avoid the need to overwrite the name. # - # (Classic from before `nonlocal` declarations were a thing; it was added in 3.0, - # see https://www.python.org/dev/peps/pep-3104/ ) + # (Classic from before `nonlocal` declarations were a thing. They were added in 3.0; + # for historical interest, see https://www.python.org/dev/peps/pep-3104/ ) with testset("scoping, using a box"): # TODO: better example with continuations: From dbbdeb846107ced051f2051e874d552d613fd5db Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 22:18:30 +0200 Subject: [PATCH 680/832] wording --- unpythonic/syntax/tests/test_conts.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 4439ef6c..7494ee64 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -789,9 +789,9 @@ def f(): # TODO: Fair enough, that's not standard Python. So let's just disable this for now. # with testset("scoping, in presence of nonlocal"): # # TODO: better example - # # It shouldn't matter in this example whether we declare the `x` in the - # # continuations `nonlocal`, because once the parent returns, the only - # # places that can access its locals *from that activation* are the + # # It shouldn't matter in this particular example whether we declare the `x` + # # in the continuations `nonlocal`, because once the parent returns, the + # # only places that can access its locals *from that activation* are the # # continuation closures *created by that activation*. # with continuations: # def f(): From 56368dbd4305df784c4c74038bbd9e812e23bfb3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 22:18:34 +0200 Subject: [PATCH 681/832] add comment --- unpythonic/syntax/tests/test_conts.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 7494ee64..4af40469 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -750,6 +750,8 @@ def append_stuff_to(lst): test[lst == [1, 2, 3, 4]] with testset("scoping, locals only"): + # This is the cleanest way to scope your local variables in continuations: + # just accept the fact that each continuation introduces a scope boundary. with continuations: def f(): # Original function scope From f772df49299a95ef87f54387b1580354a2101ab6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sun, 30 Jan 2022 22:18:43 +0200 Subject: [PATCH 682/832] improve example --- unpythonic/syntax/tests/test_conts.py | 60 ++++++++++++++++++--------- 1 file changed, 41 insertions(+), 19 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts.py b/unpythonic/syntax/tests/test_conts.py index 4af40469..59c58a6d 100644 --- a/unpythonic/syntax/tests/test_conts.py +++ b/unpythonic/syntax/tests/test_conts.py @@ -832,45 +832,67 @@ def f(): # k2, x = k1(None) # multi-shotting from earlier resume point # test[x == "cont 2 first time"] - # If you need to scope like `nonlocal`, use the classic solution: box the value - # to avoid the need to overwrite the name. + # If you need to scope like `nonlocal`, use the classic solution: box the value, + # so you have no need to overwrite the name; you can replace the thing in the box. # # (Classic from before `nonlocal` declarations were a thing. They were added in 3.0; # for historical interest, see https://www.python.org/dev/peps/pep-3104/ ) with testset("scoping, using a box"): - # TODO: better example with continuations: - def f(): - # original function scope - x = box(None) + # poor man's execution trace + def make_tracing_box_updater(thebox, trace): + def update(value): + trace.append(f"old: {unbox(thebox)}") + thebox << value + trace.append(f"new: {unbox(thebox)}") + return value + return update + + # If we wanted to replace the list instance later, we could pass the list in a box, too. + def f(lst): + # Now there is just one `x`, which is the box; we just update the contents. + # Original function scope + x = box("f") + lst.append(f"initial: {unbox(x)}") + update = make_tracing_box_updater(x, lst) - # continuation 1 scope begins here + # Continuation 1 scope begins here # (from the statement following `call_cc` onward, but including the `k1`) k1 = call_cc[get_cc()] if iscontinuation(k1): - # Now there is just one `x`, which is the box; we just update the contents. - x << "cont 1 first time" - return k1, unbox(x) + return k1, update("k1 first") + update("k1 again") - # continuation 2 scope begins here + # Continuation 2 scope begins here k2 = call_cc[get_cc()] if iscontinuation(k2): - x << "cont 2 first time" - return k2, unbox(x) + return k2, update("k2 first") + update("k2 again") - x << "cont 2 second time" return None, unbox(x) - k1, x = f() - test[x == "cont 1 first time"] + trace = [] + k1, x = f(trace) + test[x == "k1 first"] + test[trace == ['initial: f', 'old: f', 'new: k1 first']] k2, x = k1(None) # when resuming, send `None` as the new value of variable `k1` in continuation 1 - test[x == "cont 2 first time"] + test[x == "k2 first"] + test[trace == ['initial: f', 'old: f', 'new: k1 first', + 'old: k1 first', 'new: k1 again', 'old: k1 again', 'new: k2 first']] k3, x = k2(None) test[k3 is None] - test[x == "cont 2 second time"] + test[x == "k2 again"] + test[trace == ['initial: f', 'old: f', 'new: k1 first', + 'old: k1 first', 'new: k1 again', 'old: k1 again', 'new: k2 first', + 'old: k2 first', 'new: k2 again']] k2, x = k1(None) # multi-shotting from earlier resume point - test[x == "cont 2 first time"] + test[x == "k2 first"] + test[trace == ['initial: f', 'old: f', 'new: k1 first', + 'old: k1 first', 'new: k1 again', 'old: k1 again', 'new: k2 first', + 'old: k2 first', 'new: k2 again', + 'old: k2 again', 'new: k1 again', 'old: k1 again', 'new: k2 first']] + # ^^^^^^^^^^^^^^^ state as left by `k2` before the multi-shot if __name__ == '__main__': # pragma: no cover with session(__file__): From 38fd569eacb49ea175b038af944b2968d6fdc58c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 31 Jan 2022 22:09:24 +0200 Subject: [PATCH 683/832] add an adaptor to iterate over multi-shot generators --- .../syntax/tests/test_conts_multishot.py | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index 4ff0c605..4d6380bf 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -426,6 +426,55 @@ def g(): test[k.func is not k2.func] # ...but different function object instance test_raises[StopIteration, k3()] + with testset("adapting @multishot to Python's generator API"): + class MultishotIterator: + """Adapt a `@multishot` generator to Python's generator API. + + The current continuation is stored as `self.k`. It is read/write. + If you overwrite it with another continuation, the next call to + `next` or `send` will resume from that continuation instead. + + This proof-of-concept demo only supports `iter()`, `next()` and `.send(value)`. + """ + def __init__(self, k): + self.k = k + + # make writes into `self.k` type-check, for fail-fast + def _getk(self): + return self._k + def _setk(self, k): + if not (iscontinuation(k) or (isinstance(k, partial) and iscontinuation(k.func))): + raise TypeError(f"expected `k` to be a continuation or a partially applied continuation, got {k}") + self._k = k + k = property(fget=_getk, fset=_setk, doc="The current continuation. Read/write.") + + # generator API + def __iter__(self): + return self + def __next__(self): + # TODO: Should intercept the `StopIteration` and enter a special closed state, + # TODO: to prevent re-running the last part when `next()` is called for an + # TODO: "already terminated" multi-shot generator. + result = self.k() + if isinstance(result, tuple): + self.k, x = result + else: + self.k, x = result, None + return x + def send(self, value): + result = self.k(value) + if isinstance(result, tuple): + self.k, x = result + else: + self.k, x = result, None + return x + # TODO: Supporting `throw` needs changes to the `@multishot` macro. + # Particularly, when the continuation receives a value, check if it + # is an exception type or exception instance, and if so, raise it. + # basic use + test[[x for x in MultishotIterator(g())] == [1, 2, 3]] + # TODO: advanced example, exercise all features + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() From dbd1ba4aef3e6e8c809539e4c80060807ad7a624 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 31 Jan 2022 23:59:01 +0200 Subject: [PATCH 684/832] improve multi-shot generators See #80. --- .../syntax/tests/test_conts_multishot.py | 252 ++++++++++++------ 1 file changed, 172 insertions(+), 80 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index 4d6380bf..0193ca32 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -1,42 +1,24 @@ # -*- coding: utf-8 -*- """Multi-shot generator demo using the pattern `k = call_cc[get_cc()]`. -This is a barebones implementation, which does not even conform to Python's -generator API. +This is a barebones implementation. -We provide everything in one file, so we use `mcpyrate`'s multi-phase compilation. +We provide everything in one file, so we use `mcpyrate`'s multi-phase compilation +to be able to define the macros in the same module that uses them. Because `with continuations` is a two-pass macro, it will first expand any -`@multishot` inside the block before performing its own processing, which is -exactly what we want. +`@multishot` inside the block before performing its own processing, which +is exactly what we want. We could force the ordering with the metatool +`mcpyrate.metatools.expand_first` that was added in `mcpyrate` 3.6.0, +but we don't need to do that. -We could force the ordering with the metatool `mcpyrate.metatools.expand_first` -added in `mcpyrate` 3.6.0, but we don't need to do that. +We provide a minimal `MultishotIterator` wrapper that makes a `@multishot` +multi-shot generator conform to the most basic parts of Python's generator API. +A full implementation of the generator API would require much more: -Exercise to the reader: - -To make these multi-shot generators support the most basic parts -of the API of Python's native generators, make a wrapper object: - - - `__iter__` on the original function should create the wrapper object - and initialize it. Stash the continuation from the implicit initial - resume point. - - - `__next__` needs a stash for the most recent continuation - per activation of the multi-shot generator. It should run - the most recent continuation (with no arguments) until the next `myield`, - stash the new continuation, and return the myielded value, if any. - - - `send` should send a value into the most recent continuation - (thus resuming). - -A full implementation of the generator API requires much more: - - - `close` - - `throw` - - Think hard on how to handle exceptions. + - There is no `yield from` (delegation); needs a custom `myield_from`. + - Think hard about exception handling. - Particularly, a `yield` inside a `finally` block is a classic catch. - - `yield from` (delegation); needs a custom `myield_from`. """ from mcpyrate.multiphase import macros, phase @@ -55,6 +37,7 @@ import sys from mcpyrate.quotes import macros, q, n, a, h # noqa: F811 + from unpythonic.misc import safeissubclass from unpythonic.syntax import macros, call_cc # noqa: F811 from mcpyrate import namemacro, gensym @@ -93,6 +76,8 @@ def myield_function(tree, syntax, **kw): def multishot(tree, syntax, expander, **kw): """[syntax, block] Make a function into a multi-shot generator. + Only meaningful inside a `with continuations` block. This is not checked. + Multi-shot yield is spelled `myield`. When using `multishot`, be sure to macro-import also `myield`, so that `multishot` knows which name you want to use to refer to the `myield` construct (it is automatically queried @@ -228,6 +213,9 @@ def transform(self, tree): a[var] = h[call_cc][h[get_cc]()] if h[iscontinuation](a[var]): return a[var], a[value] + # For `throw` support: if we are sent an exception instance or class, raise it. + elif isinstance(a[var], BaseException) or h[safeissubclass](a[var], BaseException): + raise a[var] return quoted # `k = myield` @@ -239,6 +227,8 @@ def transform(self, tree): a[var] = h[call_cc][h[get_cc]()] if h[iscontinuation](a[var]): return a[var] + elif isinstance(a[var], BaseException) or h[safeissubclass](a[var], BaseException): + raise a[var] return quoted # `myield[value]` @@ -249,6 +239,11 @@ def transform(self, tree): a[var] = h[call_cc][h[get_cc]()] if h[iscontinuation](a[var]): return h[partial](a[var], None), a[value] + # For `throw` support: `MultishotIterator` digs the `.func` from inside the `partial` + # to force a send, even though this variant of `myield` cannot receive a value by + # a normal `send`. + elif isinstance(a[var], BaseException) or h[safeissubclass](a[var], BaseException): + raise a[var] return quoted # `myield` @@ -258,11 +253,13 @@ def transform(self, tree): a[var] = h[call_cc][h[get_cc]()] if h[iscontinuation](a[var]): return h[partial](a[var], None) + elif isinstance(a[var], BaseException) or h[safeissubclass](a[var], BaseException): + raise a[var] return quoted return self.generic_visit(tree) - class ReturnToStopIterationTransformer(ASTTransformer): + class ReturnToRaiseStopIterationTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): # do not recurse into hygienic captures return tree @@ -277,7 +274,7 @@ def transform(self, tree): with q as quoted: raise h[StopIteration] return quoted - # `return value` + # `return expr` with q as quoted: raise h[StopIteration](a[tree.value]) return quoted @@ -296,12 +293,12 @@ def transform(self, tree): with q as quoted: return tree.body.extend(quoted) - tree.body = ReturnToStopIterationTransformer().visit(tree.body) + tree.body = ReturnToRaiseStopIterationTransformer().visit(tree.body) # Inject a bare `myield` resume point at the beginning of the function body. # This makes the resulting function work somewhat like a Python generator. # When initially called, the arguments are bound, and you get a continuation; - # then resuming that continuation starts the actual computation. + # then resuming that continuation actually starts executing the function body. tree.body.insert(0, ast.Expr(value=ast.Name(id=names_of_myield[0]))) # Transform multishot yields (`myield`) into `call_cc`. @@ -313,6 +310,144 @@ def transform(self, tree): # macro-import from higher phase; we're now in phase 0 from __self__ import macros, multishot, myield # noqa: F811, F401 +class MultishotIterator: + """Adapt a `@multishot` generator to Python's generator API. + + Example:: + + with continuations: + @multishot + def g(): + myield[1] + myield[2] + myield[3] + + # Instantiating the multi-shot generator returns a continuation; + # we can send that into a `MultishotIterator`. The resulting iterator + # behaves almost like a standard generator. + mi = MultishotIterator(g()) + assert [x for x in mi] == [1, 2, 3] + + `k`: A continuation, or a partially applied continuation + (e.g. one that does not usefully expect a value; + an `myield` with no assignment target will return such). + + The initial continuation to start execution from. + + Each `next` or `.send` will call the current `self.k`, and then overwrite + `self.k` with the new continuation returned by the multi-shot generator. + If the multi-shot generator raises `StopIteration` (so there is no new + continuation), the `MultishotIterator` marks itself as closed, and re-raises. + + The current continuation is stored as `self.k`. It is read/write, + type-checked at write time. + + If you overwrite `self.k` with another continuation, the next call + to `next` or `.send` will resume from that continuation instead. + If the iterator was closed, overwriting `self.k` will re-open it. + + This proof-of-concept demo only supports a subset of the generator API: + + - `iter(mi)` + - `next(mi)`, + - `mi.send(value)` + - `mi.throw(exc)` + - `mi.close()` + + where `mi` is a `MultishotIterator` instance. + """ + def __init__(self, k): + self.k = k + self._closed = False + + # make writes into `self.k` type-check, for fail-fast + def _getk(self): + return self._k + def _setk(self, k): + if not (iscontinuation(k) or (isinstance(k, partial) and iscontinuation(k.func))): + raise TypeError(f"expected `k` to be a continuation or a partially applied continuation, got {k}") + self._k = k + self._closed = False + k = property(fget=_getk, fset=_setk, doc="The current continuation. Read/write.") + + # Internal method that implements `next` and `.send`. + def _advance(self, mode, value=None): + assert mode in ("next", "send") + if self._closed: + raise StopIteration + # Intercept possible `StopIteration` and enter the closed + # state, to prevent re-running the last continuation (that + # raised `StopIteration`) when `next()` is called again. + try: + if mode == "next": + result = self.k() + else: # mode == "send" + result = self.k(value) + except StopIteration: # no new continuation + self._closed = True + raise + if isinstance(result, tuple): + self.k, x = result + else: + self.k, x = result, None + return x + + # generator API + def __iter__(self): + return self + def __next__(self): + return self._advance("next") + def send(self, value): + return self._advance("send", value) + + # The `throw` and `close` methods are not so useful as with regular + # generators, due to there being no concept of paused execution. + # + # The continuation is a separate nested closure, and it is not + # possible to usefully straddle a `try` or `with` across the + # boundary. + # + # For example, `with` only takes effect whenever it is "entered + # from the top", and it will release the context as soon as the + # multi-shot generator `myield`s the continuation. + # + # `throw` pretty much just enters the continuation function, and + # makes it raise an exception; in true multi-shot fashion, the same + # continuation can still be resumed later (also without making it + # raise that time). + # + # `close` is only useful in that closing makes the multi-shot generator + # reject any further attempts to `next` or `.send` (unless you then + # overwrite the continuation manually). + # + # For an example of what serious languages that have `call_cc` do, see + # Racket's `dynamic-wind` construct ("wind" as in "winding/unwinding the call stack"). + # It's the supercharged big sister of Python's `with` construct that accounts for + # execution topologies where control may leave the block, and then suddenly return + # to the middle of it later (most often due to the invocation of a continuation + # that was created inside that block). + # https://docs.racket-lang.org/reference/cont.html#%28def._%28%28quote._~23~25kernel%29._dynamic-wind%29%29 + def throw(self, exc): + # If we are stopped at an `myield` that has no assignment target, so + # that it normally does not expect a value, we unwrap the original + # continuation from the `partial` to force-send the exception. + k = self.k.func if isinstance(self.k, partial) else self.k + k(exc) + + # https://stackoverflow.com/questions/60137570/explanation-of-generator-close-with-exception-handling + def close(self): + if self._closed: + return + self._closed = True + try: + self.throw(GeneratorExit) + except GeneratorExit: + return # ok! + # Any other exception is propagated. + else: # No exception means that the generator is trying to yield something. + raise RuntimeError("@multishot generator attempted to `myield` a value while it was being closed") + + def runtests(): # To start with, here's a sketch of what we want to do. with testset("multi-shot generators with the pattern call_cc[get_cc()]"): @@ -395,7 +530,7 @@ def f(): k2, x2 = k1() test[x2 == 42] - # The first example rewritten to use the macro. + # The first example rewritten to use the macro: with testset("multi-shot generators with @multishot"): with continuations: @multishot @@ -426,51 +561,8 @@ def g(): test[k.func is not k2.func] # ...but different function object instance test_raises[StopIteration, k3()] - with testset("adapting @multishot to Python's generator API"): - class MultishotIterator: - """Adapt a `@multishot` generator to Python's generator API. - - The current continuation is stored as `self.k`. It is read/write. - If you overwrite it with another continuation, the next call to - `next` or `send` will resume from that continuation instead. - - This proof-of-concept demo only supports `iter()`, `next()` and `.send(value)`. - """ - def __init__(self, k): - self.k = k - - # make writes into `self.k` type-check, for fail-fast - def _getk(self): - return self._k - def _setk(self, k): - if not (iscontinuation(k) or (isinstance(k, partial) and iscontinuation(k.func))): - raise TypeError(f"expected `k` to be a continuation or a partially applied continuation, got {k}") - self._k = k - k = property(fget=_getk, fset=_setk, doc="The current continuation. Read/write.") - - # generator API - def __iter__(self): - return self - def __next__(self): - # TODO: Should intercept the `StopIteration` and enter a special closed state, - # TODO: to prevent re-running the last part when `next()` is called for an - # TODO: "already terminated" multi-shot generator. - result = self.k() - if isinstance(result, tuple): - self.k, x = result - else: - self.k, x = result, None - return x - def send(self, value): - result = self.k(value) - if isinstance(result, tuple): - self.k, x = result - else: - self.k, x = result, None - return x - # TODO: Supporting `throw` needs changes to the `@multishot` macro. - # Particularly, when the continuation receives a value, check if it - # is an exception type or exception instance, and if so, raise it. + # Using a `@multishot` as if it was a standard generator: + with testset("MultishotIterator: adapting @multishot to Python's generator API"): # basic use test[[x for x in MultishotIterator(g())] == [1, 2, 3]] # TODO: advanced example, exercise all features From f6cdf2009068a6b471533fe9282aff60776625cb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 1 Feb 2022 01:29:15 +0200 Subject: [PATCH 685/832] use `isnewscope` --- unpythonic/syntax/tests/test_conts_multishot.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index 0193ca32..8b416ace 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -46,6 +46,7 @@ from mcpyrate.walkers import ASTTransformer from unpythonic.syntax import get_cc, iscontinuation + from unpythonic.syntax.scopeanalyzer import isnewscope def myield_function(tree, syntax, **kw): """[syntax, name/expr] Yield from a multi-shot generator. @@ -197,9 +198,7 @@ class MultishotYieldTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): # do not recurse into hygienic captures return tree - # respect scope boundaries - if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, - ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): + if isnewscope(tree): # respect scope boundaries return tree # `k = myield[value]` @@ -263,9 +262,7 @@ class ReturnToRaiseStopIterationTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): # do not recurse into hygienic captures return tree - # respect scope boundaries - if type(tree) in (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, - ast.ListComp, ast.DictComp, ast.SetComp, ast.GeneratorExp): + if isnewscope(tree): # respect scope boundaries return tree if type(tree) is ast.Return: From 4347b2c2abbce819d8b4eb04c7e381581e2f5ccb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 1 Feb 2022 01:44:25 +0200 Subject: [PATCH 686/832] extend multi-shot generator tests --- .../syntax/tests/test_conts_multishot.py | 27 ++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index 8b416ace..bb311dab 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -562,7 +562,32 @@ def g(): with testset("MultishotIterator: adapting @multishot to Python's generator API"): # basic use test[[x for x in MultishotIterator(g())] == [1, 2, 3]] - # TODO: advanced example, exercise all features + + # Re-using `g` from above: + mig = MultishotIterator(g()) + test[next(mig) == 1] + k = mig.k # stash the current continuation tracked by the `MultishotIterator` + test[next(mig) == 2] + test[next(mig) == 3] + mig.k = k # multi-shot: rewind to the point we stashed + test[next(mig) == 2] + test[next(mig) == 3] + + # Re-using `f` from above: + mif = MultishotIterator(f()) + test[next(mif) is None] + k = mif.k + test[next(mif) == 42] + test[next(mif) is None] + test[mif.send(23) == 42] + test_raises[StopIteration, mif.send(17)] + mif.k = k # rewind + test[next(mif) == 42] + test[next(mif) is None] + test[mif.send(23) == 42] + test_raises[StopIteration, mif.send(17)] + + # TODO: advanced examples, exercise all features if __name__ == '__main__': # pragma: no cover with session(__file__): From ef4656ececb58965cd0bb83877a111e57de46c6a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 1 Feb 2022 01:46:19 +0200 Subject: [PATCH 687/832] mark TODO in multi-shot generators --- unpythonic/syntax/tests/test_conts_multishot.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index bb311dab..db77e591 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -367,6 +367,8 @@ def _setk(self, k): self._closed = False k = property(fget=_getk, fset=_setk, doc="The current continuation. Read/write.") + # TODO: For thread safety, we should lock writes to `self._closed`, + # TODO: as well as make `_advance` behave atomically. # Internal method that implements `next` and `.send`. def _advance(self, mode, value=None): assert mode in ("next", "send") From f2a2e6f250d9200e4699c2973bd4e011997df555 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 28 Apr 2022 15:29:14 +0300 Subject: [PATCH 688/832] ETAEstimator: fix bug with "unknown" ETA when all items done --- unpythonic/timeutil.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/timeutil.py b/unpythonic/timeutil.py index abec77cb..075fd1c1 100644 --- a/unpythonic/timeutil.py +++ b/unpythonic/timeutil.py @@ -113,7 +113,7 @@ def _elapsed(self) -> float: def _formatted_eta(self) -> str: elapsed = self.elapsed estimate = self.estimate - if estimate: + if estimate is not None: total = elapsed + estimate formatted_estimate = format_human_time(estimate) formatted_total = format_human_time(total) From 7b4da3e9a3a7a68a7e9c4248f806f5f24df02a3c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 24 Oct 2022 10:47:30 +0300 Subject: [PATCH 689/832] document a numerical differentiation trick (complex Taylor series) --- unpythonic/tests/test_fpnumerics.py | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/unpythonic/tests/test_fpnumerics.py b/unpythonic/tests/test_fpnumerics.py index a6751ae6..6cd44933 100644 --- a/unpythonic/tests/test_fpnumerics.py +++ b/unpythonic/tests/test_fpnumerics.py @@ -5,7 +5,7 @@ Based on various sources; links provided in the source code comments. """ -from ..syntax import macros, test # noqa: F401 +from ..syntax import macros, test, warn # noqa: F401 from ..test.fixtures import session, testset, returns_normally from operator import add, mul @@ -193,6 +193,30 @@ def best_differentiate_with_tol(h0, f, x, eps): # Thanks to super_improve, this actually requires taking only three terms. test[abs(best_differentiate_with_tol(0.1, sin, pi / 2, 1e-8)) < 1e-11] + # This is strictly speaking not FP, but it is worth noting that + # numerical derivatives of real-valued functions can also be estimated + # using a not very well known trick based on complex numbers. + # + # Consider the Taylor series + # f(x + iε) = f(x) + i ε f'(x) + O(ε²) + # Therefore + # real(f(x + iε)) = f(x) + O(ε²) + # imag(f(x + iε) / ε) = f'(x) + # No cancellation, so we can take a really small ε (e.g. ε = 1e-150). + # + # This comes from Goodfellow, Bengio and Courville (2016): Deep Learning, MIT press, p. 434: + # https://www.deeplearningbook.org/contents/guidelines.html + try: + # We need a `sin` that can handle complex numbers, so stdlib's won't cut the mustard. + import numpy as np + eps = 1e-150 + def complex_diff(f, x): + return np.imag((f(x + eps * 1j) / eps)) + # This is so accurate in this simple case that we can test for floating point equality. + test[complex_diff(np.sin, 0.1) == np.cos(0.1)] + except ImportError: + warn["Could not import NumPy; alternative numerical differentiation test skipped."] + # pi approximation with Euler series acceleration # # See SICP, 2nd ed., sec. 3.5.3. From d15adf467b2431a768a063ef438382cd1d026485 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 24 Oct 2022 11:04:12 +0300 Subject: [PATCH 690/832] add original citations for complex differentiation trick --- unpythonic/tests/test_fpnumerics.py | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/unpythonic/tests/test_fpnumerics.py b/unpythonic/tests/test_fpnumerics.py index 6cd44933..e745db57 100644 --- a/unpythonic/tests/test_fpnumerics.py +++ b/unpythonic/tests/test_fpnumerics.py @@ -197,15 +197,30 @@ def best_differentiate_with_tol(h0, f, x, eps): # numerical derivatives of real-valued functions can also be estimated # using a not very well known trick based on complex numbers. # - # Consider the Taylor series + # Let f be a complex analytic function (or a complex analytic piece of a piecewise defined + # function) that takes on real values for inputs on the real line. Consider the Taylor series # f(x + iε) = f(x) + i ε f'(x) + O(ε²) - # Therefore + # where x is a real number, i = √-1, and ε is a small real number. We have # real(f(x + iε)) = f(x) + O(ε²) # imag(f(x + iε) / ε) = f'(x) + # This gives us both f(x) and f'(x) with one complex-valued computation. # No cancellation, so we can take a really small ε (e.g. ε = 1e-150). # - # This comes from Goodfellow, Bengio and Courville (2016): Deep Learning, MIT press, p. 434: - # https://www.deeplearningbook.org/contents/guidelines.html + # This comes from + # Goodfellow, Bengio and Courville (2016): Deep Learning, MIT press, p. 434: + # https://www.deeplearningbook.org/contents/guidelines.html + # who cite it to originate from + # William Squire and George Trapp (1998). Using Complex Variables to Estimate Derivatives + # of Real Functions. SIAM Review, 40(1), 110-112. http://doi.org/10.1137/S003614459631241X + # who, in turn, cite it to originate from + # J. N. Lyness and C. B. Moler. 1967. Numerical differentiation of analytic functions, + # SIAM J. Numer. Anal., 4, pp. 202–210. + # and + # J. N. Lyness. 1967. Numerical algorithms based on the theory of complex variables, + # Proc. ACM 22nd Nat. Conf., Thompson Book Co., Washington, DC, pp. 124–134. + # + # So this technique has been known since the late 1960s, but even as of this writing, + # 55 years later (2022), it has not seen much use. try: # We need a `sin` that can handle complex numbers, so stdlib's won't cut the mustard. import numpy as np From 53e1d7dfc86ad11c7bb12251f3f067564a8b05ff Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 25 Oct 2022 12:08:06 +0300 Subject: [PATCH 691/832] improve complex differentiation trick example --- unpythonic/tests/test_fpnumerics.py | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/unpythonic/tests/test_fpnumerics.py b/unpythonic/tests/test_fpnumerics.py index e745db57..e7f8eea3 100644 --- a/unpythonic/tests/test_fpnumerics.py +++ b/unpythonic/tests/test_fpnumerics.py @@ -10,7 +10,8 @@ from operator import add, mul from itertools import repeat -from math import sin, pi, log2 +from math import sin, cos, pi, log2 +from cmath import sin as complex_sin from ..fun import curry from ..funutil import Values @@ -219,18 +220,19 @@ def best_differentiate_with_tol(h0, f, x, eps): # J. N. Lyness. 1967. Numerical algorithms based on the theory of complex variables, # Proc. ACM 22nd Nat. Conf., Thompson Book Co., Washington, DC, pp. 124–134. # + # See also + # Joaquim J Martins, Peter Sturdza, Juan J Alonso. The complex-step derivative approximation. + # ACM Transactions on Mathematical Software, Association for Computing Machinery, 2003, 29, + # pp.245-262. 10.1145/838250.838251. hal-01483287. + # https://hal.archives-ouvertes.fr/hal-01483287/document + # # So this technique has been known since the late 1960s, but even as of this writing, # 55 years later (2022), it has not seen much use. - try: - # We need a `sin` that can handle complex numbers, so stdlib's won't cut the mustard. - import numpy as np - eps = 1e-150 - def complex_diff(f, x): - return np.imag((f(x + eps * 1j) / eps)) - # This is so accurate in this simple case that we can test for floating point equality. - test[complex_diff(np.sin, 0.1) == np.cos(0.1)] - except ImportError: - warn["Could not import NumPy; alternative numerical differentiation test skipped."] + eps = 1e-150 + def complex_diff(f, x): + return (f(x + eps * 1j) / eps).imag + # This is so accurate in this simple case that we can test for floating point equality. + test[complex_diff(complex_sin, 0.1) == cos(0.1)] # pi approximation with Euler series acceleration # From b0573f3987fd7c208a3dc66fc531fed589a62b21 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Nov 2022 14:22:40 +0200 Subject: [PATCH 692/832] add note about get_cc --- doc/essays.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/essays.md b/doc/essays.md index 696af287..df4282c8 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -51,7 +51,7 @@ On a point raised [here by the BDFL](https://www.artima.com/weblogs/viewpost.jsp It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose `lambda x: [expr0, expr1, ...]` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) -As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. +As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired. The solution to *that* issue is `let/cc`, which in `unpythonic`, becomes `k = call_cc[get_cc()]`.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. Finally, there is the issue of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/)). It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its user, and it is not very popular in the programming community at large. From 473d22ad934e6bcc8a9fc5e633b8551983cfb8cd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Nov 2022 14:25:09 +0200 Subject: [PATCH 693/832] link anagram.py --- doc/essays.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/essays.md b/doc/essays.md index df4282c8..cd514966 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -51,7 +51,7 @@ On a point raised [here by the BDFL](https://www.artima.com/weblogs/viewpost.jsp It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose `lambda x: [expr0, expr1, ...]` will have to do for a multi-expression lambda. Unless I decide at some point to make a source filter for [`mcpyrate`](https://github.com/Technologicat/mcpyrate) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. (And I do not want to maintain a [Pyparsing](https://github.com/pyparsing/pyparsing) grammar to parse a modified version of Python.) -As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired. The solution to *that* issue is `let/cc`, which in `unpythonic`, becomes `k = call_cc[get_cc()]`.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your anagram-making algorithm only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. +As for true multi-shot continuations, `unpythonic.syntax` has `with continuations` for that, but I am not sure if I will ever use it in production code. Most of the time, it seems to me full continuations are a solution looking for a problem. (A very elegant solution, even if the usability of the `call/cc` interface leaves much to be desired. The solution to *that* issue is `let/cc`, which in `unpythonic`, becomes `k = call_cc[get_cc()]`.) For everyday use, one-shot continuations (a.k.a. resumable functions, a.k.a. generators in Python) are often all that is needed to simplify certain patterns, especially those involving backtracking. I am a big fan of the idea that, for example, you can make your [anagram-making algorithm](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/anagram.py) only yield valid anagrams, with the backtracking state (to eliminate dead-ends) implicitly stored in the paused generator! However, having multi-shot continuations is great for teaching the concept of continuations in a programming course, when teaching in Python. Finally, there is the issue of implicitly encouraging subtly incompatible Python-like languages (see the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/)). It is pretty much the point of language-level extensibility, to allow users to do that if they want. I would not worry about it. Racket is *designed* for extensibility, and its community seems to be doing just fine - they even *encourage* the creation of new languages to solve problems. On the other hand, Racket demands some sophistication on the part of its user, and it is not very popular in the programming community at large. From 0db5fb1b2651ffff57f09e0bc07f8e61a4dbe945 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Nov 2022 14:25:33 +0200 Subject: [PATCH 694/832] update last updated --- doc/essays.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/essays.md b/doc/essays.md index cd514966..24928ff3 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -25,7 +25,7 @@ For now, essays are listed in chronological order, most recent last. # What Belongs in Python? -*Originally written in 2020; updated 9 June 2021.* +*Originally written in 2020; updated 9 June 2021; small update 16 November 2022.* You may feel that [my hovercraft is full of eels](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html). It is because they come with the territory. From 4f85957bf64e1b786da0679eade3fe602793ceee Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Nov 2022 14:30:52 +0200 Subject: [PATCH 695/832] mention Jupyter --- doc/essays.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/essays.md b/doc/essays.md index 24928ff3..245519ca 100644 --- a/doc/essays.md +++ b/doc/essays.md @@ -62,7 +62,7 @@ For general programming in the early 2020s, Python still has the ecosystem advan # Common Lisp, Python, and productivity -*Originally written in 2020; updated 9 June 2021.* +*Originally written in 2020; updated 9 June 2021; small update 16 November 2022.* The various essays Paul Graham wrote near the turn of the millennium, especially [Revenge of the Nerds (2002)](http://paulgraham.com/icad.html), have given the initial impulse to many programmers for studying Lisp. The essays are well written and have provided a lot of exposure for the Lisp family of languages. So how does the programming world look in that light now, 20 years later? @@ -78,7 +78,7 @@ As for productivity, [it may be](https://medium.com/smalltalk-talk/lisp-smalltal Haskell aims at code-data equivalence from a third angle (memoized pure functions are in essence infinite lookup tables), but I have not used it in practice, so I do not have the experience to say whether this is enough to make it feel powerful in a similar way. -Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world is not that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead - without restarting the whole app at each change. Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. In web applications, [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a small step in a somewhat similar direction (as long as one can restart the server app easily, to make it use the latest definitions). +Image-based programming (live programming) is a common factor between Pharo and Common Lisp + Swank. This is another productivity booster that much of the programming world is not that familiar with. It eliminates not only the edit/compile/restart cycle, but the edit/restart cycle as well, making the workflow a concurrent *edit/run* instead - without restarting the whole app at each change. Julia has [Revise.jl](https://github.com/timholy/Revise.jl) for something similar. In web applications, [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a small step in a somewhat similar direction (as long as one can restart the server app easily, to make it use the latest definitions). Notebooks (such as [Jupyter](https://jupyter.org/)) provide the edit/run paradigm for scientific scripts. But to know exactly what Common Lisp has to offer, **yes**, it does make sense to learn it. As baroque as some parts are, there are a lot of great ideas there. [Conditions](http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html) are one. [CLOS](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) is another. (Nowadays [Julia](https://docs.julialang.org/en/v1/manual/methods/) has CLOS-style [multiple-dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/).) More widely, in the ecosystem, Swank is one. From a53947ef3dd0ee7d99e98c19d284cb1230381704 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 19 Sep 2024 14:30:28 +0300 Subject: [PATCH 696/832] syntax-highlight source code in test output --- unpythonic/syntax/testingtools.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 116f4bc4..73fcad01 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -823,7 +823,7 @@ def _test_expr(tree): # For this reason, we provide `with expand_testing_macros_first`, which # in itself is a code-walking block macro, whose only purpose is to force # `test[]` and its sisters to expand first.) - sourcecode = unparse(tree) + sourcecode = unparse(tree, color=True, expander=dyn._macro_expander) envname = gensym("e") # for injecting the captured value @@ -866,7 +866,7 @@ def _record_value(envname, sourcecode, value): def _inject_value_recorder(envname, tree): # wrap tree with the the[] handler recorder = q[h[_record_value]] # TODO: stash hygienic value? return q[a[recorder](n[envname], - u[unparse(tree)], + u[unparse(tree, color=True, expander=dyn._macro_expander)], a[tree])] def _transform_important_subexpr(tree, envname): # The the[] mark mechanism is invoked outside-in, because for reporting, @@ -915,7 +915,7 @@ def _test_expr_signals_or_raises(tree, syntaxname, asserter): raise SyntaxError(f"Expected one of {syntaxname}[exctype, expr], {syntaxname}[exctype, expr, message]") # pragma: no cover # Same remark about outside-in source code capture as in `_test_expr`. - sourcecode = unparse(tree) + sourcecode = unparse(tree, color=True, expander=dyn._macro_expander) # Name our lambda to make the stack trace more understandable. # For consistency, the name matches that used by `_test_expr`. @@ -952,7 +952,7 @@ def _test_block(block_body, args): raise SyntaxError('Expected `with test:` or `with test[message]:`') # pragma: no cover # Same remark about outside-in source code capture as in `_test_expr`. - sourcecode = unparse(block_body) + sourcecode = unparse(block_body, color=True, expander=dyn._macro_expander) envname = gensym("e") # for injecting the captured value @@ -1024,7 +1024,7 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): raise SyntaxError(f'Expected `with {syntaxname}(exctype):` or `with {syntaxname}[exctype, message]:`') # pragma: no cover # Same remark about outside-in source code capture as in `_test_expr`. - sourcecode = unparse(block_body) + sourcecode = unparse(block_body, color=True, expander=dyn._macro_expander) testblock_function_name = gensym("_test_block") thetest = q[(a[asserter])(a[exctype], From aca4c40e6129ed87471bcd26e17e5d16ad457764 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 19 Sep 2024 14:30:41 +0300 Subject: [PATCH 697/832] unpythonic.env.env: add pickle support --- unpythonic/env.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/unpythonic/env.py b/unpythonic/env.py index eac43868..d5538e4a 100644 --- a/unpythonic/env.py +++ b/unpythonic/env.py @@ -55,9 +55,16 @@ class env: "_direct_write", "_reserved_names") _direct_write = ("_env", "_finalized") + # For pickle support, since unpickling calls `__new__` but not `__init__`. + # If `self._env` is not present, `__getattr__` will crash with an infinite loop. So create it as early as possible. + def __new__(cls, **kwargs): + instance = super().__new__(cls) + instance._env = {} + instance._finalized = False # "let" sets this once env setup done + instance.__init__(**kwargs) + return instance + def __init__(self, **bindings): - self._env = {} - self._finalized = False # "let" sets this once env setup done for name, value in bindings.items(): setattr(self, name, value) From 5067ef4c1d7d95b9f0c8d6c242c42baf852dd57f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 19 Sep 2024 14:46:36 +0300 Subject: [PATCH 698/832] update CHANGELOG --- CHANGELOG.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 39b80951..34ecadff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,27 @@ -**0.15.2** (in progress, last updated 28 January 2022) +**0.15.3** (in progress, last updated 19 September 2024) *No user-visible changes yet.* +--- + +**0.15.2** (19 September 2024) + +This time, just a small but important fix. + +**Fixed**: + +- `unpythonic.env.env` is now pickleable. Save your fancy bunches into `.pickle` files and load them back! + +**Future plans**: + +Contrary to appearances, this project is not dead. But it already does most of what I personally need it to do, so it is pretty much in maintenance mode. And it has not required much maintenance over the past two years. + +We still plan to officially support Python 3.11+ later, as well as to update all constructs with assignment semantics to use the more appropriate `:=` operator, when/if I find the time to do so. The syntax uses `<<` for historical reasons - these constructs were originally implemented in 2018, on Python 3.4, back when `:=` did not exist. + +The most likely upgrade timeframe is when I personally switch to Python 3.11+, and something breaks. That is also when I'll likely next upgrade the sister project `mcpyrate`. + + --- **0.15.1** (28 January 2022) - *New Year's edition*: From 59b08967549e9072a5c23934171bd18e47171891 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 19 Sep 2024 15:01:03 +0300 Subject: [PATCH 699/832] pre-emptive version bump --- unpythonic/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 2b0f5ad4..c9701cb3 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.15.2' +__version__ = '0.15.3' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From 08b521a1785689b8ee599ea9765339eb222eef4a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 25 Sep 2024 17:28:25 +0300 Subject: [PATCH 700/832] require Python 3.8+ --- CHANGELOG.md | 8 ++++++-- README.md | 2 +- setup.py | 2 -- unpythonic/syntax/astcompat.py | 5 +++-- unpythonic/syntax/tests/test_lambdatools.py | 7 +++---- 5 files changed, 13 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34ecadff..0a4884dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,10 @@ -**0.15.3** (in progress, last updated 19 September 2024) +**0.15.3** (in progress, last updated 25 September 2024) -*No user-visible changes yet.* +**IMPORTANT**: + +- Minimum Python language version is now 3.8. +- Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. Code has not been fully cleaned of historical cruft yet, so parts of it may still work in these versions. +- 3.8 becomes EOL after October 2024, so support for that version might be dropped soon, too. --- diff --git a/README.md b/README.md index 8b5414c8..49a97a31 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ None required. - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -The 0.15.x series should run on CPython 3.6, 3.7, 3.8, 3.9 and 3.10, and PyPy3 (language versions 3.6, 3.7 and 3.8); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +The 0.15.x series should run on CPython 3.8, 3.9 and 3.10, and PyPy3 (language version 3.8); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation diff --git a/setup.py b/setup.py index 75575e8a..d95e890f 100644 --- a/setup.py +++ b/setup.py @@ -90,8 +90,6 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa "Operating System :: POSIX :: Linux", "Programming Language :: Python", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.6", - "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", diff --git a/unpythonic/syntax/astcompat.py b/unpythonic/syntax/astcompat.py index 4dd444e7..32cbc4e1 100644 --- a/unpythonic/syntax/astcompat.py +++ b/unpythonic/syntax/astcompat.py @@ -25,8 +25,9 @@ NamedExpr = _NoSuchNodeType # No new AST node types in Python 3.9. - -# TODO: any new AST node types in Python 3.10? (release expected in October 2021) +# No new AST node types in Python 3.10. +# TODO: Any new AST node types in Python 3.11? +# TODO: Any new AST node types in Python 3.12? # -------------------------------------------------------------------------------- # Deprecated AST node types diff --git a/unpythonic/syntax/tests/test_lambdatools.py b/unpythonic/syntax/tests/test_lambdatools.py index 8a4ad43b..0f1a4d18 100644 --- a/unpythonic/syntax/tests/test_lambdatools.py +++ b/unpythonic/syntax/tests/test_lambdatools.py @@ -57,10 +57,9 @@ def runtests(): foo = let[[f7 << (lambda x: x)] in f7] # let-binding: name as "f7" # noqa: F821 test[foo.__name__ == "f7"] - warn["NamedExpr test currently disabled for syntactic compatibility with Python 3.6 and 3.7."] - # if foo2 := (lambda x: x): # NamedExpr a.k.a. walrus operator (Python 3.8+) - # pass - # test[foo2.__name__ == "foo2"] + if foo2 := (lambda x: x): # NamedExpr a.k.a. walrus operator (Python 3.8+) + pass + test[foo2.__name__ == "foo2"] # function call with named arg def foo(func1, func2): From 7889f7f004a9b6ba645fcd3681fdc8fe54c15fee Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 25 Sep 2024 17:28:36 +0300 Subject: [PATCH 701/832] WIP: walrus syntax for env-assignment (almost complete) --- CHANGELOG.md | 4 + unpythonic/syntax/letdo.py | 122 ++++++++++---------- unpythonic/syntax/letdoutil.py | 126 +++++++++++++-------- unpythonic/syntax/tests/test_letdo.py | 92 +++++++++++++-- unpythonic/syntax/tests/test_letdoutil.py | 130 +++++++++++++++++++++- 5 files changed, 360 insertions(+), 114 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a4884dd..8fee44da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ **0.15.3** (in progress, last updated 25 September 2024) +**New**: + +- Walrus syntax `name := value` is now supported, and preferred, for all env-assignments. Old syntax `name << value` still works, and will remain working at least until v0.16.0, whenever that is. + **IMPORTANT**: - Minimum Python language version is now 3.8. diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index ee007f1e..e81ba13a 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -92,7 +92,7 @@ def where(tree, *, syntax, **kw): Usage:: - let[body, where[k0 << v0, ...]] + let[body, where[k0 := v0, ...]] Only meaningful for declaring the bindings in a let-where, for all expression-form let constructs: `let`, `letseq`, `letrec`, `let_syntax`, @@ -100,7 +100,7 @@ def where(tree, *, syntax, **kw): """ if syntax != "name": raise SyntaxError("where (unpythonic.syntax.letdo.where) is a name macro only") # pragma: no cover - raise SyntaxError("where (unpythonic.syntax.letdo.where) is only meaningful in a let[body, where[k0 << v0, ...]]") # pragma: no cover + raise SyntaxError("where (unpythonic.syntax.letdo.where) is only meaningful in a let[body, where[k0 := v0, ...]]") # pragma: no cover @parametricmacro def let(tree, *, args, syntax, expander, **kw): @@ -110,18 +110,18 @@ def let(tree, *, args, syntax, expander, **kw): Usage:: - let[k0 << v0, ...][body] - let[k0 << v0, ...][[body0, ...]] + let[k0 := v0, ...][body] + let[k0 := v0, ...][[body0, ...]] where ``body`` is an expression. The names bound by ``let`` are local; they are available in ``body``, and do not exist outside ``body``. Alternative haskelly syntax is also available:: - let[[k0 << v0, ...] in body] - let[[k0 << v0, ...] in [body0, ...]] - let[body, where[k0 << v0, ...]] - let[[body0, ...], where[k0 << v0, ...]] + let[[k0 := v0, ...] in body] + let[[k0 := v0, ...] in [body0, ...]] + let[body, where[k0 := v0, ...]] + let[[body0, ...], where[k0 := v0, ...]] For a body with multiple expressions, use an extra set of brackets, as shown above. This inserts a ``do``. Only the outermost extra brackets @@ -133,9 +133,14 @@ def let(tree, *, args, syntax, expander, **kw): Each ``name`` in the same ``let`` must be unique. - Rebinding of let-bound variables inside `body` is supported with `unpythonic` - env-assignment syntax, ``x << 42``. This is an expression, performing the - assignment, and returning the new value. + Starting at v0.15.3, rebinding of let-bound variables inside `body` + is supported using the walrus assignment syntax, ``x := 42``. + The new syntax is preferred, but the old one is still available + for backward compatibility. + + From v0.15.0 to v0.15.2, rebinding of let-bound variables inside `body` + is supported with `unpythonic` env-assignment syntax, ``x << 42``. + This is an expression, performing the assignment, and returning the new value. In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see ``do`` for details. @@ -210,9 +215,9 @@ def dlet(tree, *, args, syntax, expander, **kw): Example:: - @dlet[x << 0] + @dlet[x := 0] def count(): - x << x + 1 + (x := x + 1) # walrus requires parens here; or use `x << x + 1` return x assert count() == 1 assert count() == 2 @@ -222,7 +227,7 @@ def count(): ``let`` environment *for the entirety of that lexical scope*. (This is modeled after Python's standard scoping rules.) - **CAUTION**: assignment to the let environment is ``name << value``; + **CAUTION**: assignment to the let environment is ``name := value``; the regular syntax ``name = value`` creates a local variable in the lexical scope of the ``def``. """ @@ -240,9 +245,9 @@ def dletseq(tree, *, args, syntax, expander, **kw): Example:: - @dletseq[x << 1, - x << x + 1, - x << x + 2] + @dletseq[x := 1, + x := x + 1, + x := x + 2] def g(a): return a + x assert g(10) == 14 @@ -259,8 +264,8 @@ def dletrec(tree, *, args, syntax, expander, **kw): Example:: - @dletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), - oddp << (lambda x: (x != 0) and evenp(x - 1))] + @dletrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), + oddp := (lambda x: (x != 0) and evenp(x - 1))] def f(x): return evenp(x) assert f(42) is True @@ -280,7 +285,7 @@ def blet(tree, *, args, syntax, expander, **kw): Example:: - @blet[x << 21] + @blet[x := 21] def result(): return 2 * x assert result == 42 @@ -297,9 +302,9 @@ def bletseq(tree, *, args, syntax, expander, **kw): Example:: - @bletseq[x << 1, - x << x + 1, - x << x + 2] + @bletseq[x := 1, + x := x + 1, + x := x + 2] def result(): return x assert result == 4 @@ -316,8 +321,8 @@ def bletrec(tree, *, args, syntax, expander, **kw): Example:: - @bletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), - oddp << (lambda x: (x != 0) and evenp(x - 1))] + @bletrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), + oddp := (lambda x: (x != 0) and evenp(x - 1))] def result(): return evenp(42) assert result is True @@ -414,11 +419,12 @@ def _letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): """Common transformations for let-like operations. Namely:: + x := val --> e.set('x', val) x << val --> e.set('x', val) x --> e.x (when x appears in load context) # ... -> lambda e: ... (applied if dowrap=True) - lhsnames: names to recognize on the LHS of x << val as belonging to this env + lhsnames: names to recognize on the LHS of x := val as belonging to this env rhsnames: names to recognize anywhere in load context as belonging to this env These are separate mainly for ``do[]``, so that we can have new bindings @@ -433,7 +439,7 @@ def _letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): return tree def _transform_envassignment(tree, lhsnames, envset): - """x << val --> e.set('x', val) (for names bound in this environment)""" + """`x := val` or `x << val` --> `e.set('x', val)` (for names bound in this environment)""" # names_in_scope: according to Python's standard binding rules, see scopeanalyzer.py. # Variables defined in let envs are thus not listed in `names_in_scope`. def transform(tree, names_in_scope): @@ -446,7 +452,7 @@ def transform(tree, names_in_scope): return scoped_transform(tree, callback=transform) def _transform_name(tree, rhsnames, envname): - """x --> e.x (in load context; for names bound in this environment)""" + """`x` --> `e.x` (in load context; for names bound in this environment)""" # names_in_scope: according to Python's standard binding rules, see scopeanalyzer.py. # Variables defined in let envs are thus not listed in `names_in_scope`. def transform(tree, names_in_scope): @@ -468,7 +474,7 @@ def transform(tree, names_in_scope): # leave it alone. if type(tree) is Name and tree.id in rhsnames and tree.id not in names_in_scope: hasctx = hasattr(tree, "ctx") # macro-created nodes might not have a ctx. - if hasctx and type(tree.ctx) is not Load: # let variables are rebound using `<<`, not `=`. + if hasctx and type(tree.ctx) is not Load: # let variables are rebound using <<`, not `=`. # TODO: doesn't work for `:=`, which *is* an assignment. Fix this; needs some changes to `scoped_transform`. return tree attr_node = q[n[f"{envname}.{tree.id}"]] if hasctx: @@ -551,20 +557,20 @@ def _let_decorator_impl(bindings, body, mode, kind): def _dletseq_impl(bindings, body, kind): # What we want: # - # @dletseq[x << 1, - # x << x + 1, - # x << x + 2] + # @dletseq[x := 1, + # x := x + 1, + # x := x + 2] # def g(*args, **kwargs): # return x # assert g() == 4 # # --> # - # @dlet[x << 1] + # @dlet[x := 1] # def g(*args, **kwargs, e1): # original args from tree go to the outermost def - # @dlet[x << x + 1] # on RHS, important for e1.x to be in scope + # @dlet[x := x + 1] # on RHS, important for e1.x to be in scope # def g2(*, e2): - # @dlet[x << x + 2] + # @dlet[x := x + 2] # def g3(*, e3): # expansion proceeds from inside out # return e3.x # original args travel here by the closure property # return g3() @@ -625,7 +631,7 @@ def local(tree, *, syntax, **kw): Usage:: - local[name << value] + local[name := value] Only meaningful in a ``do[...]``, ``do0[...]``, or an implicit ``do`` (extra bracket syntax). @@ -637,7 +643,7 @@ def local(tree, *, syntax, **kw): on the RHS. This means that if you want, you can declare a local ``x`` that takes its - initial value from a nonlocal ``x``, by ``local[x << x]``. Here the ``x`` + initial value from a nonlocal ``x``, by ``local[x := x]``. Here the ``x`` on the RHS is the nonlocal one (since the declaration has not yet taken effect), and the ``x`` on the LHS is the name given to the new local variable that only exists inside the ``do``. Any references to ``x`` in any further @@ -680,14 +686,14 @@ def do(tree, *, syntax, expander, **kw): Example:: - do[local[x << 42], + do[local[x := 42], print(x), - x << 23, + x := 23, x] This is sugar on top of ``unpythonic.seq.do``, but with some extra features. - - To declare and initialize a local name, use ``local[name << value]``. + - To declare and initialize a local name, use ``local[name := value]``. The operator ``local`` is syntax, not really a function, and it only exists inside a ``do``. There is also an operator ``delete`` @@ -702,7 +708,7 @@ def do(tree, *, syntax, expander, **kw): - Names declared within the same ``do`` must be unique. Re-declaring the same name is an expansion-time error. - - To assign to an already declared local name, use ``name << value``. + - To assign to an already declared local name, use ``name := value``. **local name declarations** @@ -711,7 +717,7 @@ def do(tree, *, syntax, expander, **kw): result = [] let((lst, []))[do[result.append(lst), # the let "lst" - local[lst << lst + [1]], # LHS: do "lst", RHS: let "lst" + local[lst := lst + [1]], # LHS: do "lst", RHS: let "lst" result.append(lst)]] # the do "lst" assert result == [[], [1]] @@ -753,14 +759,14 @@ def do(tree, *, syntax, expander, **kw): uses, the ambiguity does not arise. The transformation inserts not only the word ``do``, but also the outermost brackets. For example:: - let[x << 1, - y << 2][[ + let[x := 1, + y := 2][[ [x, y]]] transforms to:: - let[x << 1, - y << 2][do[[ # "do[" is inserted between the two opening brackets + let[x := 1, + y := 2][do[[ # "do[" is inserted between the two opening brackets [x, y]]]] # and its closing "]" is inserted here which already gets rid of the ambiguity. @@ -770,24 +776,24 @@ def do(tree, *, syntax, expander, **kw): Macros are expanded in an inside-out order, so a nested ``let`` shadows names, if the same names appear in the ``do``:: - do[local[x << 17], - let[x << 23][ + do[local[x := 17], + let[x := 23][ print(x)], # 23, the "x" of the "let" print(x)] # 17, the "x" of the "do" The reason we require local names to be declared is to allow write access to lexically outer environments from inside a ``do``:: - let[x << 17][ - do[x << 23, # no "local[...]"; update the "x" of the "let" - local[y << 42], # "y" is local to the "do" + let[x := 17][ + do[x := 23, # no "local[...]"; update the "x" of the "let" + local[y := 42], # "y" is local to the "do" print(x, y)]] With the extra bracket syntax, the latter example can be written as:: - let[x << 17][[ - x << 23, - local[y << 42], + let[x := 17][[ + x := 23, + local[y := 42], print(x, y)]] It's subtly different in that the first version has the do-items in a tuple, @@ -833,11 +839,11 @@ def transform(self, tree): expr = islocaldef(tree) if expr: if not isenvassign(expr): - raise SyntaxError("local[...] takes exactly one expression of the form 'name << value'") # pragma: no cover + raise SyntaxError("local[...] takes exactly one expression of the form 'name := value' or 'name << value'") # pragma: no cover view = UnexpandedEnvAssignView(expr) self.collect(view.name) - view.value = self.visit(view.value) # nested local[] (e.g. from `do0[local[y << 5],]`) - return expr # `local[x << 21]` --> `x << 21`; compiling *that* makes the env-assignment occur. + view.value = self.visit(view.value) # nested local[] (e.g. from `do0[local[y := 5],]`) + return expr # `local[x := 21]` --> `x := 21`; compiling *that* makes the env-assignment occur. return tree # don't recurse! c = LocaldefCollector() tree = c.visit(tree) @@ -918,7 +924,7 @@ def _do0(tree): raise SyntaxError("do0 body: expected a sequence of comma-separated expressions") # pragma: no cover elts = tree.elts # Use `local[]` and `do[]` as hygienically captured macros. - newelts = [q[a[_our_local][_do0_result << a[elts[0]]]], # noqa: F821, local[] defines it inside the do[]. + newelts = [q[a[_our_local][_do0_result := a[elts[0]]]], # noqa: F821, local[] defines it inside the do[]. *elts[1:], q[_do0_result]] # noqa: F821 return q[a[_our_do][t[newelts]]] # do0[] is also just a do[] diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index ee1f6207..0b7e20be 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -13,7 +13,7 @@ from mcpyrate import unparse from mcpyrate.core import Done -from .astcompat import getconstant, Str +from .astcompat import getconstant, Str, NamedExpr from .nameutil import isx, getname letf_name = "letter" # must match what ``unpythonic.syntax.letdo._let_expr_impl`` uses in its output. @@ -68,13 +68,15 @@ def canonize_bindings(elts, letsyntax_mode=False): # public as of v0.14.3+ [Tuple(elts=[k0, v0]), ...] elts: `list` of bindings, one of:: + [k0 := v0, ...] # v0.15.3+: new env-assignment syntax, preferred + [k := v] # v0.15.3+ + [k0 << v0, ...] # v0.15.0+: previous env-assignment syntax + [k << v] # v0.15.0+ + [[k0, v0], ...] # v0.15.0+: accept also brackets (for consistency) + [[k, v]] # v0.15.0+ [(k0, v0), ...] # multiple bindings contained in a tuple [(k, v),] # single binding contained in a tuple also ok [k, v] # special single binding format, missing tuple container - [[k0, v0], ...] # v0.15.0+: accept also brackets (for consistency) - [[k, v]] # v0.15.0+ - [k0 << v0, ...] # v0.15.0+: accept also env-assignment syntax - [k << v] # v0.15.0+ where the ks and vs are AST nodes. @@ -85,31 +87,52 @@ def canonize_bindings(elts, letsyntax_mode=False): # public as of v0.14.3+ def iskvpairbinding(lst): return len(lst) == 2 and _isbindingtarget(lst[0], letsyntax_mode) - if len(elts) == 1 and isenvassign(elts[0], letsyntax_mode): # [k << v] - return [Tuple(elts=[elts[0].left, elts[0].right])] + if len(elts) == 1: + if isenvassign(elts[0], letsyntax_mode) is LShift: # [k << v] + return [Tuple(elts=[elts[0].left, elts[0].right])] + if isenvassign(elts[0], letsyntax_mode) is NamedExpr: # [k := v] + return [Tuple(elts=[elts[0].target, elts[0].value])] if len(elts) == 2 and iskvpairbinding(elts): # [k, v] return [Tuple(elts=elts)] # TODO: `mcpyrate`: just `q[t[elts]]`? if all((type(b) is Tuple and iskvpairbinding(b.elts)) for b in elts): # [(k0, v0), ...] return elts if all((type(b) is List and iskvpairbinding(b.elts)) for b in elts): # [[k0, v0], ...] return [Tuple(elts=b.elts) for b in elts] - if all(isenvassign(b, letsyntax_mode) for b in elts): # [k0 << v0, ...] - return [Tuple(elts=[b.left, b.right]) for b in elts] - raise SyntaxError("expected bindings to be `(k0, v0), ...`, `[k0, v0], ...`, or `k0 << v0, ...`, or a single `k, v`, or `k << v`") # pragma: no cover + if all(isenvassign(b, letsyntax_mode) for b in elts): # [k0 << v0, ...] or [k0 := v0, ...] + out = [] + for b in elts: + if isenvassign(b, letsyntax_mode) is LShift: + out.append(Tuple(elts=[b.left, b.right])) + else: # NamedExpr + out.append(Tuple(elts=[b.target, b.value])) + return out + raise SyntaxError("expected bindings to be `k0 := v0, ...`, `k0 << v0, ...`, `[k0, v0], ...`, or `(k0, v0), ...`, or a single `k := v`, `k << v`, or `k, v`") # pragma: no cover def isenvassign(tree, letsyntax_mode=False): - """Detect whether tree is an unpythonic ``env`` assignment, ``name << value``. + """Detect whether tree is an unpythonic ``env`` assignment. + + Starting at v0.15.3: new env-assignment syntax ``name := value`` is recommended. + + From v0.15.0 to v0.15.2, env-assignment used the syntax ``name << value``. + This is still available for backward compatibility. - The only way this differs from a general left-shift is that the LHS must be - an ``ast.Name``. + Return value is one of the constants: + `NamedExpr`: `tree` is an env-assignment, with modern syntax. + `LShift`: `tree` is an env-assignment, with classic syntax, + `False`: `tree` is not an env-assignment, + + The only way this differs from a left-shift or the usual kind of walrus assignment + is that the LHS must be an ``ast.Name``. letsyntax_mode: used by let_syntax to allow template definitions. This allows, beside a bare name `k`, the formats `k(a0, ...)` and `k[a0, ...]` to appear in the variable-name position. """ - if not (type(tree) is BinOp and type(tree.op) is LShift): - return False - return _isbindingtarget(tree.left, letsyntax_mode) + if type(tree) is BinOp and type(tree.op) is LShift and _isbindingtarget(tree.left, letsyntax_mode): + return LShift + if type(tree) is NamedExpr and _isbindingtarget(tree.target, letsyntax_mode): # added in 0.15.3 + return NamedExpr + return False # TODO: This would benefit from macro destructuring in the expander. # TODO: See https://github.com/Technologicat/mcpyrate/issues/3 @@ -167,7 +190,7 @@ def islet(tree, expanded=True): return (f"{kind}_decorator", mode) # this call was generated by _let_decorator_impl else: return (f"{kind}_expr", mode) # this call was generated by _let_expr_impl - # dlet[k0 << v0, ...] (usually in a decorator list) + # dlet[k0 := v0, ...] (usually in a decorator list) deconames = ("dlet", "dletseq", "dletrec", "blet", "bletseq", "bletrec") if type(tree) is Subscript and type(tree.value) is Name: # could be a Subscript decorator (Python 3.9+) @@ -182,8 +205,8 @@ def islet(tree, expanded=True): if not type(tree) is Subscript: return False # Note we don't care about the bindings format here. - # let[k0 << v0, ...][body] - # let(k0 << v0, ...)[body] + # let[k0 := v0, ...][body] + # let(k0 := v0, ...)[body] # ^^^^^^^^^^^^^^^^^^ macro = tree.value exprnames = ("let", "letseq", "letrec", "let_syntax", "abbrev") @@ -199,8 +222,8 @@ def islet(tree, expanded=True): elif type(macro) is Name: s = macro.id if any(s == x for x in exprnames): - # let[k0 << v0, ...][body] - # let(k0 << v0, ...)[body] + # let[k0 := v0, ...][body] + # let(k0 := v0, ...)[body] # ^^^^ expr = _get_subscript_slice(tree) h = _ishaskellylet(expr) @@ -215,19 +238,19 @@ def _ishaskellylet(tree): In other words, detect the part inside the brackets in:: - let[[k0 << v0, ...] in body] - let[body, where[k0 << v0, ...]] + let[[k0 := v0, ...] in body] + let[body, where[k0 := v0, ...]] To detect the full expression including the ``let[]``, use ``islet`` instead. """ - # let[[k0 << v0, ...] in body] - # let[(k0 << v0, ...) in body] + # let[[k0 := v0, ...] in body] + # let[(k0 := v0, ...) in body] def maybeiscontentofletin(tree): return (type(tree) is Compare and len(tree.ops) == 1 and type(tree.ops[0]) is In and type(tree.left) in (List, Tuple)) - # let[body, where[k0 << v0, ...]] - # let[body, where(k0 << v0, ...)] + # let[body, where[k0 := v0, ...]] + # let[body, where(k0 := v0, ...)] def maybeiscontentofletwhere(tree): return type(tree) is Tuple and len(tree.elts) == 2 and type(tree.elts[1]) in (Call, Subscript) @@ -294,10 +317,10 @@ def isdo(tree, expanded=True): # ----------------------------------------------------------------------------- class UnexpandedEnvAssignView: - """Destructure an env-assignment, writably. + """Destructure an unexpanded env-assignment, writably. If ``tree`` cannot be interpreted as an unpythonic ``env`` assignment - of the form ``name << value``, then ``TypeError`` is raised. + of the form ``name := value`` or ``name << value``, then ``TypeError`` is raised. For easy in-place modification of both ``name`` and ``value``. Use before the env-assignment is expanded away (so, before the ``let[]`` or ``do[]`` @@ -317,7 +340,7 @@ class UnexpandedEnvAssignView: ``value``: the thing being assigned, as an AST. - Writing to either attribute updates the original. + Writing to either attribute updates the original, preserving the syntax (`:=` or `<<`). """ def __init__(self, tree): if not isenvassign(tree): @@ -325,21 +348,34 @@ def __init__(self, tree): self._tree = tree def _getname(self): - return getname(self._tree.left, accept_attr=False) + if isenvassign(self._tree) is LShift: + return getname(self._tree.left, accept_attr=False) + else: # NamedExpr + return getname(self._tree.target, accept_attr=False) def _setname(self, newname): if not isinstance(newname, str): raise TypeError(f"expected str for new name, got {type(newname)} with value {repr(newname)}") + if isenvassign(self._tree) is LShift: + targetnode = self._tree.left + else: # NamedExpr + targetnode = self._tree.target # The `Done` may be produced by expanded `@namemacro`s. - if isinstance(self._tree.left, Done): - self._tree.left.body.id = newname + if isinstance(targetnode, Done): + targetnode.body.id = newname else: - self._tree.left.id = newname + targetnode.id = newname name = property(fget=_getname, fset=_setname, doc="The name of the assigned var, as an str. Writable.") def _getvalue(self): - return self._tree.right + if isenvassign(self._tree) is LShift: + return self._tree.right + else: # NamedExpr + return self._tree.value def _setvalue(self, newvalue): - self._tree.right = newvalue + if isenvassign(self._tree) is LShift: + self._tree.right = newvalue + else: # NamedExpr + self._tree.value = newvalue value = property(fget=_getvalue, fset=_setvalue, doc="The value of the assigned var, as an AST. Writable.") class UnexpandedLetView: @@ -353,30 +389,32 @@ class UnexpandedLetView: **Supported formats**:: - dlet[k0 << v0, ...] # decorator - let[k0 << v0, ...][body] # lispy expression - let[[k0 << v0, ...] in body] # haskelly expression - let[body, where[k0 << v0, ...]] # haskelly expression, inverted + dlet[k0 := v0, ...] # decorator + let[k0 := v0, ...][body] # lispy expression + let[[k0 := v0, ...] in body] # haskelly expression + let[body, where[k0 := v0, ...]] # haskelly expression, inverted In addition, we also support *just the bracketed part* of the haskelly formats. This is to make it easier for the macro interface to destructure these forms (for sending into the ``let`` syntax transformer). So these forms are supported, too:: - [k0 << v0, ...] in body - (body, where[k0 << v0, ...]) + [k0 := v0, ...] in body + (body, where[k0 := v0, ...]) Finally, in any of these, the bindings subform can actually be in any of the formats: - [k0 << v0, ...] # preferred, v0.15.0+ + [k0 := v0, ...] # preferred, v0.15.3+ + [k0 << v0, ...] # preferred, v0.15.0 to v0.15.2 (k0 << v0, ...) [[k0, v0], ...] [(k0, v0), ...] ([k0, v0], ...) ((k0, v0), ...) k, v - k << v # preferred for a single binding, v0.15.0+ + k := v # preferred for a single binding, v0.15.3+ + k << v # preferred for a single binding, v0.15.0 to v0.15.2 This is a data abstraction that hides the detailed structure of the AST, since there are many alternate syntaxes that can be used for a ``let`` diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 08f81d6b..423dc8eb 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -17,22 +17,47 @@ x = "the global x" # for lexical scoping tests def runtests(): - with testset("do (imperative code in an expression)"): + with testset("do (imperative code in an expression) (new env-assignment syntax 0.15.3+)"): # Macro wrapper for unpythonic.seq.do (imperative code in expression position) - # - Declare and initialize a local variable with ``local[var << value]``. + # - Declare and initialize a local variable with ``local[var := value]``. # Is in scope from the next expression onward, for the (lexical) remainder # of the do. - # - Assignment is ``var << value``. Valid from any level inside the ``do`` + # - Assignment is ``var := value``. Valid from any level inside the ``do`` # (including nested ``let`` constructs and similar). # - No need for ``lambda e: ...`` wrappers. Inserted automatically, # so the lines are only evaluated as the underlying seq.do() runs. + d1 = do[local[x := 17], + print(x), + x := 23, + x] + test[d1 == 23] + + # Since we repurposed an existing assignment operator, let's check we didn't accidentally assign to the function scope. + test_raises[NameError, x, "only the `do[]` should have an `x` here"] + + # v0.14.0: do[] now supports deleting previously defined local names with delete[] + a = 5 + d = do[local[a := 17], # noqa: F841, yes, d is unused. + test[a == 17], + delete[a], + test[a == 5], # lexical scoping + True] + + test_raises[KeyError, do[delete[a], ], "should have complained about deleting nonexistent local 'a'"] + + # do0[]: like do[], but return the value of the **first** expression + d2 = do0[local[y := 5], # noqa: F821, `local` defines the name on the LHS of the `<<`. + print("hi there, y =", y), # noqa: F821 + 42] # evaluated but not used + test[d2 == 5] + + with testset("do (imperative code in an expression) (previous modern env-assignment syntax)"): d1 = do[local[x << 17], print(x), x << 23, - x] # do[] returns the value of the last expression + x] # do[] returns the value of the last expression # noqa: F823, it's the `x` from `do[]`, not from the enclosing scope. test[d1 == 23] - # v0.14.0: do[] now supports deleting previously defined local names with delete[] a = 5 d = do[local[a << 17], # noqa: F841, yes, d is unused. test[a == 17], @@ -42,14 +67,41 @@ def runtests(): test_raises[KeyError, do[delete[a], ], "should have complained about deleting nonexistent local 'a'"] - # do0[]: like do[], but return the value of the **first** expression d2 = do0[local[y << 5], # noqa: F821, `local` defines the name on the LHS of the `<<`. print("hi there, y =", y), # noqa: F821 42] # evaluated but not used test[d2 == 5] # Let macros. Lexical scoping supported. - with testset("let, letseq, letrec basic usage"): + with testset("let, letseq, letrec basic usage (new env-assignment syntax 0.15.3+)"): + # parallel binding, i.e. bindings don't see each other + test[let[x := 17, + y := 23][ # noqa: F821, `let` defines `y` here. + (x, y)] == (17, 23)] # noqa: F821 + + # sequential binding, i.e. Scheme/Racket let* + test[letseq[x := 1, + y := x + 1][ # noqa: F821 + (x, y)] == (1, 2)] # noqa: F821 + + test[letseq[x := 1, + x := x + 1][ # in a letseq, rebinding the same name is ok + x] == 2] + + # letrec sugars unpythonic.lispylet.letrec, removing the need for quotes on LHS + # and "lambda e: ..." wrappers on RHS (these are inserted by the macro): + test[letrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `letrec` defines `evenp` here. + oddp := (lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 + evenp(42)] is True] # noqa: F821 + + # nested letrecs work, too - each environment is internally named by a gensym + # so that outer ones "show through": + test[letrec[z := 9000][ # noqa: F821 + letrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 + oddp := (lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 + (evenp(42), z)]] == (True, 9000)] # noqa: F821 + + with testset("let, letseq, letrec basic usage (previous modern env-assignment syntax)"): # parallel binding, i.e. bindings don't see each other test[let[x << 17, y << 23][ # noqa: F821, `let` defines `y` here. @@ -98,6 +150,32 @@ def runtests(): "should not be able to rebind the same name in the same let"] # implicit do: an extra set of brackets denotes a multi-expr body + with testset("implicit do (extra bracket syntax for multi-expr let body) (new env-assignment syntax v0.15.3+)"): + a = let[x := 1, + y := 2][[ # noqa: F821 + y := 1337, # noqa: F821 + (x, y)]] # noqa: F821 + test[a == (1, 1337)] + + # only the outermost extra brackets denote a multi-expr body + a = let[(x, 1), + (y, 2)][[ # noqa: F821 + [1, 2]]] + test[a == [1, 2]] + + # implicit do works also in letseq, letrec + a = letseq[x := 1, + y := x + 1][[ # noqa: F821 + x := 1337, + (x, y)]] # noqa: F821 + test[a == (1337, 2)] + + a = letrec[x := 1, + y := x + 1][[ # noqa: F821 + x := 1337, + (x, y)]] # noqa: F821 + test[a == (1337, 2)] + with testset("implicit do (extra bracket syntax for multi-expr let body)"): a = let[x << 1, y << 2][[ # noqa: F821 diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index 45725233..a5e46bb0 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -41,7 +41,9 @@ def validate(lst): test[validate(the[canonize_bindings(q[k0, v0].elts)])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings(q[((k0, v0),)].elts)])] # noqa: F821 test[validate(the[canonize_bindings(q[(k0, v0), (k1, v1)].elts)])] # noqa: F821 + test[validate(the[canonize_bindings([q[k0 := v0]])])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings([q[k0 << v0]])])] # noqa: F821, it's quoted. + test[validate(the[canonize_bindings(q[k0 := v0, k1 := v1].elts)])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings(q[k0 << v0, k1 << v1].elts)])] # noqa: F821, it's quoted. # -------------------------------------------------------------------------------- @@ -51,13 +53,19 @@ def validate(lst): # need this utility, so we must test it first. with testset("isenvassign"): test[not isenvassign(q[x])] # noqa: F821 + test[isenvassign(q[x := 42])] # noqa: F821 test[isenvassign(q[x << 42])] # noqa: F821 with testset("islet"): test[not islet(q[x])] # noqa: F821 test[not islet(q[f()])] # noqa: F821 - # modern notation for bindings + # unpythonic 0.15.3+, Python 3.8+ + test[islet(the[expandrq[let[x := 21][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` + test[islet(the[expandrq[let[[x := 21] in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 + test[islet(the[expandrq[let[2 * x, where[x := 21]]]]) == ("expanded_expr", "let")] # noqa: F821 + + # unpythonic 0.15.0 to 0.15.2, previous modern notation for bindings test[islet(the[expandrq[let[x << 21][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` test[islet(the[expandrq[let[[x << 21] in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 test[islet(the[expandrq[let[2 * x, where[x << 21]]]]) == ("expanded_expr", "let")] # noqa: F821 @@ -67,18 +75,30 @@ def validate(lst): test[islet(the[expandrq[let[(x, 21) in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 test[islet(the[expandrq[let[2 * x, where(x, 21)]]]) == ("expanded_expr", "let")] # noqa: F821 + # unpythonic 0.15.3+, Python 3.8+ + with expandrq as testdata: + @dlet(x := 21) # noqa: F821 + def f0(): + return 2 * x # noqa: F821 + test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] + + # unpythonic 0.15.0 to 0.15.2, previous modern notation for bindings with expandrq as testdata: @dlet(x << 21) # noqa: F821 def f1(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] + # classic notation for bindings with expandrq as testdata: @dlet((x, 21)) # noqa: F821 def f2(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] + testdata = q[let[x := 21][2 * x]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("lispy_expr", "let")] + testdata = q[let[x << 21][2 * x]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("lispy_expr", "let")] @@ -95,6 +115,8 @@ def f2(): testdata = q[let[2 * x, where(x, 21)]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("where_expr", "let")] + testdata = q[let[[x := 21, y := 2] in y * x]] # noqa: F821 + test[islet(the[testdata], expanded=False) == ("in_expr", "let")] testdata = q[let[[x << 21, y << 2] in y * x]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("in_expr", "let")] testdata = q[let[((x, 21), (y, 2)) in y * x]] # noqa: F821 @@ -120,6 +142,12 @@ def f4(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]], expanded=False) == ("decorator", "dlet")] + with q as testdata: + @dlet(x := 21) # noqa: F821 + def f5(): + return 2 * x # noqa: F821 + test[islet(the[testdata[0].decorator_list[0]], expanded=False) == ("decorator", "dlet")] + with testset("islet integration with autocurry"): # NOTE: We have to be careful with how we set up the test data here. # @@ -167,6 +195,10 @@ def f4(): test[not isdo(q[x])] # noqa: F821 test[not isdo(q[f()])] # noqa: F821 + # unpythonic 0.15.3+, Python 3.8+ + test[isdo(the[expandrq[do[x := 21, # noqa: F821 + 2 * x]]]) == "expanded"] # noqa: F821 + test[isdo(the[expandrq[do[x << 21, # noqa: F821 2 * x]]]) == "expanded"] # noqa: F821 @@ -177,6 +209,21 @@ def f4(): thedo = testdata[0].value test[isdo(the[thedo]) == "curried"] + # unpythonic 0.15.3+, Python 3.8+ + testdata = q[do[x := 21, # noqa: F821 + 2 * x]] # noqa: F821 + test[isdo(the[testdata], expanded=False) == "do"] + + testdata = q[do0[23, # noqa: F821 + x := 21, # noqa: F821 + 2 * x]] # noqa: F821 + test[isdo(the[testdata], expanded=False) == "do0"] + + testdata = q[someothermacro[x := 21, # noqa: F821 + 2 * x]] # noqa: F821 + test[not isdo(the[testdata], expanded=False)] + + # previous modern notation testdata = q[do[x << 21, # noqa: F821 2 * x]] # noqa: F821 test[isdo(the[testdata], expanded=False) == "do"] @@ -193,6 +240,30 @@ def f4(): # -------------------------------------------------------------------------------- # Destructuring - envassign + with testset("envassign destructuring (new env-assign syntax v0.15.3+)"): + testdata = q[x := 42] # noqa: F821 + view = UnexpandedEnvAssignView(testdata) + + # read + test[view.name == "x"] + test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 42] # Python 3.8: ast.Constant + + # write + view.name = "y" + view.value = q[23] + test[view.name == "y"] + test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 23] # Python 3.8: ast.Constant + + # it's a live view + test[unparse(testdata) == "(y := 23)"] # syntax type `:=` vs. `<<` is preserved + + # error cases + test_raises[TypeError, + UnexpandedEnvAssignView(q[x]), # noqa: F821 + "not an env assignment"] + with test_raises[TypeError, "name must be str"]: + view.name = 1234 + with testset("envassign destructuring"): testdata = q[x << 42] # noqa: F821 view = UnexpandedEnvAssignView(testdata) @@ -208,7 +279,7 @@ def f4(): test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 23] # Python 3.8: ast.Constant # it's a live view - test[unparse(testdata) == "(y << 23)"] + test[unparse(testdata) == "(y << 23)"] # syntax type `:=` vs. `<<` is preserved # error cases test_raises[TypeError, @@ -245,6 +316,8 @@ def testletdestructuring(testdata): test[unparse(view.body) == "(z * t)"] # lispy expr + testdata = q[let[x := 21, y := 2][y * x]] # noqa: F821 + testletdestructuring(testdata) testdata = q[let[x << 21, y << 2][y * x]] # noqa: F821 testletdestructuring(testdata) testdata = q[let[[x, 21], [y, 2]][y * x]] # noqa: F821 @@ -253,6 +326,8 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # haskelly let-in + testdata = q[let[[x := 21, y := 2] in y * x]] # noqa: F821 + testletdestructuring(testdata) testdata = q[let[[x << 21, y << 2] in y * x]] # noqa: F821 testletdestructuring(testdata) testdata = q[let[(x << 21, y << 2) in y * x]] # noqa: F821 @@ -267,6 +342,8 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # haskelly let-where + testdata = q[let[y * x, where[x := 21, y := 2]]] # noqa: F821 + testletdestructuring(testdata) testdata = q[let[y * x, where[x << 21, y << 2]]] # noqa: F821 testletdestructuring(testdata) testdata = q[let[y * x, where(x << 21, y << 2)]] # noqa: F821 @@ -281,6 +358,8 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # disembodied haskelly let-in (just the content, no macro invocation) + testdata = q[[x := 21, y := 2] in y * x] # noqa: F821 + testletdestructuring(testdata) testdata = q[[x << 21, y << 2] in y * x] # noqa: F821 testletdestructuring(testdata) testdata = q[(x << 21, y << 2) in y * x] # noqa: F821 @@ -295,6 +374,8 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # disembodied haskelly let-where (just the content, no macro invocation) + testdata = q[y * x, where[x := 21, y := 2]] # noqa: F821 + testletdestructuring(testdata) testdata = q[y * x, where[x << 21, y << 2]] # noqa: F821 testletdestructuring(testdata) testdata = q[y * x, where(x << 21, y << 2)] # noqa: F821 @@ -311,7 +392,7 @@ def testletdestructuring(testdata): # decorator with q as testdata: @dlet((x, 21), (y, 2)) # noqa: F821 - def f5(): + def f6(): return 2 * x # noqa: F821 # read @@ -392,7 +473,7 @@ def testexpandedletdestructuring(testdata): # decorator with expandrq as testdata: @dlet((x, 21), (y, 2)) # noqa: F821 - def f6(): + def f7(): return 2 * x # noqa: F821 view = ExpandedLetView(testdata[0].decorator_list[0]) test_raises[TypeError, @@ -488,7 +569,7 @@ def testbindings(*expected): # decorator, letrec with expandrq as testdata: @dletrec((x, 21), (y, 2)) # noqa: F821 - def f7(): + def f8(): return 2 * x # noqa: F821 view = ExpandedLetView(testdata[0].decorator_list[0]) test_raises[TypeError, @@ -517,6 +598,45 @@ def f7(): # -------------------------------------------------------------------------------- # Destructuring - unexpanded do + with testset("do destructuring (unexpanded) (new env-assign syntax v0.15.3+)"): + testdata = q[do[local[x := 21], # noqa: F821 + 2 * x]] # noqa: F821 + view = UnexpandedDoView(testdata) + # read + thebody = view.body + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + thing = thebody[0].slice + else: + thing = thebody[0].slice.value + test[isenvassign(the[thing])] + # write + # This mutates the original, but we have to assign `view.body` to trigger the setter. + thebody[0] = q[local[x := 9001]] # noqa: F821 + view.body = thebody + + # implicit do, a.k.a. extra bracket syntax + testdata = q[let[[local[x := 21], # noqa: F821 + 2 * x]]] # noqa: F821 + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + theimplicitdo = testdata.slice + else: + theimplicitdo = testdata.slice.value + view = UnexpandedDoView(theimplicitdo) + # read + thebody = view.body + if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. + thing = thebody[0].slice + else: + thing = thebody[0].slice.value + test[isenvassign(the[thing])] + # write + thebody[0] = q[local[x := 9001]] # noqa: F821 + view.body = thebody + + test_raises[TypeError, + UnexpandedDoView(q[x]), # noqa: F821 + "not a do form"] + with testset("do destructuring (unexpanded)"): testdata = q[do[local[x << 21], # noqa: F821 2 * x]] # noqa: F821 From a09e59727e096c9841b15d5ba56722269e1d321d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:21:40 +0300 Subject: [PATCH 702/832] improve docstring --- unpythonic/syntax/letdo.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index e81ba13a..6e293b98 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -424,10 +424,10 @@ def _letlike_transform(tree, envname, lhsnames, rhsnames, setter, dowrap=True): x --> e.x (when x appears in load context) # ... -> lambda e: ... (applied if dowrap=True) - lhsnames: names to recognize on the LHS of x := val as belonging to this env + lhsnames: names to recognize on the LHS of env-assignment (`x := val` or `x << val`) as belonging to this env rhsnames: names to recognize anywhere in load context as belonging to this env - These are separate mainly for ``do[]``, so that we can have new bindings + The LHS/RHS names are separate mainly for ``do[]``, so that we can have new bindings take effect only in following exprs. setter: function, (k, v) --> v, side effect to set e.k to v From 106a406abfcf4945e71edfd46f58a0761b69382d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:21:59 +0300 Subject: [PATCH 703/832] add comment --- unpythonic/syntax/letdo.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 6e293b98..42c1f3a6 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -458,6 +458,8 @@ def _transform_name(tree, rhsnames, envname): def transform(tree, names_in_scope): # This transformation is deceptively simple, hence requires some comment: # + # - The goal is to transform read accesses to let variables, `x` --> `e.x`. + # # - Attributes (and Subscripts) work, because we are called again for # the `value` part of the `Attribute` (or `Subscript`) node, which # then gets transformed if it's a `Name` matching our rules. From 7ed50bd21aed5bf2970ff34e0ad243e46b400f2f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:22:19 +0300 Subject: [PATCH 704/832] Fixed. Actually this was already doing what it should. --- unpythonic/syntax/letdo.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 42c1f3a6..e2cb6240 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -475,8 +475,8 @@ def transform(tree, names_in_scope): # in those parts of code where it is used, so an outer let will # leave it alone. if type(tree) is Name and tree.id in rhsnames and tree.id not in names_in_scope: - hasctx = hasattr(tree, "ctx") # macro-created nodes might not have a ctx. - if hasctx and type(tree.ctx) is not Load: # let variables are rebound using <<`, not `=`. # TODO: doesn't work for `:=`, which *is* an assignment. Fix this; needs some changes to `scoped_transform`. + hasctx = hasattr(tree, "ctx") # Macro-created nodes might not have a ctx. + if hasctx and type(tree.ctx) is not Load: # Ignore assignments and deletes. return tree attr_node = q[n[f"{envname}.{tree.id}"]] if hasctx: From de34fe8af43e7f58871679055a5480455c41e547 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:22:34 +0300 Subject: [PATCH 705/832] add some clarifying tests --- unpythonic/syntax/tests/test_letdo.py | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 423dc8eb..9415a604 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -485,6 +485,33 @@ def test14(): test14() x = "the nonlocal x" # restore the test environment + # v0.15.3+: walrus syntax + @dlet[x := "the env x"] + def test15(): + def inner(): + (x := "updated env x") # noqa: F841, this writes to the let env since there is no `x` in an intervening scope, according to Python's standard rules. + inner() + return x + test[test15() == "updated env x"] + + @dlet[x := "the env x"] + def test16(): + def inner(): + x = "the inner x" # noqa: F841, unused on purpose, for testing. An assignment *statement* does NOT write to the let env. + inner() + return x + test[test16() == "the env x"] + + @dlet[x := "the env x"] + def test17(): + x = "the local x" # This lexical variable shadows the env x. + def inner(): + # The env x is shadowed. Since we don't say `nonlocal x`, this creates a new lexical variable scoped to `inner`. + (x := "the inner x") # noqa: F841, unused on purpose, for testing. + inner() + return x + test[test17() == "the local x"] + # in do[] (also the implicit do), local[] takes effect from the next item test[let[x << "the let x", y << None][ # noqa: F821 From 1689a0d358edd06aa91879f09bb205b1234b9f98 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:22:47 +0300 Subject: [PATCH 706/832] update docs including changes for 0.15.1, 0.15.2 --- doc/features.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/doc/features.md b/doc/features.md index af688656..681835e6 100644 --- a/doc/features.md +++ b/doc/features.md @@ -120,6 +120,8 @@ The exception are the features marked **[M]**, which are primarily intended as a - [`pack`: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) - [`namelambda`: rename a function](#namelambda-rename-a-function) - [`timer`: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) +- [`format_human_time`: seconds to days, hours, minutes, seconds](#format_human_time-seconds-to-days-hours-minutes-seconds) +- [`ETAEstimator`: estimate the time of completion of a long-running task](#etaestimator-estimate-the-time-of-completion-of-a-long-running-task) - [`getattrrec`, `setattrrec`: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) - [`arities`, `kwargs`, `resolve_bindings`: Function signature inspection utilities](#arities-kwargs-resolve_bindings-function-signature-inspection-utilities) - [`Popper`: a pop-while iterator](#popper-a-pop-while-iterator) @@ -373,6 +375,8 @@ letrec[[evenp << (lambda x: ### `env`: the environment +**Changed in v0.15.2.** *`env` objects are now pickleable.* + The environment used by all the `let` constructs and `assignonce` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. It is somewhat similar to [`types.SimpleNamespace`](https://docs.python.org/3/library/types.html#types.SimpleNamespace), but with many extra features. For details, see `unpythonic.env.env` (and note the unfortunate module name). Our `env` allows things like: @@ -4850,6 +4854,39 @@ with timer(p=True): # if p, auto-print result The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the `with ... as ...` syntax. After the context exits, the *Δt* is available in its `dt` attribute. The timer instance itself stays alive due to Python's scoping rules. +### `format_human_time`: seconds to days, hours, minutes, seconds + +**Added in v0.15.1.** + +Convert a duration from seconds (`float` or `int`) to a human-readable string of days, hours, minutes and seconds. + +```python +assert format_human_time(30) == "30 seconds" +assert format_human_time(90) == "01:30" # mm:ss +assert format_human_time(3690) == "01:01:30" # hh:mm:ss +assert format_human_time(86400 + 3690) == "1 day 01:01:30" +assert format_human_time(2 * 86400 + 3690) == "2 days 01:01:30" +``` + + +### `ETAEstimator`: estimate the time of completion of a long-running task + +**Added in v0.15.1.** + +Simple but useful: + +```python +n = 1000 +est = ETAEstimator(total=n, keep_last=10) +for k in range(n): + print(f"Processing item {k + 1} out of {n}, {est.formatted_eta}") + ... # do something + est.tick() +``` + +The ETA estimate is automatically formatted using `format_human_time` (see above) to maximize readability. + + ### `getattrrec`, `setattrrec`: access underlying data in an onion of wrappers ```python From d4cfe1e83eed3b6c5e421ffae7a50b04b4c19b71 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:23:02 +0300 Subject: [PATCH 707/832] update macro docs: new env-assignment syntax `x := 42` --- doc/macros.md | 262 +++++++++++++++++++++++++++++--------------------- 1 file changed, 152 insertions(+), 110 deletions(-) diff --git a/doc/macros.md b/doc/macros.md index 1b6147c6..cd8493b8 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -100,6 +100,8 @@ Macros that introduce new ways to bind identifiers. ### `let`, `letseq`, `letrec` as macros +**Changed in v0.15.3.** *Added support for the walrus operator `:=` for env-assignment. This is the new preferred syntax to establish let-bindings. All old syntaxes are still supported for backward compatibility.* + **Changed in v0.15.0.** *Added support for env-assignment syntax in the bindings subform. For consistency with other env-assignments, this is now the preferred syntax to establish let-bindings. Additionally, the old lispy syntax now accepts also brackets, for consistency with the use of brackets for macro invocations.* These macros provide properly lexically scoped `let` constructs, no boilerplate: @@ -107,28 +109,32 @@ These macros provide properly lexically scoped `let` constructs, no boilerplate: ```python from unpythonic.syntax import macros, let, letseq, letrec -let[x << 17, # parallel binding, i.e. bindings don't see each other - y << 23][ +let[x := 17, # parallel binding, i.e. bindings don't see each other + y := 23][ print(x, y)] -letseq[x << 1, # sequential binding, i.e. Scheme/Racket let* - y << x + 1][ +letseq[x := 1, # sequential binding, i.e. Scheme/Racket let* + y := x + 1][ print(x, y)] -letrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding, sequentially evaluated - oddp << (lambda x: (x != 0) and evenp(x - 1))][ +letrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding, sequentially evaluated + oddp := (lambda x: (x != 0) and evenp(x - 1))][ print(evenp(42))] ``` Even with just one binding, the syntax remains the same: ```python -let[x << 21][2 * x] +let[x := 21][2 * x] ``` There must be at least one binding; `let[][...]` is a syntax error, since Python's parser rejects an empty subscript slice. -Bindings are established using the `unpythonic` *env-assignment* syntax, `name << value`. The let-bindings can be rebound in the body with the same env-assignment syntax, e.g. `x << 42`. +Bindings are established using standard assignment expression syntax, `name := value`. The let-bindings can be rebound in the body with the same syntax, e.g. `x := 42`. + +The old `unpythonic` env-assignment syntax, `name << value`, is also supported for backward compatibility. This was the preferred syntax in v0.15.0 to v0.15.2. + +**CAUTION**: All let-bindings must be established in the bindings subform. If you absolutely need to do establish more bindings in the body, see the sequencing construct `do[]` and its syntax `local[x := 42]`. The same syntax for the bindings subform is used by: @@ -143,18 +149,18 @@ The same syntax for the bindings subform is used by: The following Haskell-inspired, perhaps more pythonic alternative syntaxes are also available: ```python -let[[x << 21, - y << 17, - z << 4] in +let[[x := 21, + y := 17, + z := 4] in x + y + z] let[x + y + z, - where[x << 21, - y << 17, - z << 4]] + where[x := 21, + y := 17, + z := 4]] -let[[x << 21] in 2 * x] -let[2 * x, where[x << 21]] +let[[x := 21] in 2 * x] +let[2 * x, where[x := 21]] ``` These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the `...` in `let[...]`. @@ -223,20 +229,20 @@ The issue has been fixed in Python 3.9. If you already only use 3.9 and later, p The `let` constructs can use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body ([like in `multilambda`](#multilambda-supercharge-your-lambdas)): ```python -let[x << 1, - y << 2][[ # note extra [ - y << x + y, +let[x := 1, + y := 2][[ # note extra [ + y := x + y, print(y)]] -let[[x << 1, - y << 2] in - [y << x + y, # body starts here +let[[x := 1, + y := 2] in + [y := x + y, # body starts here print(y)]] -let[[y << x + y, +let[[y := x + y, print(y)], # body ends here - where[x << 1, - y << 2]] + where[x := 1, + y := 2]] ``` The let macros implement this by inserting a `do[...]` (see below). In a multiple-expression body, a separate internal definition context exists for local variables that are not part of the `let`; see [the `do` macro for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). @@ -244,17 +250,17 @@ The let macros implement this by inserting a `do[...]` (see below). In a multipl Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a `let` form with only one body expression, double the brackets on the *body* part: ```python -let[x << 1, - y << 2][[ +let[x := 1, + y := 2][[ [x, y]]] -let[[x << 1, - y << 2] in +let[[x := 1, + y := 2] in [[x, y]]] let[[[x, y]], - where[x << 1, - y << 2]] + where[x := 1, + y := 2]] ``` The outermost brackets delimit the `let` form itself, the middle ones activate multiple-expression mode, and the innermost ones denote a list. @@ -262,31 +268,33 @@ The outermost brackets delimit the `let` form itself, the middle ones activate m Only brackets are affected; parentheses are interpreted as usual, so returning a literal tuple works as expected: ```python -let[x << 1, - y << 2][ +let[x := 1, + y := 2][ (x, y)] -let[[x << 1, - y << 2] in +let[[x := 1, + y := 2] in (x, y)] let[(x, y), - where[x << 1, - y << 2]] + where[x := 1, + y := 2]] ``` #### Notes -The main difference of the `let` family to Python's own named expressions (a.k.a. the walrus operator, added in Python 3.8) is that `x := 42` does not create a scope, but `let[x << 42][...]` does. The walrus operator assigns to the name `x` in the scope it appears in, whereas in the `let` expression, the `x` only exists in that expression. +The main difference of the `let` family to Python's own named expressions (a.k.a. the walrus operator, added in Python 3.8) is that `x := 42` does not create a scope, but `let[x := 42][...]` does. The walrus operator assigns to the name `x` in the scope it appears in, whereas in the `let` expression, the `x` only exists in that expression. -`let` and `letrec` expand into the `unpythonic.lispylet` constructs, implicitly inserting the necessary boilerplate: the `lambda e: ...` wrappers, quoting variable names in definitions, and transforming `x` to `e.x` for all `x` declared in the bindings. Assignment syntax `x << 42` transforms to `e.set('x', 42)`. The implicit environment parameter `e` is actually named using a gensym, so lexically outer environments automatically show through. `letseq` expands into a chain of nested `let` expressions. +As of v0.15.3, this is somewhat complicated by the fact that now the syntax `x := 42` can be used to rebind let variables. See the unit test examples for `@dlet` above, at the beginning of the `let` section. + +`let` and `letrec` expand into the `unpythonic.lispylet` constructs, implicitly inserting the necessary boilerplate: the `lambda e: ...` wrappers, quoting variable names in definitions, and transforming `x` to `e.x` for all `x` declared in the bindings. Assignment syntax `x := 42` transforms to `e.set('x', 42)`. The implicit environment parameter `e` is actually named using a gensym, so lexically outer environments automatically show through. `letseq` expands into a chain of nested `let` expressions. All the `let` macros respect lexical scope, so this works as expected: ```python -letrec[z << 1][[ +letrec[z := 1][[ print(z), - letrec[z << 2][ + letrec[z := 2][ print(z)]]] ``` @@ -304,83 +312,109 @@ Examples: ```python from unpythonic.syntax import macros, dlet, dletseq, dletrec, blet, bletseq, bletrec -@dlet[x << 0] # up to Python 3.8, use `@dlet(x << 0)` instead +@dlet[x := 0] # up to Python 3.8, use `@dlet(x := 0)` instead (decorator subscripting was added in 3.9) def count(): - x << x + 1 # update `x` in let env + (x := x + 1) # update `x` in let env return x assert count() == 1 assert count() == 2 -@dletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), - oddp << (lambda x: (x != 0) and evenp(x - 1))] +@dletrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), + oddp := (lambda x: (x != 0) and evenp(x - 1))] def f(x): return evenp(x) assert f(42) is True assert f(23) is False -@dletseq[x << 1, - x << x + 1, - x << x + 2] +@dletseq[x := 1, + x := x + 1, + x := x + 2] def g(a): return a + x assert g(10) == 14 # block versions: the def takes no arguments, runs immediately, and is replaced by the return value. -@blet[x << 21] +@blet[x := 21] def result(): return 2*x assert result == 42 -@bletrec[evenp << (lambda x: (x == 0) or oddp(x - 1)), - oddp << (lambda x: (x != 0) and evenp(x - 1))] +@bletrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), + oddp := (lambda x: (x != 0) and evenp(x - 1))] def result(): return evenp(42) assert result is True -@bletseq[x << 1, - x << x + 1, - x << x + 2] +@bletseq[x := 1, + x := x + 1, + x := x + 2] def result(): return x assert result == 4 ``` -**CAUTION**: assignment to the let environment uses the syntax `name << value`, as always with `unpythonic` environments. The standard Python syntax `name = value` creates a local variable, as usual - *shadowing any variable with the same name from the `let`*. +**CAUTION**: assignment to the let environment uses the assignment expression syntax `name := value`. The assignment statement `name = value` creates a local variable, as usual - *shadowing any variable with the same name from the `let`*. -The write of a `name << value` always occurs to the lexically innermost environment (as seen from the write site) that has that `name`. If no lexically surrounding environment has that `name`, *then* the expression remains untransformed, and means a left-shift (if `name` happens to be otherwise defined). +The write of a `name := value` always occurs to the lexically innermost environment (as seen from the write site) that has that `name`. If no lexically surrounding environment has that `name`, *then* the expression remains untransformed, and means binding a new lexical variable in the nearest enclosing scope, as per Python's standard rules. -**CAUTION**: formal parameters of a function definition, local variables, and any names declared as `global` or `nonlocal` in a given lexical scope shadow names from the `let` environment. Mostly, this applies *to the entirety of that lexical scope*. This is modeled after Python's standard scoping rules. +**CAUTION**: formal parameters of a function definition, local variables, and any names declared as `global` or `nonlocal` in a given lexical scope shadow names from an enclosing `let` environment. Mostly, this applies *to the entirety of that lexical scope*. This is modeled after Python's standard scoping rules. As an exception to the rule, for the purposes of the scope analysis performed by `unpythonic.syntax`, creations and deletions *of lexical local variables* take effect from the next statement, and remain in effect for the **lexically** remaining part of the current scope. This allows `x = ...` to see the old bindings on the RHS, as well as allows the client code to restore access to a surrounding env's `x` (by deleting a local `x` shadowing it) when desired. To clarify, here is a sampling from [the unit tests](../unpythonic/syntax/tests/test_letdo.py): ```python -@dlet[x << "the env x"] +@dlet[x := "the env x"] def f(): - return x + return x # No lexical variable `x` exists; this refers to the env `x`. assert f() == "the env x" -@dlet[x << "the env x"] +@dlet[x := "the env x"] def f(): - x = "the local x" + x = "the local x" # The lexical variable shadows the env `x`. return x assert f() == "the local x" -@dlet[x << "the env x"] +@dlet[x := "the env x"] def f(): return x - x = "the unused local x" + x = "the unused local x" # This appears *lexically after* the read access on the previous line. assert f() == "the env x" +@dlet[x := "the env x"] +def test15(): + def inner(): + (x := "updated env x") # noqa: F841, this writes to the let env since there is no `x` in an intervening scope, according to Python's standard rules. + inner() + return x +assert test15() == "updated env x" + +@dlet[x := "the env x"] +def test16(): + def inner(): + x = "the inner x" # noqa: F841, unused on purpose, for testing. An assignment *statement* does NOT write to the let env. + inner() + return x +assert test16() == "the env x" + +@dlet[x := "the env x"] +def test17(): + x = "the local x" # This lexical variable shadows the env x. + def inner(): + # The env x is shadowed. Since we don't say `nonlocal x`, this creates a new lexical variable scoped to `inner`. + (x := "the inner x") # noqa: F841, unused on purpose, for testing. + inner() + return x +assert test17() == "the local x" + x = "the global x" -@dlet[x << "the env x"] +@dlet[x := "the env x"] def f(): global x return x assert f() == "the global x" -@dlet[x << "the env x"] +@dlet[x := "the env x"] def f(): x = "the local x" del x # deleting a local, ok! @@ -389,7 +423,7 @@ assert f() == "the env x" try: x = "the global x" - @dlet[x << "the env x"] + @dlet[x := "the env x"] def f(): global x del x # ignored by unpythonic's scope analysis, deletion of globals is too dynamic @@ -464,28 +498,28 @@ def verylongfunctionname(x=1): return x # works as an expr macro -y = let_syntax[f << verylongfunctionname][[ # extra brackets: implicit do in body +y = let_syntax[f := verylongfunctionname][[ # extra brackets: implicit do in body print(f()), f(5)]] assert y == 5 -y = let_syntax[f[a] << verylongfunctionname(2*a)][[ # template with formal parameter "a" +y = let_syntax[f[a] := verylongfunctionname(2*a)][[ # template with formal parameter "a" print(f[2]), f[3]]] assert y == 6 -y = let_syntax[[f << verylongfunctionname] in +y = let_syntax[[f := verylongfunctionname] in [print(f()), f(5)]] y = let_syntax[[print(f()), f(5)], - where[f << verylongfunctionname]] -y = let_syntax[[f[a] << verylongfunctionname(2*a)] in + where[f := verylongfunctionname]] +y = let_syntax[[f[a] := verylongfunctionname(2*a)] in [print(f[2]), f[3]]] y = let_syntax[[print(f[2]), f[3]], - where[f[a] << verylongfunctionname(2*a)]] + where[f[a] := verylongfunctionname(2*a)]] # works as a block macro with let_syntax: @@ -545,8 +579,8 @@ The `expr` and `block` operators, if used, must be macro-imported. They may only > >Within each step, the substitutions are applied **in definition order**: > -> - If the bindings are `[x << y, y << z]`, then an `x` at the use site transforms to `z`. So does a `y` at the use site. -> - But if the bindings are `[y << z, x << y]`, then an `x` at the use site transforms to `y`, and only an explicit `y` at the use site transforms to `z`. +> - If the bindings are `[x := y, y := z]`, then an `x` at the use site transforms to `z`. So does a `y` at the use site. +> - But if the bindings are `[y := z, x := y]`, then an `x` at the use site transforms to `y`, and only an explicit `y` at the use site transforms to `z`. > >Even in block templates, arguments are always expressions, because invoking a template uses the subscript syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above. @@ -561,15 +595,15 @@ When used as an expr macro, all bindings are registered first, and then the body The `abbrev` macro is otherwise exactly like `let_syntax`, but it expands outside-in. Hence, it has no lexically scoped nesting support, but it has the power to locally rename also macros, because the `abbrev` itself expands before any macros invoked in its body. This allows things like: ```python -abbrev[m << macrowithverylongname][ +abbrev[m := macrowithverylongname][ m[tree1] if m[tree2] else m[tree3]] -abbrev[[m << macrowithverylongname] in +abbrev[[m := macrowithverylongname] in m[tree1] if m[tree2] else m[tree3]] abbrev[m[tree1] if m[tree2] else m[tree3], - where[m << macrowithverylongname]] + where[m := macrowithverylongname]] ``` -which is sometimes useful when writing macros. (But using `mcpyrate`, note that you can just as-import a macro if you need to rename it.) +which is sometimes useful when writing macros. But using `mcpyrate`, note that you can just as-import a macro if you need to rename it. **CAUTION**: `let_syntax` is essentially a toy macro system within the real macro system. The usual caveats of macro systems apply. Especially, `let_syntax` and `abbrev` support absolutely no form of hygiene. Be very, very careful to avoid name conflicts. @@ -609,6 +643,8 @@ Macros that run multiple expressions, in sequence, in place of one expression. ### `do` as a macro: stuff imperative code into an expression, *with style* +**Changed in v0.15.3.** *Env-assignments now use the walrus syntax `x := 42`. The old syntax `x << 42` is still supported for backward compatibility.* + We provide an `expr` macro wrapper for `unpythonic.do` and `unpythonic.do0`, with some extra features. This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see the functions in the module [`unpythonic.fploop`](../unpythonic/fploop.py) (`looped` and `looped_over`). @@ -616,21 +652,21 @@ This essentially allows writing imperative code in any expression position. For ```python from unpythonic.syntax import macros, do, local, delete -y = do[local[x << 17], +y = do[local[x := 17], print(x), - x << 23, + x := 23, x] print(y) # --> 23 a = 5 -y = do[local[a << 17], +y = do[local[a := 17], print(a), # --> 17 delete[a], print(a), # --> 5 True] ``` -Local variables are declared and initialized with `local[var << value]`, where `var` is a bare name. To explicitly denote "no value", just use `None`. The syntax `delete[...]` allows deleting a `local[...]` binding. This uses `env.pop()` internally, so a `delete[...]` returns the value the deleted local variable had at the time of deletion. (This also means that if you manually use the `do()` function in some code without macros, you can `env.pop(...)` in a do-item if needed.) +Local variables are declared and initialized with `local[var := value]`, where `var` is a bare name. To explicitly denote "no value", just use `None`. The syntax `delete[...]` allows deleting a `local[...]` binding. This uses `env.pop()` internally, so a `delete[...]` returns the value the deleted local variable had at the time of deletion. (This also means that if you manually use the `do()` function in some code without macros, you can `env.pop(...)` in a do-item if needed.) The `local[]` and `delete[]` declarations may only appear at the top level of a `do[]`, `do0[]`, or implicit `do` (extra bracket syntax, e.g. for the body of a `let` form). In any invalid position, `local[]` and `delete[]` are considered a syntax error at macro expansion time. @@ -638,13 +674,13 @@ A `local` declaration comes into effect in the expression following the one wher ```python result = [] -let[lst << []][[result.append(lst), # the let "lst" - local[lst << lst + [1]], # LHS: do "lst", RHS: let "lst" +let[lst := []][[result.append(lst), # the let "lst" + local[lst := lst + [1]], # LHS: do "lst", RHS: let "lst" result.append(lst)]] # the do "lst" assert result == [[], [1]] ``` -Already declared local variables are updated with `var << value`. Updating variables in lexically outer environments (e.g. a `let` surrounding a `do`) uses the same syntax. +Already declared local variables are updated with `var := value`. Updating variables in lexically outer environments (e.g. a `let` surrounding a `do`) uses the same syntax.
The reason we require local variables to be declared is to allow write access to lexically outer environments. @@ -677,21 +713,21 @@ with multilambda: echo = lambda x: [print(x), x] assert echo("hi there") == "hi there" - count = let[x << 0][ - lambda: [x << x + 1, # x belongs to the surrounding let + count = let[x := 0][ + lambda: [x := x + 1, # x belongs to the surrounding let x]] assert count() == 1 assert count() == 2 - test = let[x << 0][ - lambda: [x << x + 1, - local[y << 42], # y is local to the implicit do + test = let[x := 0][ + lambda: [x := x + 1, + local[y := 42], # y is local to the implicit do (x, y)]] assert test() == (1, 42) assert test() == (2, 42) myadd = lambda x, y: [print("myadding", x, y), - local[tmp << x + y], + local[tmp := x + y], print("result is", tmp), tmp] assert myadd(2, 3) == 5 @@ -716,14 +752,14 @@ from unpythonic.syntax import macros, namedlambda with namedlambda: f = lambda x: x**3 # assignment: name as "f" assert f.__name__ == "f" - gn, hn = let[x << 42, g << None, h << None][[ - g << (lambda x: x**2), # env-assignment: name as "g" - h << f, # still "f" (no literal lambda on RHS) + gn, hn = let[x := 42, g := None, h := None][[ + g := (lambda x: x**2), # env-assignment: name as "g" + h := f, # still "f" (no literal lambda on RHS) (g.__name__, h.__name__)]] assert gn == "g" assert hn == "f" - foo = let[[f7 << (lambda x: x)] in f7] # let-binding: name as "f7" + foo = let[[f7 := (lambda x: x)] in f7] # let-binding: name as "f7" def foo(func1, func2): assert func1.__name__ == "func1" @@ -750,10 +786,10 @@ The naming is performed using the function `unpythonic.namelambda`, which will r - Named expressions (a.k.a. walrus operator, Python 3.8+), `f := lambda ...: ...`. **Added in v0.15.0.** - - Expression-assignment to an unpythonic environment, `f << (lambda ...: ...)` + - Expression-assignment to an unpythonic environment, `f := (lambda ...: ...)`, and the old syntax `f << (lambda ...: ...)`. - Env-assignments are processed lexically, just like regular assignments. This should not cause problems, because left-shifting by a literal lambda most often makes no sense (whence, that syntax is *almost* guaranteed to mean an env-assignment). - - Let-bindings, `let[[f << (lambda ...: ...)] in ...]`, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). + - Let-bindings, `let[[f := (lambda ...: ...)] in ...]`, using any let syntax supported by unpythonic (here using the haskelly let-in with env-assign style bindings just as an example). - Named argument in a function call, as in `foo(f=lambda ...: ...)`. **Added in v0.14.2.** @@ -804,8 +840,8 @@ from unpythonic.syntax import macros, multilambda, quicklambda, fn, local from unpythonic.syntax import _ # optional, makes IDEs happy with quicklambda, multilambda: - func = fn[[local[x << _], - local[y << _], + func = fn[[local[x := _], + local[y := _], x + y]] assert func(1, 2) == 3 ``` @@ -861,26 +897,28 @@ Let's use the let-over-lambda idiom: ```python def foo(n0): - return let[[n << n0] in - (lambda i: n << n + i)] + return let[[n := n0] in + (lambda i: (n := n + i))] ``` -This is already shorter, but the `let` is used only for (in effect) altering the passed-in value of `n0`; we do not place any other variables into the `let` environment. Considering the source text already introduces a name `n0` which is just used to initialize `n`, that's an extra element that could be eliminated. +This is already shorter, but the `let` is used only for (in effect) storing the passed-in value of `n0`; we do not place any other variables into the `let` environment. Considering the source text already introduces a name `n0` which is just used to initialize `n`, that's an extra element that could be eliminated. Enter the `envify` macro, which automates this: ```python with envify: def foo(n): - return lambda i: n << n + i + return lambda i: (n := n + i) ``` +Note this does not work without `envify`, because then the assignment expression will create a local variable (local to the lambda) instead of rebinding the outer existing `n`. + Combining with `autoreturn` yields the fewest-source-code-elements optimal solution to the accumulator puzzle: ```python with autoreturn, envify: def foo(n): - lambda i: n << n + i + lambda i: (n := n + i) ``` The `with` block adds a few elements, but if desired, it can be refactored into the definition of a custom dialect using `mcpyrate`. See [dialect examples](dialects.md). @@ -1234,6 +1272,8 @@ Hence, if porting some code that uses `call/cc` from Racket to Python, in the Py Observe that while our outermost `call_cc` already somewhat acts like a prompt (in the sense of delimited continuations), we are currently missing the ability to set a prompt wherever (inside code that already uses `call_cc` somewhere) and make the continuation terminate there. So what we have right now is something between proper delimited continuations and classic whole-computation continuations - not really [co-values](http://okmij.org/ftp/continuations/undelimited.html), but not really delimited continuations, either. +(TODO: If I interpret the wiki page right, our `call_cc` performs the job of `reset`; the called function forms the body of the `reset`. The `cc` argument passed into the called function performs the job of `shift`.) + For various possible program topologies that continuations may introduce, see [these clarifying pictures](callcc_topology.pdf). For full documentation, see the docstring of `unpythonic.syntax.continuations`. The unit tests [[1]](../unpythonic/syntax/tests/test_conts.py) [[2]](../unpythonic/syntax/tests/test_conts_escape.py) [[3]](../unpythonic/syntax/tests/test_conts_gen.py) [[4]](../unpythonic/syntax/tests/test_conts_topo.py) may also be useful as usage examples. @@ -1761,6 +1801,8 @@ For code using **conditions and restarts**: there is no special integration betw ### `forall`: nondeterministic evaluation +**Changed in v0.15.3.** *Env-assignment now uses the assignment expression syntax `x := range(3)`. The old syntax `x << range(3)` is still supported for backward compatibility.* + This is essentially a macro implementation of Haskell's do-notation for Python, specialized to the List monad. The `forall[]` expr macro behaves the same as the multiple-body-expression tuple comprehension `unpythonic.forall`, but the macro is implemented purely by AST transformation, using real lexical variables. @@ -1771,23 +1813,23 @@ The implementation is generic and very short; if interested, see the module [`un from unpythonic.syntax import macros, forall from unpythonic.syntax import insist, deny # regular functions, not macros -out = forall[y << range(3), - x << range(3), +out = forall[y := range(3), + x := range(3), insist(x % 2 == 0), (x, y)] assert out == ((0, 0), (2, 0), (0, 1), (2, 1), (0, 2), (2, 2)) # pythagorean triples -pt = forall[z << range(1, 21), # hypotenuse - x << range(1, z+1), # shorter leg - y << range(x, z+1), # longer leg +pt = forall[z := range(1, 21), # hypotenuse + x := range(1, z+1), # shorter leg + y := range(x, z+1), # longer leg insist(x*x + y*y == z*z), (x, y, z)] assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), (8, 15, 17), (9, 12, 15), (12, 16, 20)) ``` -Assignment, **with** List-monadic magic, is `var << iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). +Assignment, **with** List-monadic magic, is `var := iterable`. It is only valid at the top level of the `forall` (e.g. not inside any possibly nested `let`). `insist` and `deny` are not macros; they are just the functions from `unpythonic.amb`, re-exported for convenience. From f8a1bf65045d2e949b28020308bc5fc7aabef522 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:29:30 +0300 Subject: [PATCH 708/832] estimate: don't bother if no items remaining --- unpythonic/timeutil.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/timeutil.py b/unpythonic/timeutil.py index 075fd1c1..b56924ae 100644 --- a/unpythonic/timeutil.py +++ b/unpythonic/timeutil.py @@ -102,6 +102,8 @@ def _estimate(self) -> typing.Optional[float]: # Maybe we could use a Lanczos downsampling filter to make the ETA # behave more smoothly? remaining = self.total - self.completed + if remaining <= 0: + return 0.0 dt_avg = sum(self.que) / len(self.que) return remaining * dt_avg estimate = property(fget=_estimate, doc="Estimate of time remaining, in seconds. Computed when read; read-only. If no tasks have been marked completed yet, the estimate is `None`.") From 83dd632e7192a3ac5de2bffb284129e54303ac18 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:33:25 +0300 Subject: [PATCH 709/832] update CHANGELOG --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8fee44da..735625da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,20 @@ - Walrus syntax `name := value` is now supported, and preferred, for all env-assignments. Old syntax `name << value` still works, and will remain working at least until v0.16.0, whenever that is. +**Fixed**: + +- `ETAEstimator` edge case: at any point after all tasks have been marked completed, return a constant zero estimate for the remaining time. + **IMPORTANT**: - Minimum Python language version is now 3.8. - Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. Code has not been fully cleaned of historical cruft yet, so parts of it may still work in these versions. - 3.8 becomes EOL after October 2024, so support for that version might be dropped soon, too. +**Future plans**: + +Near-term focus will likely be on introducing support for Python 3.11 and 3.12, with no major changes to functionality. No promises though (except of the `lazy[]`/`force()` kind, which see). + --- From 5e3fed7b94510ad60aae352ab06c0ace97db5e5e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 12:53:13 +0300 Subject: [PATCH 710/832] update python_requires --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index d95e890f..8ff52f82 100644 --- a/setup.py +++ b/setup.py @@ -74,7 +74,7 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", "dynamic-variable", "macros", "lisp", "scheme", "racket", "haskell"], install_requires=[], # mcpyrate is optional for us, so we can't really put it here even though we recommend it. - python_requires=">=3.6,<3.11", + python_requires=">=3.8,<3.11", author="Juha Jeronen", author_email="juha.m.jeronen@gmail.com", url="https://github.com/Technologicat/unpythonic", From 9660f4b917fc96cf2f3a9e82372ae90d56669910 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:32:06 +0300 Subject: [PATCH 711/832] update codebase for minimum Python version 3.8 - Remove some ImportError checks imports that would fail on Python 3.6 and 3.7 - Remove check for availability of `exc.with_traceback` (was added in 3.7) - Update comments and docstrings Some of the old support code in complex, rarely used parts (e.g. the typechecker we use to implement multiple dispatch) has been left as-is. --- unpythonic/__init__.py | 2 +- unpythonic/arity.py | 12 ++++---- unpythonic/conditions.py | 21 +++++--------- unpythonic/excutil.py | 29 +++++++------------ unpythonic/let.py | 27 ++++------------- unpythonic/syntax/astcompat.py | 2 +- unpythonic/syntax/scopeanalyzer.py | 2 +- unpythonic/syntax/tests/test_scopeanalyzer.py | 5 ++-- unpythonic/tests/test_arity.py | 8 ----- unpythonic/tests/test_conditions.py | 1 - unpythonic/tests/test_dispatch.py | 2 +- unpythonic/tests/test_excutil.py | 9 ++---- unpythonic/tests/test_symbol.py | 2 +- unpythonic/typecheck.py | 20 +++++-------- 14 files changed, 47 insertions(+), 95 deletions(-) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index c9701cb3..ca8974c4 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -26,7 +26,7 @@ from .gmemo import * # noqa: F401, F403 from .gtco import * # noqa: F401, F403 from .it import * # noqa: F401, F403 -from .let import * # no guarantees on evaluation order (before Python 3.6), nice syntax # noqa: F401, F403 +from .let import * # # noqa: F401, F403 # As of 0.15.0, lispylet is nowadays primarily a code generation target API for macros. from .lispylet import (let as ordered_let, letrec as ordered_letrec, # noqa: F401 diff --git a/unpythonic/arity.py b/unpythonic/arity.py index ec4e3b9e..1ffc460f 100644 --- a/unpythonic/arity.py +++ b/unpythonic/arity.py @@ -21,7 +21,7 @@ class UnknownArity(ValueError): """Raised when the arity of a function cannot be inspected.""" # HACK: some built-ins report incorrect arities (0, 0) at least in Python 3.4 -# TODO: re-test on 3.8 and on PyPy3 (3.7), just to be sure. +# TODO: re-test on 3.8, 3.9, 3.10, 3.11, 3.12 and on PyPy3 (3.8 and later), just to be sure. # # Full list of built-ins: # https://docs.python.org/3/library/functions.html @@ -208,7 +208,7 @@ def arities(f): This uses inspect.signature; note that the signature of builtin functions cannot be inspected. This is worked around to some extent, but e.g. methods of built-in classes (such as ``list``) might not be inspectable - (at least on CPython < 3.7). + (at least on old CPython < 3.7). For bound methods, ``self`` or ``cls`` does not count toward the arity, because these are passed implicitly by Python. Note a `@classmethod` becomes @@ -352,10 +352,10 @@ def resolve_bindings(f, *args, **kwargs): This is an inspection tool, which does not actually call `f`. This is useful for memoizers and other similar decorators that need a canonical representation of `f`'s parameter bindings. - **NOTE**: As of v0.15.0, this is a thin wrapper on top of `inspect.Signature.bind`, - which was added in Python 3.5. In `unpythonic` 0.14.2 and 0.14.3, we used to have - our own implementation of the parameter binding algorithm (that ran also on Python 3.4), - but it is no longer needed, since now we support only Python 3.6 and later. + **NOTE**: This is a thin wrapper on top of `inspect.Signature.bind`, which was added in Python 3.5. + In `unpythonic` 0.14.2 and 0.14.3, we used to have our own implementation of the parameter binding + algorithm (that ran also on Python 3.4), but it is no longer needed, since as of v0.15.3, + we support only Python 3.8 and later. The only thing we do beside call `inspect.Signature.bind` is that we apply default values (from the definition of `f`) automatically. diff --git a/unpythonic/conditions.py b/unpythonic/conditions.py index 44088bc3..a02e853c 100644 --- a/unpythonic/conditions.py +++ b/unpythonic/conditions.py @@ -123,10 +123,10 @@ def signal(condition, *, cause=None, protocol=None): The return value is the input `condition`, canonized to an instance (even if originally, an exception *type* was passed to `signal`), with its `__cause__` and `__protocol__` attributes filled in, - and with a traceback attached (on Python 3.7+). For example, the - `error` protocol uses the return value to chain the unhandled signal - properly into a `ControlError` exception; as a result, the error report - looks like a standard exception chain, with nice-looking tracebacks. + and with a traceback attached. For example, the `error` protocol + uses the return value to chain the unhandled signal properly into + a `ControlError` exception; as a result, the error report looks + like a standard exception chain, with nice-looking tracebacks. If you want to error out on unhandled conditions, see `error`, which is otherwise the same as `signal`, except it raises if `signal` would have @@ -162,9 +162,8 @@ def signal(condition, *, cause=None, protocol=None): You can signal any exception or warning object, both builtins and any custom ones. - On Python 3.7 and later, the exception object representing the signaled - condition is equipped with a traceback, just like a raised exception. - On Python 3.6 this is not possible, so the traceback is `None`. + The exception object representing the signaled condition is equipped + with a traceback, just like a raised exception. """ # Since the handler is called normally, we don't unwind the call stack, # remaining inside the `signal()` call in the low-level code. @@ -225,12 +224,8 @@ def canonize(exc, err_reason): condition.__cause__ = cause condition.__protocol__ = protocol - # Embed a stack trace in the signal, like Python does for raised exceptions. - # This only works on Python 3.7 and later, because we need to create a traceback object in pure Python code. - try: - condition = equip_with_traceback(condition, stacklevel=stacklevel) - except NotImplementedError: # pragma: no cover - pass # well, we tried! + # Embed a stack trace in the signal, like Python does for raised exceptions. This API was added in Python 3.7. + condition = equip_with_traceback(condition, stacklevel=stacklevel) return condition diff --git a/unpythonic/excutil.py b/unpythonic/excutil.py index a09c1ec6..fc9c6b26 100644 --- a/unpythonic/excutil.py +++ b/unpythonic/excutil.py @@ -166,10 +166,6 @@ def equip_with_traceback(exc, stacklevel=1): # Python 3.7+ The return value is `exc`, with its traceback set to the produced traceback. - Python 3.7 and later only. - - When not supported, raises `NotImplementedError`. - This is useful mainly in special cases, where `raise` cannot be used for some reason, and a manually created exception instance needs a traceback. (The `signal` function in the conditions-and-restarts system uses this.) @@ -207,20 +203,17 @@ def equip_with_traceback(exc, stacklevel=1): # Python 3.7+ break # Python 3.7+ allows creating `types.TracebackType` objects in Python code. - try: - tracebacks = [] - nxt = None # tb_next should point toward the level where the exception occurred. - for frame in frames: # walk from top of call stack toward the root - tb = TracebackType(nxt, frame, frame.f_lasti, frame.f_lineno) - tracebacks.append(tb) - nxt = tb - if tracebacks: - tb = tracebacks[-1] # root level - else: - tb = None - except TypeError as err: # Python 3.6 or earlier - raise NotImplementedError("Need Python 3.7 or later to create traceback objects") from err - return exc.with_traceback(tb) # Python 3.7+ + tracebacks = [] + nxt = None # tb_next should point toward the level where the exception occurred. + for frame in frames: # walk from top of call stack toward the root + tb = TracebackType(nxt, frame, frame.f_lasti, frame.f_lineno) + tracebacks.append(tb) + nxt = tb + if tracebacks: + tb = tracebacks[-1] # root level + else: + tb = None + return exc.with_traceback(tb) # TODO: To reduce the risk of spaghetti user code, we could require a non-main thread's entrypoint to declare # via a decorator that it's willing to accept asynchronous exceptions, and check that mark here, making this diff --git a/unpythonic/let.py b/unpythonic/let.py index e457a46e..3b1a2536 100644 --- a/unpythonic/let.py +++ b/unpythonic/let.py @@ -106,25 +106,8 @@ def letrec(body, **bindings): body=lambda e: e.b * e.f(1)) # --> 84 - **CAUTION**: - - Simple values (non-callables) may depend on earlier definitions - in the same letrec **only in Python 3.6 and later**. - - Until Python 3.6, initialization of the bindings occurs - **in an arbitrary order**, because of the ``kwargs`` mechanism. - See PEP 468: - - https://www.python.org/dev/peps/pep-0468/ - - In Python < 3.6, in the first example above, trying to reference ``env.a`` - on the RHS of ``b`` may get either the ``lambda e: ...``, or the value ``1``, - depending on whether the binding ``a`` has been initialized at that point or not. - - If you need left-to-right initialization of bindings in Python < 3.6, - see ``unpythonic.lispylet``. - - The following applies regardless of Python version. + Simple values (non-callables) may depend on earlier definitions + in the same letrec. A callable value may depend on **any** binding, also later ones. This allows mutually recursive functions:: @@ -151,9 +134,9 @@ def letrec(body, **bindings): L = [1, 1, 3, 1, 3, 2, 3, 2, 2, 2, 4, 4, 1, 2, 3] print(u(L)) # [1, 3, 2, 4] - Works also in Python < 3.6, because here ``see`` is a callable. Hence, ``e.seen`` - doesn't have to exist when the *definition* of ``see`` is evaluated; it only has to - exist when ``e.see(x)`` is *called*. + Note that ``see`` is a callable. Hence, strictly speaking it doesn't matter + if ``e.seen`` exists when the *definition* of ``see`` is evaluated; it only + has to exist when ``e.see(x)`` is *called*. Parameters: `body`: function diff --git a/unpythonic/syntax/astcompat.py b/unpythonic/syntax/astcompat.py index 32cbc4e1..68fa5a0b 100644 --- a/unpythonic/syntax/astcompat.py +++ b/unpythonic/syntax/astcompat.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""Conditionally import AST node types only supported by recent enough Python versions (3.7+).""" +"""Conditionally import AST node types only supported by recent enough Python versions.""" __all__ = ["NamedExpr", "Num", "Str", "Bytes", "NameConstant", "Ellipsis", diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index bfa13e3e..c39180a6 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -369,7 +369,7 @@ class DelNamesCollector(ASTVisitor): def examine(self, tree): # We want to detect things like "del x": # Delete(targets=[Name(id='x', ctx=Del()),]) - # We don't currently care about "del myobj.x" or "del mydict['x']" (these examples in Python 3.6): + # We don't currently care about "del myobj.x" or "del mydict['x']" (these old examples in Python 3.6): # Delete(targets=[Attribute(value=Name(id='myobj', ctx=Load()), attr='x', ctx=Del()),]) # Delete(targets=[Subscript(value=Name(id='mydict', ctx=Load()), slice=Index(value=Str(s='x')), ctx=Del()),]) if type(tree) is Name and hasattr(tree, "ctx") and type(tree.ctx) is Del: diff --git a/unpythonic/syntax/tests/test_scopeanalyzer.py b/unpythonic/syntax/tests/test_scopeanalyzer.py index 2c2b0927..0261b4a8 100644 --- a/unpythonic/syntax/tests/test_scopeanalyzer.py +++ b/unpythonic/syntax/tests/test_scopeanalyzer.py @@ -59,9 +59,8 @@ def sleep(): # Assignment # - # At least up to Python 3.7, all assignments produce Name nodes in - # Store context on their LHS, so we don't need to care what kind of - # assignment it is. + # All assignments produce Name nodes in #tore context on their LHS, + # so we don't need to care what kind of assignment it is. test[get_names_in_store_context(getnames_store_simple) == ["x"]] with q as getnames_tuple: x, y = 1, 2 # noqa: F841 diff --git a/unpythonic/tests/test_arity.py b/unpythonic/tests/test_arity.py index 8716ecd0..46977c25 100644 --- a/unpythonic/tests/test_arity.py +++ b/unpythonic/tests/test_arity.py @@ -104,14 +104,6 @@ def instmeth(self): test[arities(target.classmeth) == (1, 1)] test[arities(target.staticmeth) == (1, 1)] - # Methods of builtin types have uninspectable arity up to Python 3.6. - # Python 3.7 seems to fix this at least for `list`, and PyPy3 (7.3.0; Python 3.6.9) - # doesn't have this error either. - if sys.version_info < (3, 7, 0) and sys.implementation.name == "cpython": # pragma: no cover - with testset("uninspectable builtin methods"): - lst = [] - test_raises[UnknownArity, arities(lst.append)] - # resolve_bindings: resolve parameter bindings established by a function # when it is called with the given args and kwargs. # diff --git a/unpythonic/tests/test_conditions.py b/unpythonic/tests/test_conditions.py index 3f7a529e..d7587832 100644 --- a/unpythonic/tests/test_conditions.py +++ b/unpythonic/tests/test_conditions.py @@ -408,7 +408,6 @@ def warn_protocol(): # An unhandled `error` or `cerror`, when it **raises** `ControlError`, # sets the cause of that `ControlError` to the original unhandled signal. # In Python 3.7+, this will also produce nice stack traces. - # In up to Python 3.6, it will at least show the chain of causes. with catch_signals(False): try: exc1 = JustTesting("Hullo") diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index fb31df1e..e5efabbc 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -272,7 +272,7 @@ def blubnify2(x: float, y: float): with testset("list_methods"): def check_formatted_multimethods(result, expected): - def _remove_space_before_typehint(string): # Python 3.6 doesn't print a space there + def _remove_space_before_typehint(string): # Python 3.6 didn't print a space there, later versions do return string.replace(": ", ":") result_list = result.split("\n") human_readable_header, *multimethod_descriptions = result_list diff --git a/unpythonic/tests/test_excutil.py b/unpythonic/tests/test_excutil.py index 858122cf..926bfb53 100644 --- a/unpythonic/tests/test_excutil.py +++ b/unpythonic/tests/test_excutil.py @@ -78,13 +78,8 @@ def runtests(): with testset("equip_with_traceback"): e = Exception("just testing") - try: - e = equip_with_traceback(e) - except NotImplementedError: - warn["equip_with_traceback only supported on Python 3.7+, skipping test."] - else: - # Can't do meaningful testing on the result, so just check it's there. - test[e.__traceback__ is not None] + e = equip_with_traceback(e) + test[e.__traceback__ is not None] # Can't do meaningful testing on the result, so just check it's there. test_raises[TypeError, equip_with_traceback("not an exception")] diff --git a/unpythonic/tests/test_symbol.py b/unpythonic/tests/test_symbol.py index 476384b1..349271f2 100644 --- a/unpythonic/tests/test_symbol.py +++ b/unpythonic/tests/test_symbol.py @@ -34,7 +34,7 @@ def runtests(): # Symbol interning has nothing to do with string interning. many = 5000 test[the[sym("λ" * many) is sym("λ" * many)]] - # To defeat string interning, used to be that 80 exotic characters + # To defeat string interning, it used to be that 80 exotic characters # would be enough in Python 3.6 to make CPython decide not to intern it, # but Python 3.7 bumped that up. test[the["λ" * many is not "λ" * many]] diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 1ab400bf..228c9dd4 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -18,17 +18,8 @@ import sys import typing -try: - _MyGenericAlias = typing._GenericAlias # Python 3.7+ -except AttributeError: # Python 3.6 and earlier # pragma: no cover - class _MyGenericAlias: # unused, but must be a class to support isinstance() check. - pass - -try: - _MySupportsIndex = typing.SupportsIndex # Python 3.8+ -except AttributeError: # Python 3.7 and earlier # pragma: no cover - class _MySupportsIndex: # unused, but must be a class to support isinstance() check. - pass +_MyGenericAlias = typing._GenericAlias # Python 3.7+ +_MySupportsIndex = typing.SupportsIndex # Python 3.8+ from .misc import safeissubclass @@ -111,6 +102,11 @@ def isoftype(value, T): # TODO: as of Python 3.8 (March 2020). https://docs.python.org/3/library/typing.html # TODO: If you add a feature to the type checker, please update this list. # + # TODO: Update this list for Python 3.9 + # TODO: Update this list for Python 3.10 + # TODO: Update this list for Python 3.11 + # TODO: Update this list for Python 3.12 + # # Python 3.6+: # NamedTuple, DefaultDict, Counter, ChainMap, # IO, TextIO, BinaryIO, @@ -190,7 +186,7 @@ def isNewType(T): # In Python 3.10, an instance of `typing.NewType` is now actually such and not just a function. Nice! if sys.version_info >= (3, 10, 0): return isinstance(T, typing.NewType) - # Python 3.6 through Python 3.9 + # Python 3.6, Python 3.7, Python 3.8, Python 3.9 # TODO: in Python 3.7+, what is the mysterious callable that doesn't have a `__qualname__`? return callable(T) and hasattr(T, "__qualname__") and T.__qualname__ == "NewType..new_type" if isNewType(T): From 53a875dcb39b132847c081d5d138bf563d93bca0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:38:50 +0300 Subject: [PATCH 712/832] CI: use Python 3.10 for coverage analysis --- .github/workflows/coverage.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index fc40d2ce..9b24ebb4 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.8] + python-version: [3.10] steps: - uses: actions/checkout@v2 From 47c659543393c8f0f54187314cbf66e9ad0e40c1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:39:00 +0300 Subject: [PATCH 713/832] CI: test on 3.8, 3.9, 3.10 --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 8be890d7..fe0eeb05 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy-3.6, pypy-3.7, pypy-3.8] + python-version: [3.8, 3.9, "3.10", pypy-3.8, pypy-3.9, pypy-3.10] steps: - uses: actions/checkout@v2 From 2cb63903fb6b3be5be2e2642c1d58724d5d59514 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:40:43 +0300 Subject: [PATCH 714/832] ugh, walrus inside subscript is only supported on 3.10+? --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index fe0eeb05..f1bac4f0 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.8, 3.9, "3.10", pypy-3.8, pypy-3.9, pypy-3.10] + python-version: ["3.10", pypy-3.10] steps: - uses: actions/checkout@v2 From 2b028fdf3d183d1ae06b5a346d4c65fb2d91e4d7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:41:35 +0300 Subject: [PATCH 715/832] ah yes, the infamous implicit float conversion 3.10 -> 3.1. --- .github/workflows/coverage.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 9b24ebb4..76d035ef 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [3.10] + python-version: ["3.10"] steps: - uses: actions/checkout@v2 From 76259018ee9fc54072c6c9765cb6a307e72412a0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:52:28 +0300 Subject: [PATCH 716/832] bump SymPy to 1.13 (for mathseq) --- requirements.txt | 2 +- unpythonic/mathseq.py | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 1840f0d6..f9ca83dd 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ mcpyrate>=3.6.0 -sympy>=1.4 +sympy>=1.13 diff --git a/unpythonic/mathseq.py b/unpythonic/mathseq.py index 03250e80..dc35342e 100644 --- a/unpythonic/mathseq.py +++ b/unpythonic/mathseq.py @@ -57,6 +57,11 @@ class _NoSuchType: mpf = _NoSuchType mpf_almosteq = None +try: + import sympy +except ImportError: # pragma: no cover, optional at runtime, but installed at development time. + sympy = None + def _numsign(x): """The sign function, for numeric inputs.""" if x == 0: @@ -265,6 +270,8 @@ def s(*spec): def is_almost_int(x): try: + if sympy and isinstance(x, sympy.Expr): + x = sympy.N(x) return almosteq(float(round(x)), x) except TypeError: # likely a SymPy expression that didn't simplify to a number return False From e62fe6f4e412f8e2ef42d330903f5aef4bb9e997 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:56:13 +0300 Subject: [PATCH 717/832] CI: re-enable 3.8 and 3.9. Let's make those work, too... --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index f1bac4f0..14ab32fb 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.10", pypy-3.10] + python-version: ["3.8", "3.9", "3.10", pypy-3.8, pypy-3.9, pypy-3.10] steps: - uses: actions/checkout@v2 From 26ed664db7a7cd26d6dacf0aeb73f1cd5a7639bd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 13:58:46 +0300 Subject: [PATCH 718/832] fix CI badge URL (https://github.com/badges/shields/issues/8671) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 49a97a31..8345c2b4 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing features for Python, mainly from the list processing tradition, but with some Haskellisms mixed in. We extend the language with a set of [syntactic macros](https://en.wikipedia.org/wiki/Macro_(computer_science)#Syntactic_macros). We also provide an in-process, background [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) server for live inspection and hot-patching. The emphasis is on **clear, pythonic syntax**, **making features work together**, and **obsessive correctness**. -![100% Python](https://img.shields.io/github/languages/top/Technologicat/unpythonic) ![supported language versions](https://img.shields.io/pypi/pyversions/unpythonic) ![supported implementations](https://img.shields.io/pypi/implementation/unpythonic) ![CI status](https://img.shields.io/github/workflow/status/Technologicat/unpythonic/Python%20package) [![codecov](https://codecov.io/gh/Technologicat/unpythonic/branch/master/graph/badge.svg)](https://codecov.io/gh/Technologicat/unpythonic) +![100% Python](https://img.shields.io/github/languages/top/Technologicat/unpythonic) ![supported language versions](https://img.shields.io/pypi/pyversions/unpythonic) ![supported implementations](https://img.shields.io/pypi/implementation/unpythonic) ![CI status](https://img.shields.io/github/actions/workflow/status/Technologicat/unpythonic/python-package.yml?branch=master) [![codecov](https://codecov.io/gh/Technologicat/unpythonic/branch/master/graph/badge.svg)](https://codecov.io/gh/Technologicat/unpythonic) ![version on PyPI](https://img.shields.io/pypi/v/unpythonic) ![PyPI package format](https://img.shields.io/pypi/format/unpythonic) ![dependency status](https://img.shields.io/librariesio/github/Technologicat/unpythonic) ![license: BSD](https://img.shields.io/pypi/l/unpythonic) ![open issues](https://img.shields.io/github/issues/Technologicat/unpythonic) [![PRs welcome](https://img.shields.io/badge/PRs-welcome-brightgreen)](http://makeapullrequest.com/) From 49b772c9dc8cefbbea9b8817dd3cd724541c73df Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:14:10 +0300 Subject: [PATCH 719/832] fix Python 3.8 and 3.9 compatibility Walrus inside subscript (without parentheses) was added in Python 3.10. --- unpythonic/syntax/letdo.py | 9 +++- unpythonic/syntax/tests/test_letdo.py | 51 +++++++++++++---------- unpythonic/syntax/tests/test_letdoutil.py | 38 +++++++++-------- 3 files changed, 56 insertions(+), 42 deletions(-) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index e2cb6240..5bda9cf7 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -217,7 +217,7 @@ def dlet(tree, *, args, syntax, expander, **kw): @dlet[x := 0] def count(): - (x := x + 1) # walrus requires parens here; or use `x << x + 1` + (x := x + 1) return x assert count() == 1 assert count() == 2 @@ -926,7 +926,12 @@ def _do0(tree): raise SyntaxError("do0 body: expected a sequence of comma-separated expressions") # pragma: no cover elts = tree.elts # Use `local[]` and `do[]` as hygienically captured macros. - newelts = [q[a[_our_local][_do0_result := a[elts[0]]]], # noqa: F821, local[] defines it inside the do[]. + # + # Python 3.8 and Python 3.9 require the parens around the walrus when used inside a subscript. + # TODO: Remove the parens when we bump minimum Python to 3.10. + # From https://docs.python.org/3/whatsnew/3.10.html: + # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). + newelts = [q[a[_our_local][(_do0_result := a[elts[0]])]], # noqa: F821, local[] defines it inside the do[]. *elts[1:], q[_do0_result]] # noqa: F821 return q[a[_our_do][t[newelts]]] # do0[] is also just a do[] diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 9415a604..3d94167c 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -26,7 +26,12 @@ def runtests(): # (including nested ``let`` constructs and similar). # - No need for ``lambda e: ...`` wrappers. Inserted automatically, # so the lines are only evaluated as the underlying seq.do() runs. - d1 = do[local[x := 17], + # + # Python 3.8 and Python 3.9 require the parens around the walrus when used inside a subscript. + # TODO: Remove the parens (in all walrus-inside-subscript instances in this file) when we bump minimum Python to 3.10. + # From https://docs.python.org/3/whatsnew/3.10.html: + # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). + d1 = do[local[(x := 17)], print(x), x := 23, x] @@ -37,7 +42,7 @@ def runtests(): # v0.14.0: do[] now supports deleting previously defined local names with delete[] a = 5 - d = do[local[a := 17], # noqa: F841, yes, d is unused. + d = do[local[(a := 17)], # noqa: F841, yes, d is unused. test[a == 17], delete[a], test[a == 5], # lexical scoping @@ -46,7 +51,7 @@ def runtests(): test_raises[KeyError, do[delete[a], ], "should have complained about deleting nonexistent local 'a'"] # do0[]: like do[], but return the value of the **first** expression - d2 = do0[local[y := 5], # noqa: F821, `local` defines the name on the LHS of the `<<`. + d2 = do0[local[(y := 5)], # noqa: F821, `local` defines the name on the LHS of the `<<`. print("hi there, y =", y), # noqa: F821 42] # evaluated but not used test[d2 == 5] @@ -75,30 +80,30 @@ def runtests(): # Let macros. Lexical scoping supported. with testset("let, letseq, letrec basic usage (new env-assignment syntax 0.15.3+)"): # parallel binding, i.e. bindings don't see each other - test[let[x := 17, - y := 23][ # noqa: F821, `let` defines `y` here. + test[let[(x := 17), + (y := 23)][ # noqa: F821, `let` defines `y` here. (x, y)] == (17, 23)] # noqa: F821 # sequential binding, i.e. Scheme/Racket let* - test[letseq[x := 1, - y := x + 1][ # noqa: F821 + test[letseq[(x := 1), + (y := x + 1)][ # noqa: F821 (x, y)] == (1, 2)] # noqa: F821 - test[letseq[x := 1, - x := x + 1][ # in a letseq, rebinding the same name is ok + test[letseq[(x := 1), + (x := x + 1)][ # in a letseq, rebinding the same name is ok x] == 2] # letrec sugars unpythonic.lispylet.letrec, removing the need for quotes on LHS # and "lambda e: ..." wrappers on RHS (these are inserted by the macro): - test[letrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821, `letrec` defines `evenp` here. - oddp := (lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 + test[letrec[(evenp := (lambda x: (x == 0) or oddp(x - 1))), # noqa: F821, `letrec` defines `evenp` here. + (oddp := (lambda x: (x != 0) and evenp(x - 1)))][ # noqa: F821 evenp(42)] is True] # noqa: F821 # nested letrecs work, too - each environment is internally named by a gensym # so that outer ones "show through": - test[letrec[z := 9000][ # noqa: F821 - letrec[evenp := (lambda x: (x == 0) or oddp(x - 1)), # noqa: F821 - oddp := (lambda x: (x != 0) and evenp(x - 1))][ # noqa: F821 + test[letrec[(z := 9000)][ # noqa: F821 + letrec[(evenp := (lambda x: (x == 0) or oddp(x - 1))), # noqa: F821 + (oddp := (lambda x: (x != 0) and evenp(x - 1)))][ # noqa: F821 (evenp(42), z)]] == (True, 9000)] # noqa: F821 with testset("let, letseq, letrec basic usage (previous modern env-assignment syntax)"): @@ -151,8 +156,8 @@ def runtests(): # implicit do: an extra set of brackets denotes a multi-expr body with testset("implicit do (extra bracket syntax for multi-expr let body) (new env-assignment syntax v0.15.3+)"): - a = let[x := 1, - y := 2][[ # noqa: F821 + a = let[(x := 1), + (y := 2)][[ # noqa: F821 y := 1337, # noqa: F821 (x, y)]] # noqa: F821 test[a == (1, 1337)] @@ -164,14 +169,14 @@ def runtests(): test[a == [1, 2]] # implicit do works also in letseq, letrec - a = letseq[x := 1, - y := x + 1][[ # noqa: F821 + a = letseq[(x := 1), + (y := x + 1)][[ # noqa: F821 x := 1337, (x, y)]] # noqa: F821 test[a == (1337, 2)] - a = letrec[x := 1, - y := x + 1][[ # noqa: F821 + a = letrec[(x := 1), + (y := x + 1)][[ # noqa: F821 x := 1337, (x, y)]] # noqa: F821 test[a == (1337, 2)] @@ -486,7 +491,7 @@ def test14(): x = "the nonlocal x" # restore the test environment # v0.15.3+: walrus syntax - @dlet[x := "the env x"] + @dlet[(x := "the env x")] def test15(): def inner(): (x := "updated env x") # noqa: F841, this writes to the let env since there is no `x` in an intervening scope, according to Python's standard rules. @@ -494,7 +499,7 @@ def inner(): return x test[test15() == "updated env x"] - @dlet[x := "the env x"] + @dlet[(x := "the env x")] def test16(): def inner(): x = "the inner x" # noqa: F841, unused on purpose, for testing. An assignment *statement* does NOT write to the let env. @@ -502,7 +507,7 @@ def inner(): return x test[test16() == "the env x"] - @dlet[x := "the env x"] + @dlet[(x := "the env x")] def test17(): x = "the local x" # This lexical variable shadows the env x. def inner(): diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index a5e46bb0..c0d82321 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -38,12 +38,16 @@ def validate(lst): if type(k) is not Name: return False # pragma: no cover, only reached if the test fails. return True + # Python 3.8 and Python 3.9 require the parens around the walrus when used inside a subscript. + # TODO: Remove the parens (in all walrus-inside-subscript instances in this file) when we bump minimum Python to 3.10. + # From https://docs.python.org/3/whatsnew/3.10.html: + # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). test[validate(the[canonize_bindings(q[k0, v0].elts)])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings(q[((k0, v0),)].elts)])] # noqa: F821 test[validate(the[canonize_bindings(q[(k0, v0), (k1, v1)].elts)])] # noqa: F821 - test[validate(the[canonize_bindings([q[k0 := v0]])])] # noqa: F821, it's quoted. + test[validate(the[canonize_bindings([q[(k0 := v0)]])])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings([q[k0 << v0]])])] # noqa: F821, it's quoted. - test[validate(the[canonize_bindings(q[k0 := v0, k1 := v1].elts)])] # noqa: F821, it's quoted. + test[validate(the[canonize_bindings(q[(k0 := v0), (k1 := v1)].elts)])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings(q[k0 << v0, k1 << v1].elts)])] # noqa: F821, it's quoted. # -------------------------------------------------------------------------------- @@ -53,7 +57,7 @@ def validate(lst): # need this utility, so we must test it first. with testset("isenvassign"): test[not isenvassign(q[x])] # noqa: F821 - test[isenvassign(q[x := 42])] # noqa: F821 + test[isenvassign(q[(x := 42)])] # noqa: F821 test[isenvassign(q[x << 42])] # noqa: F821 with testset("islet"): @@ -61,9 +65,9 @@ def validate(lst): test[not islet(q[f()])] # noqa: F821 # unpythonic 0.15.3+, Python 3.8+ - test[islet(the[expandrq[let[x := 21][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` + test[islet(the[expandrq[let[(x := 21)][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` test[islet(the[expandrq[let[[x := 21] in 2 * x]]]) == ("expanded_expr", "let")] # noqa: F821 - test[islet(the[expandrq[let[2 * x, where[x := 21]]]]) == ("expanded_expr", "let")] # noqa: F821 + test[islet(the[expandrq[let[2 * x, where[(x := 21)]]]]) == ("expanded_expr", "let")] # noqa: F821 # unpythonic 0.15.0 to 0.15.2, previous modern notation for bindings test[islet(the[expandrq[let[x << 21][2 * x]]]) == ("expanded_expr", "let")] # noqa: F821, `let` defines `x` @@ -96,7 +100,7 @@ def f2(): return 2 * x # noqa: F821 test[islet(the[testdata[0].decorator_list[0]]) == ("expanded_decorator", "let")] - testdata = q[let[x := 21][2 * x]] # noqa: F821 + testdata = q[let[(x := 21)][2 * x]] # noqa: F821 test[islet(the[testdata], expanded=False) == ("lispy_expr", "let")] testdata = q[let[x << 21][2 * x]] # noqa: F821 @@ -196,7 +200,7 @@ def f5(): test[not isdo(q[f()])] # noqa: F821 # unpythonic 0.15.3+, Python 3.8+ - test[isdo(the[expandrq[do[x := 21, # noqa: F821 + test[isdo(the[expandrq[do[(x := 21), # noqa: F821 2 * x]]]) == "expanded"] # noqa: F821 test[isdo(the[expandrq[do[x << 21, # noqa: F821 @@ -210,16 +214,16 @@ def f5(): test[isdo(the[thedo]) == "curried"] # unpythonic 0.15.3+, Python 3.8+ - testdata = q[do[x := 21, # noqa: F821 + testdata = q[do[(x := 21), # noqa: F821 2 * x]] # noqa: F821 test[isdo(the[testdata], expanded=False) == "do"] testdata = q[do0[23, # noqa: F821 - x := 21, # noqa: F821 + (x := 21), # noqa: F821 2 * x]] # noqa: F821 test[isdo(the[testdata], expanded=False) == "do0"] - testdata = q[someothermacro[x := 21, # noqa: F821 + testdata = q[someothermacro[(x := 21), # noqa: F821 2 * x]] # noqa: F821 test[not isdo(the[testdata], expanded=False)] @@ -241,7 +245,7 @@ def f5(): # Destructuring - envassign with testset("envassign destructuring (new env-assign syntax v0.15.3+)"): - testdata = q[x := 42] # noqa: F821 + testdata = q[(x := 42)] # noqa: F821 view = UnexpandedEnvAssignView(testdata) # read @@ -316,7 +320,7 @@ def testletdestructuring(testdata): test[unparse(view.body) == "(z * t)"] # lispy expr - testdata = q[let[x := 21, y := 2][y * x]] # noqa: F821 + testdata = q[let[(x := 21), (y := 2)][y * x]] # noqa: F821 testletdestructuring(testdata) testdata = q[let[x << 21, y << 2][y * x]] # noqa: F821 testletdestructuring(testdata) @@ -374,7 +378,7 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # disembodied haskelly let-where (just the content, no macro invocation) - testdata = q[y * x, where[x := 21, y := 2]] # noqa: F821 + testdata = q[y * x, where[(x := 21), (y := 2)]] # noqa: F821 testletdestructuring(testdata) testdata = q[y * x, where[x << 21, y << 2]] # noqa: F821 testletdestructuring(testdata) @@ -599,7 +603,7 @@ def f8(): # Destructuring - unexpanded do with testset("do destructuring (unexpanded) (new env-assign syntax v0.15.3+)"): - testdata = q[do[local[x := 21], # noqa: F821 + testdata = q[do[local[(x := 21)], # noqa: F821 2 * x]] # noqa: F821 view = UnexpandedDoView(testdata) # read @@ -611,11 +615,11 @@ def f8(): test[isenvassign(the[thing])] # write # This mutates the original, but we have to assign `view.body` to trigger the setter. - thebody[0] = q[local[x := 9001]] # noqa: F821 + thebody[0] = q[local[(x := 9001)]] # noqa: F821 view.body = thebody # implicit do, a.k.a. extra bracket syntax - testdata = q[let[[local[x := 21], # noqa: F821 + testdata = q[let[[local[(x := 21)], # noqa: F821 2 * x]]] # noqa: F821 if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. theimplicitdo = testdata.slice @@ -630,7 +634,7 @@ def f8(): thing = thebody[0].slice.value test[isenvassign(the[thing])] # write - thebody[0] = q[local[x := 9001]] # noqa: F821 + thebody[0] = q[local[(x := 9001)]] # noqa: F821 view.body = thebody test_raises[TypeError, From df8bf45b22fd73ba286a2bec1af22bf4098c2f88 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:15:56 +0300 Subject: [PATCH 720/832] fix Python 3.8 and 3.9 compatibility, second attempt Try, try, try again until it works; Merrily, merrily, merrily, until it works. --- unpythonic/syntax/tests/test_letdo.py | 2 +- unpythonic/syntax/tests/test_letdoutil.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 3d94167c..215257b4 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -33,7 +33,7 @@ def runtests(): # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). d1 = do[local[(x := 17)], print(x), - x := 23, + (x := 23), x] test[d1 == 23] diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index c0d82321..9b59a78d 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -346,7 +346,7 @@ def testletdestructuring(testdata): testletdestructuring(testdata) # haskelly let-where - testdata = q[let[y * x, where[x := 21, y := 2]]] # noqa: F821 + testdata = q[let[y * x, where[(x := 21), (y := 2)]]] # noqa: F821 testletdestructuring(testdata) testdata = q[let[y * x, where[x << 21, y << 2]]] # noqa: F821 testletdestructuring(testdata) From d1c8d33b9e649801f04f3f082f9f26d77ffadc7b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:19:02 +0300 Subject: [PATCH 721/832] fix Python 3.8 and 3.9 compatibility, third attempt Try, try, try again until it works; Merrily, merrily, merrily, until it works. --- unpythonic/syntax/tests/test_letdo.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 215257b4..93ce7ce2 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -491,7 +491,7 @@ def test14(): x = "the nonlocal x" # restore the test environment # v0.15.3+: walrus syntax - @dlet[(x := "the env x")] + @dlet(x := "the env x") def test15(): def inner(): (x := "updated env x") # noqa: F841, this writes to the let env since there is no `x` in an intervening scope, according to Python's standard rules. @@ -499,7 +499,7 @@ def inner(): return x test[test15() == "updated env x"] - @dlet[(x := "the env x")] + @dlet(x := "the env x") def test16(): def inner(): x = "the inner x" # noqa: F841, unused on purpose, for testing. An assignment *statement* does NOT write to the let env. @@ -507,7 +507,7 @@ def inner(): return x test[test16() == "the env x"] - @dlet[(x := "the env x")] + @dlet(x := "the env x") def test17(): x = "the local x" # This lexical variable shadows the env x. def inner(): From 913975906131feb15db1d844b89058fb1f331a82 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:29:53 +0300 Subject: [PATCH 722/832] add note that Python got native pattern matching in 3.10 --- doc/readings.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/readings.md b/doc/readings.md index 75ec095d..68eaa106 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -245,6 +245,7 @@ Python clearly wants to be an impure-FP language. A decorator with arguments *is - [pyrsistent: Persistent/Immutable/Functional data structures for Python](https://github.com/tobgu/pyrsistent) - [pampy: Pattern matching for Python](https://github.com/santinic/pampy) (pure Python, no AST transforms!) + - Note that Python got [native support for pattern matching in 3.10](https://docs.python.org/3/whatsnew/3.10.html#pep-634-structural-pattern-matching) using the `match`/`case` statement. - [List of languages that compile to Python](https://github.com/vindarel/languages-that-compile-to-python) including Hy, a Lisp (in the [Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2) family) that can use Python libraries. From b46d0917de3872446835df07f30dcda742a68e29 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:30:10 +0300 Subject: [PATCH 723/832] add note on 3.8 and 3.9 compatibility when using := syntax --- doc/macros.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/doc/macros.md b/doc/macros.md index cd8493b8..8fc70043 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -134,7 +134,17 @@ Bindings are established using standard assignment expression syntax, `name := v The old `unpythonic` env-assignment syntax, `name << value`, is also supported for backward compatibility. This was the preferred syntax in v0.15.0 to v0.15.2. -**CAUTION**: All let-bindings must be established in the bindings subform. If you absolutely need to do establish more bindings in the body, see the sequencing construct `do[]` and its syntax `local[x := 42]`. +**NOTE**: All let-bindings must be established in the bindings subform. If you absolutely need to do establish more bindings in the body, see the sequencing construct `do[]` and its syntax `local[x := 42]`. + +**NOTE**: Language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). The syntax accepted when running on Python 3.8 or 3.9 is: + +```python +let[(x := 17), + (y := 23)][ + print(x, y)] +``` + +That is, Python 3.8 and 3.9 require parentheses around each let binding if you use the new `:=` syntax, because syntactically, the bindings subform looks like a subscript. The unit tests use this syntax so that they work on 3.8 and 3.9. But for new code using Python 3.10 or later, it is preferable to omit the parentheses to improve readability. The same syntax for the bindings subform is used by: From 82df7750a782a72a1c47dd6971a40e8aac888171 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:37:55 +0300 Subject: [PATCH 724/832] Changelog: codebase cleanup done --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 735625da..70452b7e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ **IMPORTANT**: - Minimum Python language version is now 3.8. -- Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. Code has not been fully cleaned of historical cruft yet, so parts of it may still work in these versions. +- Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. - 3.8 becomes EOL after October 2024, so support for that version might be dropped soon, too. **Future plans**: From 47d6cc628d94208bbe37a64dcec785160e93fa98 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 14:58:03 +0300 Subject: [PATCH 725/832] update README examples to use walrus operator --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 8345c2b4..7a5a000a 100644 --- a/README.md +++ b/README.md @@ -594,13 +594,13 @@ As usual in test frameworks, the testing constructs behave somewhat like `assert ```python from unpythonic.syntax import macros, let, letseq, letrec -x = let[[a << 1, b << 2] in a + b] -y = letseq[[c << 1, # LET SEQuential, like Scheme's let* - c << 2 * c, - c << 2 * c] in +x = let[[a := 1, b := 2] in a + b] +y = letseq[[c := 1, # LET SEQuential, like Scheme's let* + c := 2 * c, + c := 2 * c] in c] -z = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECursive, like in Scheme - oddp << (lambda x: (x != 0) and evenp(x - 1))] +z = letrec[[evenp := (lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECursive, like in Scheme + oddp := (lambda x: (x != 0) and evenp(x - 1))] in evenp(42)] ```
@@ -611,10 +611,10 @@ z = letrec[[evenp << (lambda x: (x == 0) or oddp(x - 1)), # LET mutually RECurs ```python from unpythonic.syntax import macros, dlet -# Up to Python 3.8, use `@dlet(x << 0)` instead -@dlet[x << 0] # let-over-lambda for Python +# In Python 3.8, use `@dlet(x << 0)` instead; in Python 3.9, use `@dlet(x := 0)` +@dlet[x := 0] # let-over-lambda for Python def count(): - return x << x + 1 # `name << value` rebinds in the let env + return x := x + 1 # `name := value` rebinds in the let env assert count() == 1 assert count() == 2 ``` @@ -626,8 +626,8 @@ assert count() == 2 ```python from unpythonic.syntax import macros, do, local, delete -x = do[local[a << 21], - local[b << 2 * a], +x = do[local[a := 21], + local[b := 2 * a], print(b), delete[b], # do[] local variables can be deleted, too 4 * a] @@ -751,8 +751,8 @@ assert square.__name__ == "square" # - brackets denote a multiple-expression lambda body # (if you want to have one expression that is a literal list, # double the brackets: `lambda x: [[5 * x]]`) -# - local[name << value] makes an expression-local variable -g = lambda x: [local[y << 2 * x], +# - local[name := value] makes an expression-local variable +g = lambda x: [local[y := 2 * x], y + 1] assert g(10) == 21 ``` From eefbad52c9bdaf7148c0e4c3d0b156785a46ebfe Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 15:00:39 +0300 Subject: [PATCH 726/832] update note of supported Python versions --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7a5a000a..702ac4ae 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ None required. - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -The 0.15.x series should run on CPython 3.8, 3.9 and 3.10, and PyPy3 (language version 3.8); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +As of v0.15.3, `unpythonic` runs on CPython 3.8, 3.9 and 3.10, and PyPy3 (language versions 3.8, 3.9, 3.10); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. New Python versions are added and old ones are removed following the [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation From b3bed10318a810d4e508411e6c03165f0562a214 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 15:12:51 +0300 Subject: [PATCH 727/832] update CHANGELOG --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 70452b7e..7a6b3887 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,10 +3,14 @@ **New**: - Walrus syntax `name := value` is now supported, and preferred, for all env-assignments. Old syntax `name << value` still works, and will remain working at least until v0.16.0, whenever that is. + - Note that language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). + That is, if you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. + - All documentation is written in Python 3.10 syntax; all unit tests are written in Python 3.8 syntax. **Fixed**: - `ETAEstimator` edge case: at any point after all tasks have been marked completed, return a constant zero estimate for the remaining time. +- Fix borkage in `mathseq` when running with SymPy 1.13 (SymPy is only used in tests). Bump SymPy version to 1.13. **IMPORTANT**: From 809dfa7b26ceb8dfed06611e8a3d08f1e701d71d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 15:34:22 +0300 Subject: [PATCH 728/832] CPython 3.11 seems to have fixed some uninspectable builtins. Made that test conditional on running on Python older than 3.11. --- unpythonic/tests/test_fun.py | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/unpythonic/tests/test_fun.py b/unpythonic/tests/test_fun.py index 3b8f38db..cfcd7551 100644 --- a/unpythonic/tests/test_fun.py +++ b/unpythonic/tests/test_fun.py @@ -254,16 +254,17 @@ def double(x): with dyn.let(curry_context=["whatever"]): return the[curry(double, 2, nosucharg="foo")] == Values(4, nosucharg="foo") - # This doesn't occur on PyPy3. + # This doesn't occur on PyPy3, or on CPython 3.11+. if sys.implementation.name == "cpython": # pragma: no cover - with testset("uninspectable builtin functions"): - test_raises[ValueError, curry(print)] # builtin function that fails `inspect.signature` - - # Internal feature, used by curry macro. If uninspectables are said to be ok, - # then attempting to curry an uninspectable simply returns the original function. - m1 = print - m2 = curry(print, _curry_allow_uninspectable=True) - test[the[m2] is the[m1]] + if sys.version_info < (3, 11, 0): + with testset("uninspectable builtin functions"): + test_raises[ValueError, curry(print)] # builtin function that fails `inspect.signature` + + # Internal feature, used by curry macro. If uninspectables are said to be ok, + # then attempting to curry an uninspectable simply returns the original function. + m1 = print + m2 = curry(print, _curry_allow_uninspectable=True) + test[the[m2] is the[m1]] with testset("curry kwargs support"): @curry From 011f3be85f37453938ee94326b767e856433ee2b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 15:42:05 +0300 Subject: [PATCH 729/832] update supported Python versions --- CHANGELOG.md | 23 ++++++++++++----------- README.md | 2 +- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a6b3887..2aa869f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,27 +1,28 @@ **0.15.3** (in progress, last updated 25 September 2024) +**IMPORTANT**: + +- Minimum Python language version is now 3.8. + - We support 3.8, 3.9, 3.10, 3.11, 3.12, and PyPy3 (language versions 3.8, 3.9, and 3.10). + - Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. +- Note this release of `unpythonic` is still in progress. Python 3.8 becomes EOL after October 2024, so support for 3.8 might be dropped before `unpythonic` 0.15.3 is released. + + **New**: +- **Python 3.12 support**. +- **Python 3.11 support**. - Walrus syntax `name := value` is now supported, and preferred, for all env-assignments. Old syntax `name << value` still works, and will remain working at least until v0.16.0, whenever that is. - Note that language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). - That is, if you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. + - If you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. - All documentation is written in Python 3.10 syntax; all unit tests are written in Python 3.8 syntax. + **Fixed**: - `ETAEstimator` edge case: at any point after all tasks have been marked completed, return a constant zero estimate for the remaining time. - Fix borkage in `mathseq` when running with SymPy 1.13 (SymPy is only used in tests). Bump SymPy version to 1.13. -**IMPORTANT**: - -- Minimum Python language version is now 3.8. -- Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. -- 3.8 becomes EOL after October 2024, so support for that version might be dropped soon, too. - -**Future plans**: - -Near-term focus will likely be on introducing support for Python 3.11 and 3.12, with no major changes to functionality. No promises though (except of the `lazy[]`/`force()` kind, which see). - --- diff --git a/README.md b/README.md index 702ac4ae..00dce689 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ None required. - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -As of v0.15.3, `unpythonic` runs on CPython 3.8, 3.9 and 3.10, and PyPy3 (language versions 3.8, 3.9, 3.10); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. New Python versions are added and old ones are removed following the [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +As of v0.15.3, `unpythonic` runs on CPython 3.8, 3.9 and 3.10, 3.11, 3.12, and PyPy3 (language versions 3.8, 3.9, 3.10); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. New Python versions are added and old ones are removed following the [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation From fb91d8ea0440dff2e02e2c041169560e558fd97a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 15:42:52 +0300 Subject: [PATCH 730/832] CI: add Python 3.11 and 3.12 --- .github/workflows/python-package.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 14ab32fb..60b8eee8 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.8", "3.9", "3.10", pypy-3.8, pypy-3.9, pypy-3.10] + python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", pypy-3.8, pypy-3.9, pypy-3.10] steps: - uses: actions/checkout@v2 From 4a2743d4fdbad430309e307cc8380ecab870e65d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 16:24:31 +0300 Subject: [PATCH 731/832] metadata: Python 3.11 and 3.12 now supported --- setup.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 8ff52f82..00617b96 100644 --- a/setup.py +++ b/setup.py @@ -74,7 +74,7 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", "dynamic-variable", "macros", "lisp", "scheme", "racket", "haskell"], install_requires=[], # mcpyrate is optional for us, so we can't really put it here even though we recommend it. - python_requires=">=3.8,<3.11", + python_requires=">=3.8,<3.13", author="Juha Jeronen", author_email="juha.m.jeronen@gmail.com", url="https://github.com/Technologicat/unpythonic", @@ -93,6 +93,8 @@ def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packa "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Software Development :: Libraries", From 3c27ab9b868a73df2a49211bb38d63888c5e3ff3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 26 Sep 2024 16:34:57 +0300 Subject: [PATCH 732/832] fix changelog date --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2aa869f7..843dbde8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -**0.15.3** (in progress, last updated 25 September 2024) +**0.15.3** (in progress, last updated 26 September 2024) **IMPORTANT**: From 144611933327ee9692b37de27b984ebab2655cde Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 10:27:56 +0300 Subject: [PATCH 733/832] update local install instructions (https://github.com/pypa/pip/issues/12330) --- README.md | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 00dce689..5d6b37a6 100644 --- a/README.md +++ b/README.md @@ -808,31 +808,21 @@ assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) **PyPI** -``pip3 install unpythonic --user`` - -or - -``sudo pip3 install unpythonic`` +``pip install unpythonic`` **GitHub** -Clone (or pull) from GitHub. Then, - -``python3 setup.py install --user`` - -or - -``sudo python3 setup.py install`` - -**Uninstall** +Clone the repo from GitHub. Then, navigate to it in a terminal, and: -Uninstallation must be invoked in a folder which has no subfolder called ``unpythonic``, so that ``pip`` recognizes it as a package name (instead of a filename). Then, - -``pip3 uninstall unpythonic`` +```bash +pip install . +``` -or +To uninstall: -``sudo pip3 uninstall unpythonic`` +```bash +pip uninstall unpythonic +``` ## Support From 64e9554d121626111d062a3c13af3e49530d4e93 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:05:57 +0300 Subject: [PATCH 734/832] let's remove Python 3.8 support later, probably in 0.15.4 --- CHANGELOG.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 843dbde8..f8ff8e3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,10 @@ -**0.15.3** (in progress, last updated 26 September 2024) +**0.15.3** (in progress, last updated 27 September 2024) **IMPORTANT**: - Minimum Python language version is now 3.8. - We support 3.8, 3.9, 3.10, 3.11, 3.12, and PyPy3 (language versions 3.8, 3.9, and 3.10). - Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. -- Note this release of `unpythonic` is still in progress. Python 3.8 becomes EOL after October 2024, so support for 3.8 might be dropped before `unpythonic` 0.15.3 is released. **New**: From dfa908d1af4d78686173a07e8edb5a37764d761f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:08:08 +0300 Subject: [PATCH 735/832] update CHANGELOG --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f8ff8e3a..ac5eca5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ - Minimum Python language version is now 3.8. - We support 3.8, 3.9, 3.10, 3.11, 3.12, and PyPy3 (language versions 3.8, 3.9, and 3.10). - - Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. + - Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. If you need `unpythonic` for Python 3.6 or 3.7, use version 0.15.2. **New**: From 0a7ca96a8d71a4a89ea3f89ab5779c22efe65006 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:11:25 +0300 Subject: [PATCH 736/832] update CHANGELOG --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac5eca5a..2116df21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,9 @@ **New**: - **Python 3.12 support**. + - As in, all tests pass, so there are no regressions. Some undiscovered interactions with new language features (`type` statement) may still be broken. - **Python 3.11 support**. + - As in, all tests pass, so there are no regressions. Some undiscovered interactions with new language features (`try`/`except*` construct) may still be broken. - Walrus syntax `name := value` is now supported, and preferred, for all env-assignments. Old syntax `name << value` still works, and will remain working at least until v0.16.0, whenever that is. - Note that language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). - If you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. From 4084a29c1785764d124f6fee9c734c7937099545 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:31:12 +0300 Subject: [PATCH 737/832] astcompat: support up to Python 3.12 --- unpythonic/syntax/astcompat.py | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/astcompat.py b/unpythonic/syntax/astcompat.py index 68fa5a0b..50be21b8 100644 --- a/unpythonic/syntax/astcompat.py +++ b/unpythonic/syntax/astcompat.py @@ -2,6 +2,9 @@ """Conditionally import AST node types only supported by recent enough Python versions.""" __all__ = ["NamedExpr", + "Match", "match_case", "MatchValue", "MatchSingleton", "MatchSequence", "MatchStar", "MatchMapping", "MatchClass", "MatchAs", "MatchOr", + "TryStar", + "TypeAlias", "TypeVar", "ParamSpec", "TypeVarTuple", "Num", "Str", "Bytes", "NameConstant", "Ellipsis", "Index", "ExtSlice", "getconstant"] @@ -25,9 +28,23 @@ NamedExpr = _NoSuchNodeType # No new AST node types in Python 3.9. -# No new AST node types in Python 3.10. -# TODO: Any new AST node types in Python 3.11? -# TODO: Any new AST node types in Python 3.12? + +try: # Python 3.10+ + from ast import (Match, match_case, + MatchValue, MatchSingleton, MatchSequence, MatchStar, + MatchMapping, MatchClass, MatchAs, MatchOr) # `match`/`case` +except ImportError: # pragma: no cover + Match = match_case = MatchValue = MatchSingleton = MatchSequence = MatchStar = MatchMapping = MatchClass = MatchAs = MatchOr = _NoSuchNodeType + +try: # Python 3.11+ + from ast import TryStar # `try`/`except*` (exception groups) +except ImportError: # pragma: no cover + TryStar = _NoSuchNodeType + +try: # Python 3.12+ + from ast import TypeAlias, TypeVar, ParamSpec, TypeVarTuple # `type` statement (type alias) +except ImportError: # pragma: no cover + TypeAlias = TypeVar = ParamSpec = TypeVarTuple = _NoSuchNodeType # -------------------------------------------------------------------------------- # Deprecated AST node types From 58388fc0f3d7950dad5eac9f4ff9643e992c9082 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:31:30 +0300 Subject: [PATCH 738/832] lazify, autocurry: leave `type` statements alone --- unpythonic/syntax/autocurry.py | 5 +++++ unpythonic/syntax/lazify.py | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index ea8c398c..18c83f91 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -10,6 +10,7 @@ from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer +from .astcompat import TypeAlias from .util import (suggest_decorator_index, isx, has_curry, sort_lambda_decorators) from ..dynassign import dyn @@ -85,6 +86,10 @@ def transform(self, tree): if is_captured_value(tree): return tree + # Python 3.12+: leave `type` statements alone (autocurrying a type declaration makes no sense) + if type(tree) is TypeAlias: + return tree + hascurry = self.state.hascurry if type(tree) is Call: # Don't auto-curry some calls we know not to need it. This is both a performance optimization diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index 0115805c..c63a78e4 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -14,6 +14,7 @@ from mcpyrate.unparser import unparse from mcpyrate.walkers import ASTTransformer +from .astcompat import TypeAlias from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, isx, getname, is_decorator) from .letdoutil import islet, isdo, ExpandedLetView @@ -648,6 +649,10 @@ def f(tree): # else forcing_mode == "off" return tree + # Python 3.12+: leave `type` statements alone (lazifying a type declaration makes no sense) + elif type(tree) is TypeAlias: + return tree + elif type(tree) in (FunctionDef, AsyncFunctionDef, Lambda): if type(tree) is Lambda and id(tree) not in userlambdas: return self.generic_visit(tree) # ignore macro-introduced lambdas (but recurse inside them) From 040e333aa4036062636ade58f3d29fa1708d91e1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:34:38 +0300 Subject: [PATCH 739/832] syntax: when detecting `try` blocks, handle TryStar as well as Try --- unpythonic/syntax/scopeanalyzer.py | 4 +++- unpythonic/syntax/tailtools.py | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index c39180a6..010db9fd 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -83,6 +83,8 @@ from mcpyrate.core import Done from mcpyrate.walkers import ASTTransformer, ASTVisitor +from .astcompat import TryStar + from ..it import uniqify def isnewscope(tree): @@ -332,7 +334,7 @@ def examine(self, tree): elif type(tree) in (Import, ImportFrom): for x in tree.names: self.collect(x.asname if x.asname is not None else x.name) - elif type(tree) is Try: + elif type(tree) in (Try, TryStar): # https://docs.python.org/3/reference/compound_stmts.html#the-try-statement # # TODO: The `err` in `except SomeException as err` is only bound within the `except` block, diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index cbfbe071..1bc3b4a5 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -26,7 +26,7 @@ from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer, ASTVisitor -from .astcompat import getconstant, NameConstant +from .astcompat import getconstant, NameConstant, TryStar from .ifexprs import aif, it from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView from .util import (isx, isec, @@ -691,7 +691,7 @@ def transform(self, tree): tree.orelse[-1] = self.visit(tree.orelse[-1]) elif type(tree) in (With, AsyncWith): tree.body[-1] = self.visit(tree.body[-1]) - elif type(tree) is Try: + elif type(tree) in (Try, TryStar): # We don't care about finalbody; typically used for unwinding only. if tree.orelse: # tail position is in else clause if present tree.orelse[-1] = self.visit(tree.orelse[-1]) From d8d4a5127fc304ab0caee1a8d8043b2c479494e2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 14:39:16 +0300 Subject: [PATCH 740/832] oops, tag the Python 3.11 changes in comments properly --- unpythonic/syntax/scopeanalyzer.py | 2 +- unpythonic/syntax/tailtools.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 010db9fd..129be3d6 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -334,7 +334,7 @@ def examine(self, tree): elif type(tree) in (Import, ImportFrom): for x in tree.names: self.collect(x.asname if x.asname is not None else x.name) - elif type(tree) in (Try, TryStar): + elif type(tree) in (Try, TryStar): # Python 3.11+: `try`/`except*` # https://docs.python.org/3/reference/compound_stmts.html#the-try-statement # # TODO: The `err` in `except SomeException as err` is only bound within the `except` block, diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 1bc3b4a5..b0b33077 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -691,7 +691,7 @@ def transform(self, tree): tree.orelse[-1] = self.visit(tree.orelse[-1]) elif type(tree) in (With, AsyncWith): tree.body[-1] = self.visit(tree.body[-1]) - elif type(tree) in (Try, TryStar): + elif type(tree) in (Try, TryStar): # Python 3.11+: `try`/`except*` # We don't care about finalbody; typically used for unwinding only. if tree.orelse: # tail position is in else clause if present tree.orelse[-1] = self.visit(tree.orelse[-1]) From 23fc4cf4b4f41325c7ccfad670adbcae7539086e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:06:11 +0300 Subject: [PATCH 741/832] add comments about missing tests until we bump up minimum Python --- unpythonic/syntax/tests/test_autocurry.py | 2 ++ unpythonic/syntax/tests/test_lazify.py | 2 ++ unpythonic/syntax/tests/test_scopeanalyzer.py | 3 +++ 3 files changed, 7 insertions(+) diff --git a/unpythonic/syntax/tests/test_autocurry.py b/unpythonic/syntax/tests/test_autocurry.py index 325736bd..f5177bed 100644 --- a/unpythonic/syntax/tests/test_autocurry.py +++ b/unpythonic/syntax/tests/test_autocurry.py @@ -11,6 +11,8 @@ from ...llist import cons, nil, ll from ...collections import frozendict +# TODO: Add test that `autocurry` leaves `type` statements alone once we bump minimum language version to Python 3.12. + def runtests(): with testset("basic usage"): with autocurry: diff --git a/unpythonic/syntax/tests/test_lazify.py b/unpythonic/syntax/tests/test_lazify.py index 1aafaf02..42d2bb56 100644 --- a/unpythonic/syntax/tests/test_lazify.py +++ b/unpythonic/syntax/tests/test_lazify.py @@ -28,6 +28,8 @@ from sys import stderr import gc +# TODO: Add test that `lazify` leaves `type` statements alone once we bump minimum language version to Python 3.12. + def runtests(): # first test the low-level tools with testset("lazyrec (lazify a container literal, recursing into sub-containers)"): diff --git a/unpythonic/syntax/tests/test_scopeanalyzer.py b/unpythonic/syntax/tests/test_scopeanalyzer.py index 0261b4a8..1b4d34d6 100644 --- a/unpythonic/syntax/tests/test_scopeanalyzer.py +++ b/unpythonic/syntax/tests/test_scopeanalyzer.py @@ -14,6 +14,9 @@ get_lexical_variables, scoped_transform) +# TODO: Add tests for `match`/`case` once we bump minimum language version to Python 3.10. +# TODO: Add tests for `try`/`except*` once we bump minimum language version to Python 3.11. + def runtests(): # test data with q as getnames_load: From aa9fccf7818170050525699d445e0b7b9b1b7a2b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:06:41 +0300 Subject: [PATCH 742/832] add comment --- unpythonic/syntax/scopeanalyzer.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 129be3d6..1f92845c 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -346,6 +346,8 @@ def examine(self, tree): # TODO: `try`, even inside the `except` blocks, will be bound in the whole parent scope. for h in tree.handlers: self.collect(h.name) + # Python 3.12+: `TypeAlias` uses a name in `Store` context on its LHS so it needs no special handling here. + # Same note as for for loops. # elif type(tree) in (With, AsyncWith): # for item in tree.items: From faab5167d53d0ce08b5770105bcc2278f81220a0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:39:26 +0300 Subject: [PATCH 743/832] Python 3.10+: handle `match`/`case` captures in scopeanalyzer --- unpythonic/syntax/scopeanalyzer.py | 31 +++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 1f92845c..ba8b5ddf 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -83,7 +83,7 @@ from mcpyrate.core import Done from mcpyrate.walkers import ASTTransformer, ASTVisitor -from .astcompat import TryStar +from .astcompat import TryStar, MatchStar, MatchMapping, MatchClass, MatchAs from ..it import uniqify @@ -313,6 +313,12 @@ def get_names_in_store_context(tree): by ``get_lexical_variables`` for the nearest lexically surrounding parent tree that represents a scope. """ + class MatchCapturesCollector(ASTVisitor): # Python 3.10+: `match`/`case` + def examine(self, tree): + if type(tree) is Name: + self.collect(tree.id) + self.generic_visit(tree) + class StoreNamesCollector(ASTVisitor): # def _collect_name_or_list(self, t): # if type(t) is Name: @@ -346,6 +352,29 @@ def examine(self, tree): # TODO: `try`, even inside the `except` blocks, will be bound in the whole parent scope. for h in tree.handlers: self.collect(h.name) + # Python 3.10+: `match`/`case` uses names in `Load` context to denote captures. + # Also there are some bare strings, and sometimes `None` actually means "_" (but doesn't capture). + # So we special-case all of this. + elif type(tree) in (MatchAs, MatchStar): # a `MatchSequence` also consists of these + if tree.name is not None: + self.collect(tree.name) + elif type(tree) is MatchMapping: + mcc = MatchCapturesCollector(tree.patterns) + mcc.visit() + for name in mcc.collected: + self.collect(name) + if tree.rest is not None: # `rest` is a capture if present + self.collect(tree.rest) + elif type(tree) is MatchClass: + mcc = MatchCapturesCollector(tree.patterns) + mcc.visit() + for name in mcc.collected: + self.collect(name) + mcc = MatchCapturesCollector(tree.kwd_patterns) + mcc.visit() + for name in mcc.collected: + self.collect(name) + # Python 3.12+: `TypeAlias` uses a name in `Store` context on its LHS so it needs no special handling here. # Same note as for for loops. From d0162c69199145a535869e2fdaddc0352081af45 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:50:38 +0300 Subject: [PATCH 744/832] move `unpythonic.syntax.astcompat` to `mcpyrate.astcompat` --- unpythonic/syntax/astcompat.py | 87 ----------------------- unpythonic/syntax/autocurry.py | 2 +- unpythonic/syntax/autoref.py | 2 +- unpythonic/syntax/lambdatools.py | 2 +- unpythonic/syntax/lazify.py | 2 +- unpythonic/syntax/letdoutil.py | 2 +- unpythonic/syntax/scopeanalyzer.py | 3 +- unpythonic/syntax/tailtools.py | 2 +- unpythonic/syntax/tests/test_letdoutil.py | 2 +- unpythonic/syntax/tests/test_util.py | 2 +- unpythonic/syntax/util.py | 2 +- 11 files changed, 10 insertions(+), 98 deletions(-) delete mode 100644 unpythonic/syntax/astcompat.py diff --git a/unpythonic/syntax/astcompat.py b/unpythonic/syntax/astcompat.py deleted file mode 100644 index 50be21b8..00000000 --- a/unpythonic/syntax/astcompat.py +++ /dev/null @@ -1,87 +0,0 @@ -# -*- coding: utf-8 -*- -"""Conditionally import AST node types only supported by recent enough Python versions.""" - -__all__ = ["NamedExpr", - "Match", "match_case", "MatchValue", "MatchSingleton", "MatchSequence", "MatchStar", "MatchMapping", "MatchClass", "MatchAs", "MatchOr", - "TryStar", - "TypeAlias", "TypeVar", "ParamSpec", "TypeVarTuple", - "Num", "Str", "Bytes", "NameConstant", "Ellipsis", - "Index", "ExtSlice", - "getconstant"] - -import ast - -from ..symbol import gensym - -_NoSuchNodeType = gensym("_NoSuchNodeType") - -# -------------------------------------------------------------------------------- -# New AST node types - -# Minimum language version supported by this module is Python 3.6. - -# No new AST node types in Python 3.7. - -try: # Python 3.8+ - from ast import NamedExpr # a.k.a. walrus operator ":=" -except ImportError: # pragma: no cover - NamedExpr = _NoSuchNodeType - -# No new AST node types in Python 3.9. - -try: # Python 3.10+ - from ast import (Match, match_case, - MatchValue, MatchSingleton, MatchSequence, MatchStar, - MatchMapping, MatchClass, MatchAs, MatchOr) # `match`/`case` -except ImportError: # pragma: no cover - Match = match_case = MatchValue = MatchSingleton = MatchSequence = MatchStar = MatchMapping = MatchClass = MatchAs = MatchOr = _NoSuchNodeType - -try: # Python 3.11+ - from ast import TryStar # `try`/`except*` (exception groups) -except ImportError: # pragma: no cover - TryStar = _NoSuchNodeType - -try: # Python 3.12+ - from ast import TypeAlias, TypeVar, ParamSpec, TypeVarTuple # `type` statement (type alias) -except ImportError: # pragma: no cover - TypeAlias = TypeVar = ParamSpec = TypeVarTuple = _NoSuchNodeType - -# -------------------------------------------------------------------------------- -# Deprecated AST node types - -try: # Python 3.8+, https://docs.python.org/3/whatsnew/3.8.html#deprecated - from ast import Num, Str, Bytes, NameConstant, Ellipsis -except ImportError: # pragma: no cover - Num = Str = Bytes = NameConstant = Ellipsis = _NoSuchNodeType - -try: # Python 3.9+, https://docs.python.org/3/whatsnew/3.9.html#deprecated - from ast import Index, ExtSlice - # We ignore the internal classes Suite, Param, AugLoad, AugStore, - # which were never used in Python 3.x. -except ImportError: # pragma: no cover - Index = ExtSlice = _NoSuchNodeType - -# -------------------------------------------------------------------------------- -# Compatibility functions - -def getconstant(tree): - """Given an AST node `tree` representing a constant, return the contained raw value. - - This encapsulates the AST differences between Python 3.8+ and older versions. - - There are no `setconstant` or `makeconstant` counterparts, because you can - just create an `ast.Constant` in Python 3.6 and later. The parser doesn't - emit them until Python 3.8, but Python 3.6+ compile `ast.Constant` just fine. - """ - if type(tree) is ast.Constant: # Python 3.8+ - return tree.value - # up to Python 3.7 - elif type(tree) is ast.NameConstant: # up to Python 3.7 # pragma: no cover - return tree.value - elif type(tree) is ast.Num: # pragma: no cover - return tree.n - elif type(tree) in (ast.Str, ast.Bytes): # pragma: no cover - return tree.s - elif type(tree) is ast.Ellipsis: # `ast.Ellipsis` is the AST node type, `builtins.Ellipsis` is `...`. # pragma: no cover - return ... - raise TypeError(f"Not an AST node representing a constant: {type(tree)} with value {repr(tree)}") # pragma: no cover diff --git a/unpythonic/syntax/autocurry.py b/unpythonic/syntax/autocurry.py index 18c83f91..0124c3b5 100644 --- a/unpythonic/syntax/autocurry.py +++ b/unpythonic/syntax/autocurry.py @@ -7,10 +7,10 @@ from mcpyrate.quotes import macros, q, a, h # noqa: F401 +from mcpyrate.astcompat import TypeAlias from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer -from .astcompat import TypeAlias from .util import (suggest_decorator_index, isx, has_curry, sort_lambda_decorators) from ..dynassign import dyn diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index e1119730..739b22d1 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -9,11 +9,11 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym, parametricmacro +from mcpyrate.astcompat import getconstant from mcpyrate.astfixers import fix_ctx from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer -from .astcompat import getconstant from .nameutil import isx from .util import ExpandedAutorefMarker from .letdoutil import isdo, islet, ExpandedDoView, ExpandedLetView diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 1b089392..4b12fd21 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -14,6 +14,7 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym +from mcpyrate.astcompat import getconstant, Str, NamedExpr from mcpyrate.expander import MacroExpander from mcpyrate.quotes import is_captured_value from mcpyrate.splicing import splice_expression @@ -25,7 +26,6 @@ from ..misc import namelambda from ..symbol import sym -from .astcompat import getconstant, Str, NamedExpr from .letdo import _implicit_do, _do from .letdoutil import islet, isenvassign, UnexpandedLetView, UnexpandedEnvAssignView, ExpandedDoView from .nameutil import getname diff --git a/unpythonic/syntax/lazify.py b/unpythonic/syntax/lazify.py index c63a78e4..af2207b7 100644 --- a/unpythonic/syntax/lazify.py +++ b/unpythonic/syntax/lazify.py @@ -9,12 +9,12 @@ from mcpyrate.quotes import macros, q, u, a, h # noqa: F401 +from mcpyrate.astcompat import TypeAlias from mcpyrate.astfixers import fix_ctx from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.unparser import unparse from mcpyrate.walkers import ASTTransformer -from .astcompat import TypeAlias from .util import (suggest_decorator_index, sort_lambda_decorators, detect_lambda, isx, getname, is_decorator) from .letdoutil import islet, isdo, ExpandedLetView diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index 0b7e20be..d286d0e7 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -11,9 +11,9 @@ import sys from mcpyrate import unparse +from mcpyrate.astcompat import getconstant, Str, NamedExpr from mcpyrate.core import Done -from .astcompat import getconstant, Str, NamedExpr from .nameutil import isx, getname letf_name = "letter" # must match what ``unpythonic.syntax.letdo._let_expr_impl`` uses in its output. diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index ba8b5ddf..8a2634eb 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -80,11 +80,10 @@ Import, ImportFrom, Try, ListComp, SetComp, GeneratorExp, DictComp, Store, Del, Global, Nonlocal) +from mcpyrate.astcompat import TryStar, MatchStar, MatchMapping, MatchClass, MatchAs from mcpyrate.core import Done from mcpyrate.walkers import ASTTransformer, ASTVisitor -from .astcompat import TryStar, MatchStar, MatchMapping, MatchClass, MatchAs - from ..it import uniqify def isnewscope(tree): diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index b0b33077..cb5f5e41 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -22,11 +22,11 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym +from mcpyrate.astcompat import getconstant, NameConstant, TryStar from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer, ASTVisitor -from .astcompat import getconstant, NameConstant, TryStar from .ifexprs import aif, it from .letdoutil import isdo, islet, ExpandedLetView, ExpandedDoView from .util import (isx, isec, diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index 9b59a78d..c6d2fc18 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -4,6 +4,7 @@ from ...syntax import macros, test, test_raises, warn, the # noqa: F401 from ...test.fixtures import session, testset +from mcpyrate.astcompat import getconstant, Num from mcpyrate.quotes import macros, q, n # noqa: F401, F811 from mcpyrate.metatools import macros, expandrq # noqa: F811 @@ -16,7 +17,6 @@ from mcpyrate import unparse -from ...syntax.astcompat import getconstant, Num from ...syntax.letdoutil import (canonize_bindings, isenvassign, islet, isdo, UnexpandedEnvAssignView, diff --git a/unpythonic/syntax/tests/test_util.py b/unpythonic/syntax/tests/test_util.py index 738f09dc..807c5da7 100644 --- a/unpythonic/syntax/tests/test_util.py +++ b/unpythonic/syntax/tests/test_util.py @@ -4,10 +4,10 @@ from ...syntax import macros, do, local, test, test_raises, fail, the # noqa: F401 from ...test.fixtures import session, testset +from mcpyrate.astcompat import getconstant, Num, Str from mcpyrate.quotes import macros, q, n, h # noqa: F401, F811 from mcpyrate.metatools import macros, expandrq # noqa: F401, F811 -from ...syntax.astcompat import getconstant, Num, Str from ...syntax.util import (isec, detect_callec, detect_lambda, is_decorator, has_tco, has_curry, has_deco, diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index f1afbf14..721b8b30 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -18,12 +18,12 @@ from ast import Call, Lambda, FunctionDef, AsyncFunctionDef, If, stmt +from mcpyrate.astcompat import getconstant from mcpyrate.core import add_postprocessor from mcpyrate.markers import ASTMarker, delete_markers from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer, ASTVisitor -from .astcompat import getconstant from .letdoutil import isdo, ExpandedDoView from .nameutil import isx, getname From 2aee647068902e830ed16d864ca9759c440ae42f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:53:11 +0300 Subject: [PATCH 745/832] update CHANGELOG --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2116df21..a67f13cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ - Note that language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). - If you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. - All documentation is written in Python 3.10 syntax; all unit tests are written in Python 3.8 syntax. +- Internal module `unpythonic.syntax.astcompat`, used by the macro layer, moved to `mcpyrate.astcompat`. This module handles version differences in the `ast` module in various versions of Python. **Fixed**: From 4809a7e2cc106e136605f06715eaeeb6141c63ac Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:55:20 +0300 Subject: [PATCH 746/832] requirements: mcpyrate -> 3.6.2 (astcompat moved there) --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index f9ca83dd..b311889a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -mcpyrate>=3.6.0 +mcpyrate>=3.6.2 sympy>=1.13 From f06c40ed45076f8aada89ca690e8fc5e15772d39 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 15:56:40 +0300 Subject: [PATCH 747/832] update CHANGELOG --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a67f13cd..1de3c53e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ - Minimum Python language version is now 3.8. - We support 3.8, 3.9, 3.10, 3.11, 3.12, and PyPy3 (language versions 3.8, 3.9, and 3.10). - Python 3.6 and 3.7 support dropped, as these language versions have officially reached end-of-life. If you need `unpythonic` for Python 3.6 or 3.7, use version 0.15.2. +- Minimum version for optional macro expander `mcpyrate` is now 3.6.2, because the `astcompat` utility module was moved there. **New**: @@ -17,7 +18,7 @@ - Note that language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). - If you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. - All documentation is written in Python 3.10 syntax; all unit tests are written in Python 3.8 syntax. -- Internal module `unpythonic.syntax.astcompat`, used by the macro layer, moved to `mcpyrate.astcompat`. This module handles version differences in the `ast` module in various versions of Python. +- Utility module `unpythonic.syntax.astcompat`, used by the macro layer, moved to `mcpyrate.astcompat`. This module handles version differences in the `ast` module in various versions of Python. **Fixed**: From 1721270d86769ce8a8db45568f10a33eaab9591f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:29:47 +0300 Subject: [PATCH 748/832] update CHANGELOG --- CHANGELOG.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1de3c53e..4bb4c685 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,13 +11,17 @@ **New**: - **Python 3.12 support**. - - As in, all tests pass, so there are no regressions. Some undiscovered interactions with new language features (`type` statement) may still be broken. + - As in, all tests pass, so there are no regressions. Some undiscovered interactions with new language features (`type` statement) may still be broken, although the most obvious cases are already implemented. - **Python 3.11 support**. - - As in, all tests pass, so there are no regressions. Some undiscovered interactions with new language features (`try`/`except*` construct) may still be broken. + - As in, all tests pass, so there are no regressions. Some undiscovered interactions with new language features (`try`/`except*` construct) may still be broken, although the most obvious cases are already implemented. - Walrus syntax `name := value` is now supported, and preferred, for all env-assignments. Old syntax `name << value` still works, and will remain working at least until v0.16.0, whenever that is. - Note that language support for using an assignment expression inside a subscript *without parenthesizing it* was [added in Python 3.10](https://docs.python.org/3/whatsnew/3.10.html#other-language-changes). - If you still use Python 3.8 or 3.9, with the new `:=` syntax you must put parentheses around each `let` binding, because syntactically, the bindings subform looks like a subscript. - All documentation is written in Python 3.10 syntax; all unit tests are written in Python 3.8 syntax. + + +**Changed**: + - Utility module `unpythonic.syntax.astcompat`, used by the macro layer, moved to `mcpyrate.astcompat`. This module handles version differences in the `ast` module in various versions of Python. From cd8fda1818ebd6c82bd887f3da254ce7158f4bcf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:33:52 +0300 Subject: [PATCH 749/832] 0.15.3 is now complete --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bb4c685..bc2270ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -**0.15.3** (in progress, last updated 27 September 2024) +**0.15.3** (27 September 2024) - *New tree snakes* edition: **IMPORTANT**: From bc5518f01817857efe34f9fc3a7e5e6b20771be1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:37:25 +0300 Subject: [PATCH 750/832] ugh, terminology --- doc/readings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/readings.md b/doc/readings.md index 68eaa106..8ded43a7 100644 --- a/doc/readings.md +++ b/doc/readings.md @@ -245,7 +245,7 @@ Python clearly wants to be an impure-FP language. A decorator with arguments *is - [pyrsistent: Persistent/Immutable/Functional data structures for Python](https://github.com/tobgu/pyrsistent) - [pampy: Pattern matching for Python](https://github.com/santinic/pampy) (pure Python, no AST transforms!) - - Note that Python got [native support for pattern matching in 3.10](https://docs.python.org/3/whatsnew/3.10.html#pep-634-structural-pattern-matching) using the `match`/`case` statement. + - Note that Python got [native support for pattern matching in 3.10](https://docs.python.org/3/whatsnew/3.10.html#pep-634-structural-pattern-matching) using the `match`/`case` construct. - [List of languages that compile to Python](https://github.com/vindarel/languages-that-compile-to-python) including Hy, a Lisp (in the [Lisp-2](https://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2) family) that can use Python libraries. From 1f2fdaa8bba46eaee028017a2ac8dfb6857444c3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:37:33 +0300 Subject: [PATCH 751/832] update CHANGELOG --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc2270ab..cc0027a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,7 @@ - `ETAEstimator` edge case: at any point after all tasks have been marked completed, return a constant zero estimate for the remaining time. - Fix borkage in `mathseq` when running with SymPy 1.13 (SymPy is only used in tests). Bump SymPy version to 1.13. +- Fix bug in scopeanalyzer: `get_names_in_store_context` now collects also names bound in `match`/`case` constructs (pattern matching, Python 3.10). --- From ccd9de5f479ee7d9c612d5d6a6216984f1c105ca Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:41:36 +0300 Subject: [PATCH 752/832] update docstring --- unpythonic/syntax/scopeanalyzer.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 8a2634eb..578a555d 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -290,8 +290,8 @@ def get_names_in_store_context(tree): This includes: - - Any ``Name`` in store context (such as on the LHS of an `Assign` - or `NamedExpr` node) + - Any ``Name`` in store context (such as on the LHS of an `Assign`, + `NamedExpr` (Python 3.8+), `TypeAlias` (Python 3.12+)) - The name of ``FunctionDef``, ``AsyncFunctionDef`` or``ClassDef`` From 91afcc9ebb8d7a826dafdc1f44f934299a1977c4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:41:44 +0300 Subject: [PATCH 753/832] update docstring, again --- unpythonic/syntax/scopeanalyzer.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 578a555d..4ec2fb03 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -303,6 +303,8 @@ def get_names_in_store_context(tree): - The names in the as-part of ``With`` + - The names bound in `match`/`case` patterns (Python 3.10+) + Duplicates may be returned; use ``set(...)`` or ``list(uniqify(...))`` on the output to remove them. From fa7f224b791e8a1b08863408e91af6dde50ab13d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:42:51 +0300 Subject: [PATCH 754/832] update docstring (this time for sure!) --- unpythonic/syntax/scopeanalyzer.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 4ec2fb03..27d7a8ce 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -301,6 +301,8 @@ def get_names_in_store_context(tree): - The exception name of any ``except`` handlers + - The exception name of any ``except*`` handlers (Python 3.11+) + - The names in the as-part of ``With`` - The names bound in `match`/`case` patterns (Python 3.10+) From 9ddb1e2409d35eea43197429348ac2d1ba844c9b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 16:49:59 +0300 Subject: [PATCH 755/832] pre-emptive version bump --- CHANGELOG.md | 9 +++++++++ unpythonic/__init__.py | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cc0027a7..4d04d186 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,12 @@ +# Changelog + +**0.15.4** (in progress, last updated 27 September 2024) + +*No user-visible changes yet.* + + +--- + **0.15.3** (27 September 2024) - *New tree snakes* edition: **IMPORTANT**: diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index ca8974c4..63b4b756 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.15.3' +__version__ = '0.15.4' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From 8d1188b9efada2c067f8ea7995e4f9147a09c6f6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 17:18:13 +0300 Subject: [PATCH 756/832] bump mcpyrate to the latest hotfix version --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index b311889a..f86f8e27 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -mcpyrate>=3.6.2 +mcpyrate>=3.6.3 sympy>=1.13 From e74eef61b75b6a803f0ca99c11b4ec7ec29eb46a Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 17:22:40 +0300 Subject: [PATCH 757/832] hotfix --- CHANGELOG.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d04d186..dc1538a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,11 @@ # Changelog -**0.15.4** (in progress, last updated 27 September 2024) +**0.15.4** (27 September 2024) - hotfix: -*No user-visible changes yet.* +**Fixed** + +- Bump `mcpyrate` to the hotfix version 3.6.3. + - This is only to make sure no one accidentally installs the broken version, `mcpyrate` 3.6.2, which had a bug in interactive console mode that wasn't caught by CI. --- From a1e7238919158b141a8e3ce42438ed8bfebb361b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 17:56:15 +0300 Subject: [PATCH 758/832] update installation instructions --- README.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 5d6b37a6..4ec9b460 100644 --- a/README.md +++ b/README.md @@ -804,21 +804,31 @@ assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ``` -## Installation +## Install & uninstall -**PyPI** +### PyPI -``pip install unpythonic`` +```bash +pip install unpythonic +``` -**GitHub** +### From source Clone the repo from GitHub. Then, navigate to it in a terminal, and: ```bash -pip install . +pip install . --no-compile ``` -To uninstall: +If you intend to use the macro layer of `unpythonic`, the `--no-compile` flag is important. It prevents an **incorrect** precompilation, without macro support, that `pip install` would otherwise do at its `bdist_wheel` step. + +For most Python projects such precompilation is just fine - it's just macro-enabled projects that shouldn't be precompiled with standard tools. + +If `--no-compile` is NOT used, the precompiled bytecode cache may cause errors such as `ImportError: cannot import name 'macros' from 'mcpyrate.quotes'`, when you try to e.g. `from unpythonic.syntax import macros, let`. In-tree, it might work, but against an installed copy, it will fail. It has happened that my CI setup did not detect this kind of failure. + +This is a common issue when using macro expanders in Python. + +### Uninstall ```bash pip uninstall unpythonic From 3da5a6efb050f042911479d1cacb11a07e5a5583 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 27 Sep 2024 17:56:35 +0300 Subject: [PATCH 759/832] wording --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4ec9b460..3d94b8b4 100644 --- a/README.md +++ b/README.md @@ -806,7 +806,7 @@ assert (my_map, double, (q, 1, 2, 3)) == (ll, 2, 4, 6) ## Install & uninstall -### PyPI +### From PyPI ```bash pip install unpythonic From 3994e16a3f4110a98e248120f7c7599715ef999f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 09:52:52 +0300 Subject: [PATCH 760/832] upgrade build system to pdm --- CHANGELOG.md | 11 +++++ makedist.sh | 2 +- pyproject.toml | 76 ++++++++++++++++++++++++++++++ requirements.txt | 2 - setup.py | 104 ----------------------------------------- unpythonic/__init__.py | 2 +- 6 files changed, 89 insertions(+), 108 deletions(-) create mode 100644 pyproject.toml delete mode 100644 requirements.txt delete mode 100644 setup.py diff --git a/CHANGELOG.md b/CHANGELOG.md index dc1538a6..2e7d84b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +**0.15.5** (16 April 2025) - hotfix: + +**Changed**: + +- Internal: Upgrade build system to `pdm`. This is important for the road ahead, since the old `setuptools` build system has been deprecated. +- Bump `mcpyrate` to the hotfix version 3.6.4. + - The only difference is (beside `mcpyrate` too internally upgrading its build system to `pdm`) that the text colorizer now works correctly also for `input` with `readline`. + + +--- + **0.15.4** (27 September 2024) - hotfix: **Fixed** diff --git a/makedist.sh b/makedist.sh index 338298d3..b6c03991 100755 --- a/makedist.sh +++ b/makedist.sh @@ -1,2 +1,2 @@ #!/bin/bash -python3 setup.py sdist bdist_wheel +pdm build diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 00000000..21abd190 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,76 @@ +[project] +name = "unpythonic" +description = "Supercharge your Python with parts of Lisp and Haskell." +authors = [ + { name = "Juha Jeronen", email = "juha.m.jeronen@gmail.com" }, +] +requires-python = ">=3.8,<3.13" + +# the `read` function and long_description_content_type from setup.py are no longer needed, +# modern build tools like pdm/hatch already know how to handle markdown if you point them at a .md file +# they will set the long_description and long_description_content_type for you +readme = "README.md" + +license = { text = "BSD" } + +# This tells whichever build backend you use (pdm in our case) to run its own mechanism to find the version +# of the project and plug it into the metadata +# details for how we instruct pdm to find the version are in the `[tool.pdm.version]` section below +dynamic = ["version"] + +dependencies = [ + "mcpyrate>=3.6.4", + "sympy>=1.13" +] +keywords=["functional-programming", "language-extension", "syntactic-macros", + "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", + "dynamic-variable", "macros", "lisp", "scheme", "racket", "haskell"] +classifiers = [ + "Development Status :: 4 - Beta", + "Environment :: Console", + "Intended Audience :: Developers", + "License :: OSI Approved :: BSD License", + "Operating System :: POSIX :: Linux", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: Implementation :: CPython", + "Programming Language :: Python :: Implementation :: PyPy", + "Topic :: Software Development :: Libraries", + "Topic :: Software Development :: Libraries :: Python Modules" +] + +[project.urls] +Repository = "https://github.com/Technologicat/unpythonic" + +[build-system] +requires = ["pdm-backend"] +build-backend = "pdm.backend" + +[tool.pdm.version] +# the `file` source tells pdm to look for a line in a file that matches the regex `__version__ = ".*"` +# The regex parse is fairly robust, it can handle arbitray whitespace and comments +source = "file" +path = "unpythonic/__init__.py" + +[tool.pdm.build] +# we don't need to explicitly inclue `mcpyrate.repl`. Unlink with setuptools, pdm automatically includes +# all packages and modules in the source tree pointed to by `includes`, minus any paths matching `excludes` +includes = ["unpythonic"] +excludes = ["**/tests", "**/__pycache__"] + +# note the exclusion of an equivalent to zip_safe. I used to think that zip_safe was a core python metadata flag +# telling pip and other python tools not to include the package in any kind of zip-import or zipapp file. +# I was wrong. zip_safe is a setuptools-specific flag that tells setuptools to not include the package in a bdist_egg +# Since bdist_eggs are no longer really used by anything and have been completely supplanted by wheels, zip_safe has no meaningful effect. +# The effect i think you hoped to achieve with zip_safe is achieved by excluding `__pycache__` folders from +# the built wheels, using the `excludes` field in the `[tool.pdm.build]` section above. + +# most python tools at this point, including mypy, have support for sourcing configuration from pyproject.toml +# making the setup.cfg file unnecessary +[tool.mypy] +show_error_codes = true diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index f86f8e27..00000000 --- a/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -mcpyrate>=3.6.3 -sympy>=1.13 diff --git a/setup.py b/setup.py deleted file mode 100644 index 00617b96..00000000 --- a/setup.py +++ /dev/null @@ -1,104 +0,0 @@ -# -*- coding: utf-8 -*- -# -"""setuptools-based setup.py for unpythonic. - -Tested on Python 3.8. - -Usage as usual with setuptools: - python3 setup.py build - python3 setup.py sdist - python3 setup.py bdist_wheel --universal - python3 setup.py install - -For details, see - http://setuptools.readthedocs.io/en/latest/setuptools.html#command-reference -or - python3 setup.py --help - python3 setup.py --help-commands - python3 setup.py --help bdist_wheel # or any command -""" - -import ast -import os - -from setuptools import setup # type: ignore[import] - - -def read(*relpath, **kwargs): # https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-setup-script - with open(os.path.join(os.path.dirname(__file__), *relpath), - encoding=kwargs.get('encoding', 'utf8')) as fh: - return fh.read() - -# Extract __version__ from the package __init__.py -# (since it's not a good idea to actually run __init__.py during the build process). -# -# http://stackoverflow.com/questions/2058802/how-can-i-get-the-version-defined-in-setup-py-setuptools-in-my-package -# -init_py_path = os.path.join("unpythonic", "__init__.py") -version = None -try: - with open(init_py_path) as f: - for line in f: - if line.startswith("__version__"): - module = ast.parse(line, filename=init_py_path) - expr = module.body[0] - assert isinstance(expr, ast.Assign) - v = expr.value - if type(v) is ast.Constant: # Python 3.8+ - # mypy understands `isinstance(..., ...)` but not `type(...) is ...`, - # and we want to match on the exact type, not any subclass that might be - # added in some future Python version. - assert isinstance(v, ast.Constant) - version = v.value - elif type(v) is ast.Str: - assert isinstance(v, ast.Str) # mypy - version = v.s - break -except FileNotFoundError: - pass -if not version: - raise RuntimeError(f"Version information not found in {init_py_path}") - -######################################################### -# Call setup() -######################################################### - -setup( - name="unpythonic", - version=version, - # `unpythonic.test` is the macro-enabled testing framework, intended for public consumption; - # the unit tests of `unpythonic` itself in `unpythonic.tests` are NOT deployed. - packages=["unpythonic", "unpythonic.syntax", "unpythonic.test", "unpythonic.net"], - provides=["unpythonic"], - keywords=["functional-programming", "language-extension", "syntactic-macros", - "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", - "dynamic-variable", "macros", "lisp", "scheme", "racket", "haskell"], - install_requires=[], # mcpyrate is optional for us, so we can't really put it here even though we recommend it. - python_requires=">=3.8,<3.13", - author="Juha Jeronen", - author_email="juha.m.jeronen@gmail.com", - url="https://github.com/Technologicat/unpythonic", - description="Supercharge your Python with parts of Lisp and Haskell.", - long_description=read("README.md"), - long_description_content_type="text/markdown", - license="BSD", - platforms=["Linux"], - classifiers=["Development Status :: 4 - Beta", - "Environment :: Console", - "Intended Audience :: Developers", - "License :: OSI Approved :: BSD License", - "Operating System :: POSIX :: Linux", - "Programming Language :: Python", - "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.8", - "Programming Language :: Python :: 3.9", - "Programming Language :: Python :: 3.10", - "Programming Language :: Python :: 3.11", - "Programming Language :: Python :: 3.12", - "Programming Language :: Python :: Implementation :: CPython", - "Programming Language :: Python :: Implementation :: PyPy", - "Topic :: Software Development :: Libraries", - "Topic :: Software Development :: Libraries :: Python Modules" - ], - zip_safe=False # macros are not zip safe, because the zip importer fails to find sources. -) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 63b4b756..9b5f4dc2 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.15.4' +__version__ = '0.15.5' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From b47afc4d2788bd2a3a040eaf41708816ac7da8c9 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 09:58:04 +0300 Subject: [PATCH 761/832] fix regressions in unit tests in Python 3.12+ --- unpythonic/syntax/tests/test_letdo.py | 4 ++-- unpythonic/syntax/tests/test_scopeanalyzer.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 93ce7ce2..97ebf0ca 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -385,13 +385,13 @@ def test3(): @dlet(x << "the env x") def test4(): - nonlocal x + nonlocal x # noqa: F824, Python 3.12+ complain about this; just testing our let construct; it's correct that there's no local `x` as per Python's normal scoping rules. return x test[test4() == "the nonlocal x"] @dlet(x << "the env x") def test5(): - global x + global x # noqa: F824, Python 3.12+ complain about this; just testing our let construct; it's correct that there's no local `x` as per Python's normal scoping rules. return x test[test5() == "the global x"] diff --git a/unpythonic/syntax/tests/test_scopeanalyzer.py b/unpythonic/syntax/tests/test_scopeanalyzer.py index 1b4d34d6..a41a64ef 100644 --- a/unpythonic/syntax/tests/test_scopeanalyzer.py +++ b/unpythonic/syntax/tests/test_scopeanalyzer.py @@ -179,8 +179,8 @@ def f3(): with q as getlexvars_fdef: y = 21 def myfunc(x, *args, kwonlyarg, **kwargs): - nonlocal y # not really needed here, except for exercising the analyzer. - global g + nonlocal y # noqa: F824, for Python 3.12+; just testing our scope analyzer; it's correct that there's no local `y`. Also, not really needed here, except for exercising the analyzer. + global g # noqa: F824, Python 3.12+ complain about this; just testing our scope analyzer; it's correct that there's no local `g`. def inner(blah): abc = 123 # noqa: F841 z = 2 * y # noqa: F841 From f61abb2d23206062cd36319e7e705f258e2a18bd Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 10:05:29 +0300 Subject: [PATCH 762/832] attempt to upgrade CI to use pdm --- .github/workflows/coverage.yml | 5 ++++- .github/workflows/python-package.yml | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 76d035ef..7b82332b 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -27,10 +27,13 @@ jobs: run: | python -m pip install --upgrade pip pip install flake8 - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + pip install pdm + pdm python install ${{ matrix.python-version }} + pdm install - name: Generate coverage report run: | pip install coverage + $(pdm venv activate) coverage run --source=. -m runtests coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 60b8eee8..df93bd1c 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -29,7 +29,9 @@ jobs: run: | python -m pip install --upgrade pip pip install flake8 - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + pip install pdm + pdm python install ${{ matrix.python-version }} + pdm install - name: Lint with flake8 run: | # stop the build if there are Python syntax errors or undefined names @@ -38,4 +40,5 @@ jobs: flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - name: Test with unpythonic.test.fixtures run: | + $(pdm venv activate) python runtests.py From e9da23bf2edd5bc6f82d4bcecb50fa33ab228e45 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 12:02:36 +0300 Subject: [PATCH 763/832] CI: trying a different way to activate the PDM venv --- .github/workflows/coverage.yml | 2 +- .github/workflows/python-package.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 7b82332b..9636c5c1 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -33,7 +33,7 @@ jobs: - name: Generate coverage report run: | pip install coverage - $(pdm venv activate) + . .venv/bin/activate coverage run --source=. -m runtests coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index df93bd1c..e61b745f 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -40,5 +40,5 @@ jobs: flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - name: Test with unpythonic.test.fixtures run: | - $(pdm venv activate) + . .venv/bin/activate python runtests.py From fa2fbe4ae1d650c3859c4b9e6a026552a738d310 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 13:34:02 +0300 Subject: [PATCH 764/832] CI: trying a third way to activate the PDM venv --- .github/workflows/coverage.yml | 4 +++- .github/workflows/python-package.yml | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 9636c5c1..44bc9995 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -30,12 +30,14 @@ jobs: pip install pdm pdm python install ${{ matrix.python-version }} pdm install + working-directory: /home/runner/work/unpythonic/ - name: Generate coverage report run: | pip install coverage - . .venv/bin/activate + source .venv/bin/activate coverage run --source=. -m runtests coverage xml + working-directory: /home/runner/work/unpythonic/ - name: Upload coverage to Codecov uses: codecov/codecov-action@v1 with: diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index e61b745f..bb6ff8cf 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -32,13 +32,16 @@ jobs: pip install pdm pdm python install ${{ matrix.python-version }} pdm install + working-directory: /home/runner/work/unpythonic/ - name: Lint with flake8 run: | # stop the build if there are Python syntax errors or undefined names flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics + working-directory: /home/runner/work/unpythonic/ - name: Test with unpythonic.test.fixtures run: | - . .venv/bin/activate + source .venv/bin/activate python runtests.py + working-directory: /home/runner/work/unpythonic/ From 32d894461a41b97a3c3a469e858cb840c5ed66e7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 13:37:39 +0300 Subject: [PATCH 765/832] CI: maybe like this then --- .github/workflows/coverage.yml | 1 - .github/workflows/python-package.yml | 1 - 2 files changed, 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 44bc9995..e7e34381 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -30,7 +30,6 @@ jobs: pip install pdm pdm python install ${{ matrix.python-version }} pdm install - working-directory: /home/runner/work/unpythonic/ - name: Generate coverage report run: | pip install coverage diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index bb6ff8cf..38b5a013 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -32,7 +32,6 @@ jobs: pip install pdm pdm python install ${{ matrix.python-version }} pdm install - working-directory: /home/runner/work/unpythonic/ - name: Lint with flake8 run: | # stop the build if there are Python syntax errors or undefined names From e43d23e8671a8ad57eb3220d5ce89b96139bcd1f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 13:40:58 +0300 Subject: [PATCH 766/832] CI: ugh --- .github/workflows/coverage.yml | 4 ++-- .github/workflows/python-package.yml | 5 ++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index e7e34381..912e5c07 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -33,10 +33,10 @@ jobs: - name: Generate coverage report run: | pip install coverage - source .venv/bin/activate + # For some reason, the install step creates the venv one directory too deep. Maybe doesn't matter. + source /home/runner/work/unpythonic/unpythonic/.venv/bin/activate coverage run --source=. -m runtests coverage xml - working-directory: /home/runner/work/unpythonic/ - name: Upload coverage to Codecov uses: codecov/codecov-action@v1 with: diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 38b5a013..cf4589f2 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -38,9 +38,8 @@ jobs: flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - working-directory: /home/runner/work/unpythonic/ - name: Test with unpythonic.test.fixtures run: | - source .venv/bin/activate + # For some reason, the install step creates the venv one directory too deep. Maybe doesn't matter. + source /home/runner/work/unpythonic/unpythonic/.venv/bin/activate python runtests.py - working-directory: /home/runner/work/unpythonic/ From f4b29dd3bad56431fe17863b97ac060ed36720fa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 13:49:39 +0300 Subject: [PATCH 767/832] CI: ugh again --- .github/workflows/coverage.yml | 14 ++++++++++---- .github/workflows/python-package.yml | 12 +++++++++--- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 912e5c07..2e05a608 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -23,18 +23,24 @@ jobs: uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - - name: Install dependencies + - name: Install tools run: | python -m pip install --upgrade pip pip install flake8 + pip install coverage pip install pdm + - name: Create virtualenv + run: | pdm python install ${{ matrix.python-version }} + - name: Install dependencies + run: | pdm install + - name: Activate virtualenv + run: | + . .venv/bin/activate + echo PATH=$PATH >> $GITHUB_ENV - name: Generate coverage report run: | - pip install coverage - # For some reason, the install step creates the venv one directory too deep. Maybe doesn't matter. - source /home/runner/work/unpythonic/unpythonic/.venv/bin/activate coverage run --source=. -m runtests coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index cf4589f2..8acd4f89 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -25,12 +25,16 @@ jobs: uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - - name: Install dependencies + - name: Install tools run: | python -m pip install --upgrade pip pip install flake8 pip install pdm + - name: Create virtualenv + run: | pdm python install ${{ matrix.python-version }} + - name: Install dependencies + run: | pdm install - name: Lint with flake8 run: | @@ -38,8 +42,10 @@ jobs: flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics + - name: Activate virtualenv + run: | + . .venv/bin/activate + echo PATH=$PATH >> $GITHUB_ENV - name: Test with unpythonic.test.fixtures run: | - # For some reason, the install step creates the venv one directory too deep. Maybe doesn't matter. - source /home/runner/work/unpythonic/unpythonic/.venv/bin/activate python runtests.py From 8b855b93d627d918798b807fb20e5f2f94d4451d Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:09:18 +0300 Subject: [PATCH 768/832] CI ughhh --- .github/workflows/coverage.yml | 9 ++++++--- .github/workflows/python-package.yml | 19 +++++++++++-------- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 2e05a608..ae1217a9 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -32,13 +32,16 @@ jobs: - name: Create virtualenv run: | pdm python install ${{ matrix.python-version }} - - name: Install dependencies + - name: Install dependencies into virtualenv run: | pdm install - name: Activate virtualenv run: | - . .venv/bin/activate - echo PATH=$PATH >> $GITHUB_ENV + # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable + # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action + source .venv/bin/activate + echo "PATH=$PATH" >> "$GITHUB_ENV" + echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Generate coverage report run: | coverage run --source=. -m runtests diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 8acd4f89..86af914c 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -30,22 +30,25 @@ jobs: python -m pip install --upgrade pip pip install flake8 pip install pdm - - name: Create virtualenv - run: | - pdm python install ${{ matrix.python-version }} - - name: Install dependencies - run: | - pdm install - name: Lint with flake8 run: | # stop the build if there are Python syntax errors or undefined names flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics + - name: Create virtualenv + run: | + pdm python install ${{ matrix.python-version }} + - name: Install dependencies into virtualenv + run: | + pdm install - name: Activate virtualenv run: | - . .venv/bin/activate - echo PATH=$PATH >> $GITHUB_ENV + # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable + # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action + source .venv/bin/activate + echo "PATH=$PATH" >> "$GITHUB_ENV" + echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | python runtests.py From 6d8e94c4d59532727fe8cb8b209a89e728fa7ada Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:17:24 +0300 Subject: [PATCH 769/832] ughhhhh --- .github/workflows/coverage.yml | 3 ++- .github/workflows/python-package.yml | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index ae1217a9..b2868703 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -37,11 +37,12 @@ jobs: pdm install - name: Activate virtualenv run: | + source .venv/bin/activate # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action - source .venv/bin/activate echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" + echo "unset PYTHONHOME" >> "$GITHUB_ENV" - name: Generate coverage report run: | coverage run --source=. -m runtests diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 86af914c..f52f2b91 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -44,11 +44,12 @@ jobs: pdm install - name: Activate virtualenv run: | + source .venv/bin/activate # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action - source .venv/bin/activate echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" + echo "unset PYTHONHOME" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | python runtests.py From 3b64b287e0449a20a01babe80795cdd94ec06ca8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:20:18 +0300 Subject: [PATCH 770/832] arghh --- .github/workflows/coverage.yml | 4 +++- .github/workflows/python-package.yml | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index b2868703..05d20cf5 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -42,9 +42,11 @@ jobs: # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - echo "unset PYTHONHOME" >> "$GITHUB_ENV" - name: Generate coverage report run: | + # coverage report + # https://stackoverflow.com/questions/70137245/how-to-remove-an-environment-variable-on-github-actions + unset PYTHONHOME coverage run --source=. -m runtests coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index f52f2b91..3908d385 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -49,7 +49,9 @@ jobs: # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - echo "unset PYTHONHOME" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | + # run the tests + # https://stackoverflow.com/questions/70137245/how-to-remove-an-environment-variable-on-github-actions + unset PYTHONHOME python runtests.py From 882a25fa2d8eeb50368fd2964ccbd76363d4848e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:40:11 +0300 Subject: [PATCH 771/832] argggghhhh --- .github/workflows/coverage.yml | 9 +++------ .github/workflows/python-package.yml | 9 +++------ 2 files changed, 6 insertions(+), 12 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 05d20cf5..3f82c48e 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -32,9 +32,6 @@ jobs: - name: Create virtualenv run: | pdm python install ${{ matrix.python-version }} - - name: Install dependencies into virtualenv - run: | - pdm install - name: Activate virtualenv run: | source .venv/bin/activate @@ -42,11 +39,11 @@ jobs: # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" + - name: Install dependencies into virtualenv + run: | + pdm install - name: Generate coverage report run: | - # coverage report - # https://stackoverflow.com/questions/70137245/how-to-remove-an-environment-variable-on-github-actions - unset PYTHONHOME coverage run --source=. -m runtests coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 3908d385..aea2cd7d 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -39,9 +39,6 @@ jobs: - name: Create virtualenv run: | pdm python install ${{ matrix.python-version }} - - name: Install dependencies into virtualenv - run: | - pdm install - name: Activate virtualenv run: | source .venv/bin/activate @@ -49,9 +46,9 @@ jobs: # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" + - name: Install dependencies into virtualenv + run: | + pdm install - name: Test with unpythonic.test.fixtures run: | - # run the tests - # https://stackoverflow.com/questions/70137245/how-to-remove-an-environment-variable-on-github-actions - unset PYTHONHOME python runtests.py From 8bd304b0741000951da1cbd64fee9a07fc9a609f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:45:07 +0300 Subject: [PATCH 772/832] argh again --- .github/workflows/coverage.yml | 3 ++- .github/workflows/python-package.yml | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 3f82c48e..1edb32f7 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -29,11 +29,12 @@ jobs: pip install flake8 pip install coverage pip install pdm - - name: Create virtualenv + - name: Install Python for pdm venv run: | pdm python install ${{ matrix.python-version }} - name: Activate virtualenv run: | + pdm use --venv in-project source .venv/bin/activate # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index aea2cd7d..1ed01595 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -36,11 +36,12 @@ jobs: flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - - name: Create virtualenv + - name: Install Python for pdm venv run: | pdm python install ${{ matrix.python-version }} - name: Activate virtualenv run: | + pdm use --venv in-project source .venv/bin/activate # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action From 2499b2d63593a969b5df105ddf3ff1263074e956 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:47:37 +0300 Subject: [PATCH 773/832] raaah --- .github/workflows/coverage.yml | 9 +++++---- .github/workflows/python-package.yml | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 1edb32f7..ff61f5f3 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -29,9 +29,13 @@ jobs: pip install flake8 pip install coverage pip install pdm - - name: Install Python for pdm venv + - name: Create virtualenv run: | pdm python install ${{ matrix.python-version }} + # "When you run pdm install the first time on a new PDM-managed project, whose Python interpreter is not decided yet, + # PDM will create a virtualenv in /.venv, and install dependencies into it." + # --https://pdm-project.org/en/latest/usage/venv/ + pdm install - name: Activate virtualenv run: | pdm use --venv in-project @@ -40,9 +44,6 @@ jobs: # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - - name: Install dependencies into virtualenv - run: | - pdm install - name: Generate coverage report run: | coverage run --source=. -m runtests diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 1ed01595..46703fa3 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -36,9 +36,13 @@ jobs: flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - - name: Install Python for pdm venv + - name: Create virtualenv run: | pdm python install ${{ matrix.python-version }} + # "When you run pdm install the first time on a new PDM-managed project, whose Python interpreter is not decided yet, + # PDM will create a virtualenv in /.venv, and install dependencies into it." + # --https://pdm-project.org/en/latest/usage/venv/ + pdm install - name: Activate virtualenv run: | pdm use --venv in-project @@ -47,9 +51,6 @@ jobs: # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action echo "PATH=$PATH" >> "$GITHUB_ENV" echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - - name: Install dependencies into virtualenv - run: | - pdm install - name: Test with unpythonic.test.fixtures run: | python runtests.py From e60d66ea4c6b19c6c52efe5e4923e7d0f6f4d31b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:48:53 +0300 Subject: [PATCH 774/832] raah2 --- .github/workflows/coverage.yml | 2 ++ .github/workflows/python-package.yml | 2 ++ 2 files changed, 4 insertions(+) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index ff61f5f3..d26037ca 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -46,6 +46,8 @@ jobs: echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Generate coverage report run: | + pdm use --venv in-project + source .venv/bin/activate coverage run --source=. -m runtests coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 46703fa3..45061f96 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -53,4 +53,6 @@ jobs: echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | + pdm use --venv in-project + source .venv/bin/activate python runtests.py From 824366fd4b22cbe5427a752393656e869f1bb09c Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:53:18 +0300 Subject: [PATCH 775/832] CI: hmm, maybe like this --- .github/workflows/coverage.yml | 6 ++---- .github/workflows/python-package.yml | 4 +--- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index d26037ca..b5104f4f 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -46,10 +46,8 @@ jobs: echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Generate coverage report run: | - pdm use --venv in-project - source .venv/bin/activate - coverage run --source=. -m runtests - coverage xml + python -m coverage run --source=. -m runtests + python -m coverage xml - name: Upload coverage to Codecov uses: codecov/codecov-action@v1 with: diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 45061f96..6bffbac7 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -1,7 +1,7 @@ # This workflow will install Python dependencies, run tests and lint with a variety of Python versions # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions # -# This version is customized to use the local flake8rc and test with unpythonic.setup.fixtures. +# This version is customized to install with pdm, use the local flake8rc, and test with unpythonic.setup.fixtures. name: Python package @@ -53,6 +53,4 @@ jobs: echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | - pdm use --venv in-project - source .venv/bin/activate python runtests.py From ada524f3968aa06a6396862e72092d9a48e13940 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:54:42 +0300 Subject: [PATCH 776/832] CI: or maybe like this --- .github/workflows/coverage.yml | 2 ++ .github/workflows/python-package.yml | 2 ++ 2 files changed, 4 insertions(+) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index b5104f4f..6d38c40a 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -46,6 +46,8 @@ jobs: echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Generate coverage report run: | + pdm use --venv in-project + source .venv/bin/activate python -m coverage run --source=. -m runtests python -m coverage xml - name: Upload coverage to Codecov diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 6bffbac7..b7c7b90e 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -53,4 +53,6 @@ jobs: echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | + pdm use --venv in-project + source .venv/bin/activate python runtests.py From 97bff1708e74d5a09ed3c4f14a9ee9504d859dfe Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 14:56:57 +0300 Subject: [PATCH 777/832] CI: maybe this will finally work? --- .github/workflows/coverage.yml | 10 +++------- .github/workflows/python-package.yml | 10 +--------- 2 files changed, 4 insertions(+), 16 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 6d38c40a..bcd5464d 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -27,23 +27,19 @@ jobs: run: | python -m pip install --upgrade pip pip install flake8 - pip install coverage pip install pdm - - name: Create virtualenv + - name: Create virtualenv and install dependencies run: | pdm python install ${{ matrix.python-version }} # "When you run pdm install the first time on a new PDM-managed project, whose Python interpreter is not decided yet, # PDM will create a virtualenv in /.venv, and install dependencies into it." # --https://pdm-project.org/en/latest/usage/venv/ pdm install - - name: Activate virtualenv + - name: Install coverage tool in virtualenv run: | pdm use --venv in-project source .venv/bin/activate - # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable - # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action - echo "PATH=$PATH" >> "$GITHUB_ENV" - echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" + pip install coverage - name: Generate coverage report run: | pdm use --venv in-project diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index b7c7b90e..0de4cfc4 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -36,21 +36,13 @@ jobs: flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - - name: Create virtualenv + - name: Create virtualenv and install dependencies run: | pdm python install ${{ matrix.python-version }} # "When you run pdm install the first time on a new PDM-managed project, whose Python interpreter is not decided yet, # PDM will create a virtualenv in /.venv, and install dependencies into it." # --https://pdm-project.org/en/latest/usage/venv/ pdm install - - name: Activate virtualenv - run: | - pdm use --venv in-project - source .venv/bin/activate - # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable - # https://stackoverflow.com/questions/74668349/how-to-activate-a-virtualenv-in-a-github-action - echo "PATH=$PATH" >> "$GITHUB_ENV" - echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> "$GITHUB_ENV" - name: Test with unpythonic.test.fixtures run: | pdm use --venv in-project From 460346eb9cfc1737771235b9a676a1f051aaaa81 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 15:02:09 +0300 Subject: [PATCH 778/832] install coverage in the correct venv --- .github/workflows/coverage.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index bcd5464d..061017b8 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -37,9 +37,8 @@ jobs: pdm install - name: Install coverage tool in virtualenv run: | - pdm use --venv in-project - source .venv/bin/activate - pip install coverage + pdm run python -m ensurepip + pdm run python -m pip install coverage - name: Generate coverage report run: | pdm use --venv in-project From 47eec053ce56779b072fda5da5b2e314ec591925 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 15:16:50 +0300 Subject: [PATCH 779/832] CI scripts, again --- .github/workflows/coverage.yml | 8 ++++---- .github/workflows/python-package.yml | 15 +++++++++++---- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 061017b8..2f176438 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -23,19 +23,19 @@ jobs: uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - - name: Install tools + - name: Install tools in CI virtualenv run: | python -m pip install --upgrade pip pip install flake8 pip install pdm - - name: Create virtualenv and install dependencies + - name: Create in-project virtualenv and install dependencies run: | pdm python install ${{ matrix.python-version }} # "When you run pdm install the first time on a new PDM-managed project, whose Python interpreter is not decided yet, # PDM will create a virtualenv in /.venv, and install dependencies into it." - # --https://pdm-project.org/en/latest/usage/venv/ + # https://pdm-project.org/en/latest/usage/venv/ pdm install - - name: Install coverage tool in virtualenv + - name: Install coverage tool in in-project virtualenv run: | pdm run python -m ensurepip pdm run python -m pip install coverage diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 0de4cfc4..91a89073 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -25,7 +25,7 @@ jobs: uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - - name: Install tools + - name: Install tools in CI venv run: | python -m pip install --upgrade pip pip install flake8 @@ -36,12 +36,19 @@ jobs: flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - - name: Create virtualenv and install dependencies + - name: Determine Python version string for PDM run: | - pdm python install ${{ matrix.python-version }} + TARGET_PYTHON_VERSION_FOR_PDM=$( python -c 'import sys; v = sys.argv[1]; print(v if "-" not in v else v.replace("-", "@"))' ${{ matrix.python-version }} ) + # We need this hack at all because CI expects e.g. "pypy-3.10", whereas PDM expects "pypy@3.10". + # Now that we have the result, send it to an environment variable so that the next step can use it. + # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable + echo "TARGET_PYTHON_VERSION=$TARGET_PYTHON_VERSION_FOR_PDM" >> "$GITHUB_ENV" + - name: Create in-project virtualenv and install dependencies + run: | + pdm python install "$TARGET_PYTHON_VERSION_FOR_PDM" # "When you run pdm install the first time on a new PDM-managed project, whose Python interpreter is not decided yet, # PDM will create a virtualenv in /.venv, and install dependencies into it." - # --https://pdm-project.org/en/latest/usage/venv/ + # https://pdm-project.org/en/latest/usage/venv/ pdm install - name: Test with unpythonic.test.fixtures run: | From 184eccabf9bf31113083d93ec8dca08ad1ea97f8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 15:27:44 +0300 Subject: [PATCH 780/832] CI: another attempt --- .github/workflows/python-package.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 91a89073..b792bdbc 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -38,11 +38,10 @@ jobs: flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - name: Determine Python version string for PDM run: | - TARGET_PYTHON_VERSION_FOR_PDM=$( python -c 'import sys; v = sys.argv[1]; print(v if "-" not in v else v.replace("-", "@"))' ${{ matrix.python-version }} ) + echo "TARGET_PYTHON_VERSION_FOR_PDM=${{ matrix.python-version }}" | tr - @ >> "$GITHUB_ENV" # We need this hack at all because CI expects e.g. "pypy-3.10", whereas PDM expects "pypy@3.10". # Now that we have the result, send it to an environment variable so that the next step can use it. # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable - echo "TARGET_PYTHON_VERSION=$TARGET_PYTHON_VERSION_FOR_PDM" >> "$GITHUB_ENV" - name: Create in-project virtualenv and install dependencies run: | pdm python install "$TARGET_PYTHON_VERSION_FOR_PDM" From 3616155027e59721d76c61ef198fad23ace184fa Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 15:33:11 +0300 Subject: [PATCH 781/832] update comments --- .github/workflows/coverage.yml | 1 + .github/workflows/python-package.yml | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 2f176438..e5559af3 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -38,6 +38,7 @@ jobs: - name: Install coverage tool in in-project virtualenv run: | pdm run python -m ensurepip + # coverage must run in the same venv as the code being tested. pdm run python -m pip install coverage - name: Generate coverage report run: | diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index b792bdbc..69417f72 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -40,7 +40,7 @@ jobs: run: | echo "TARGET_PYTHON_VERSION_FOR_PDM=${{ matrix.python-version }}" | tr - @ >> "$GITHUB_ENV" # We need this hack at all because CI expects e.g. "pypy-3.10", whereas PDM expects "pypy@3.10". - # Now that we have the result, send it to an environment variable so that the next step can use it. + # We send the result into an environment variable so that the next step can use it. # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#setting-an-environment-variable - name: Create in-project virtualenv and install dependencies run: | From 146569e36eda28208be51a5716ec82d9f71e9152 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 15:43:27 +0300 Subject: [PATCH 782/832] update CHANGELOG --- CHANGELOG.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e7d84b1..476b86b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,10 @@ **Changed**: -- Internal: Upgrade build system to `pdm`. This is important for the road ahead, since the old `setuptools` build system has been deprecated. +- Internal: Upgrade build system to `pdm`. + - This is important for the road ahead, since the old `setuptools` build system has been deprecated. + - The GitHub CI scripts for `unpythonic` now use PDM to manage the testing venv and dependencies, too. Now the tests should run the same way as they would on a local system. + - Bump `mcpyrate` to the hotfix version 3.6.4. - The only difference is (beside `mcpyrate` too internally upgrading its build system to `pdm`) that the text colorizer now works correctly also for `input` with `readline`. From 0b2f542849383d5d8fa63fb8ec0877b16f64af86 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 16:10:09 +0300 Subject: [PATCH 783/832] document how to use development mode --- README.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/README.md b/README.md index 3d94b8b4..b1334509 100644 --- a/README.md +++ b/README.md @@ -828,6 +828,52 @@ If `--no-compile` is NOT used, the precompiled bytecode cache may cause errors s This is a common issue when using macro expanders in Python. +### Development mode (for developing `unpythonic` itself) + +Starting with v0.15.5, `unpythonic` uses [PDM](https://pdm-project.org/en/latest/) to manage its dependencies. This allows easy installation of a development copy into an isolated venv (virtual environment), allowing you to break things without breaking anything else on your system (including apps and libraries that use an installed copy of `unpythonic`). + +#### Install PDM in your Python environment + +To develop `unpythonic`, if your Python environment does not have PDM, you will need to install it first: + +```bash +python -m pip install pdm +``` + +Don't worry; it won't break `pip`, `poetry`, or other similar tools. + +We will also need a Python for PDM venvs. This Python is independent of the Python that PDM itself runs on. It is the version of Python you would like to use for developing `unpythonic`. + +For example, we can make Python 3.10 available with the command: + +```bash +pdm python install 3.10 +``` + +Specifying just a version number defaults to CPython (the usual Python implementation). If you want PyPy instead, you can use e.g. `pypy@3.10`. + +#### Install the isolated venv + +Now, we will auto-create the development venv, and install `unpythonic`'s dependencies into it. In a terminal that sees your Python environment, navigate to the `unpythonic` folder, and issue the command: + +```bash +pdm install +``` + +This creates the development venv into the `.venv` hidden subfolder of the `unpythonic` folder. + +If you are a seasoned pythonista, note that there is no `requirements.txt`; the dependency list lives in `pyproject.toml`. + +#### Develop + +To activate the development venv, in a terminal that sees your Python environment, navigate to the `unpythonic` folder, and issue the command: + +```bash +$(pdm venv activate) +``` + +Note the Bash exec syntax `$(...)`; the command `pdm venv activate` just prints the actual internal activation command. + ### Uninstall ```bash From 3b054dad6ca5af735d49b053d58e4724aacdd694 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 16 Apr 2025 16:21:46 +0300 Subject: [PATCH 784/832] add note about upgrading dependencies in develop mode --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index b1334509..622ea93a 100644 --- a/README.md +++ b/README.md @@ -864,6 +864,14 @@ This creates the development venv into the `.venv` hidden subfolder of the `unpy If you are a seasoned pythonista, note that there is no `requirements.txt`; the dependency list lives in `pyproject.toml`. +#### Upgrade dependencies (later) + +To upgrade dependencies to latest available versions compatible with the specifications in `pyproject.toml`: + +```bash +pdm update +``` + #### Develop To activate the development venv, in a terminal that sees your Python environment, navigate to the `unpythonic` folder, and issue the command: From cbe315e4c28f70ea18201e3be3680f8c4dfb29ab Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 4 Feb 2026 13:03:38 +0200 Subject: [PATCH 785/832] add CLAUDE.md for Claude Code guidance Co-Authored-By: Claude Opus 4.5 --- CLAUDE.md | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..096edda5 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,79 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What is unpythonic + +A Python library providing language extensions and utilities inspired by Lisp, Haskell, and functional programming. Three-tier architecture: + +1. **Pure Python layer** (`unpythonic/`): ~45 modules of functional utilities (curry, memoize, fold, TCO, conditions/restarts, dynamic variables, linked lists, etc.). No macro dependency. +2. **Macro layer** (`unpythonic/syntax/`): Syntactic macros via `mcpyrate` providing cleaner syntax for let-bindings, autocurry, lazify, TCO, continuations, etc. +3. **Dialect layer** (`unpythonic/dialects/`): Full language variants (Lispython, Listhell, Pytkell) built on the macro layer. + +## Build and development + +Uses PDM with `pdm-backend`. Python 3.8–3.12, also PyPy 3.8–3.10. + +```bash +# Set up development environment +pdm install # creates .venv/ and installs deps +pdm use --venv in-project +source .venv/bin/activate +``` + +**Critical**: When installing from source, never use `--compile` / precompilation. Precompiled bytecode without macro support breaks macro imports like `from unpythonic.syntax import macros, let`. + +## Running tests + +Custom test framework (not pytest). Tests use macros (`test[]`, `test_raises[]`) and conditions/restarts for reporting. The test runner does not need the `macropython` wrapper—it activates macros via `import mcpyrate.activate`. + +```bash +# Run all tests (from repo root, with venv activated) +python runtests.py + +# Run a single test module directly +python -c "import mcpyrate.activate; from unpythonic.tests.test_fun import runtests; runtests()" + +# Run macro tests similarly +python -c "import mcpyrate.activate; from unpythonic.syntax.tests.test_letdo import runtests; runtests()" +``` + +Test suites discovered by `runtests.py`: +- `unpythonic/tests/test_*.py` — pure Python features +- `unpythonic/net/tests/test_*.py` — REPL server/client +- `unpythonic/syntax/tests/test_*.py` — macro features +- `unpythonic/dialects/tests/test_*.py` — dialect features + +Each test module exports a `runtests()` function. Tests are grouped with `testset()` context managers. + +## Linting + +```bash +# As in CI — hard errors (syntax errors, undefined names) +flake8 . --config=flake8rc --select=E9,F63,F7,F82 --show-source + +# Soft warnings +flake8 . --config=flake8rc --exit-zero --max-line-length=127 +``` + +## Code structure and conventions + +- **Regular code** in `unpythonic/`, **macros** in `unpythonic/syntax/`, **REPL networking** in `unpythonic/net/`, **dialects** in `unpythonic/dialects/`. +- **Tests** are in `tests/` (plural) subdirectories under the code they test. The testing *framework* lives at `unpythonic/test/` (singular). +- Each module declares `__all__` explicitly for public API. The top-level `__init__.py` re-exports via star imports. +- **Import style**: Use `from ... import ...` (not `import ...`). The from-import syntax is mandatory for macro imports and used consistently throughout. Don't rename unpythonic features with `as`—macro code depends on original bare names. +- **No star imports** in user code (only in the top-level `__init__.py` for re-export). +- **Curry-friendly signatures**: Parameters that change least often go on the left. Use `def f(func, thing0, *things)` (not `def f(func, *things)`) when at least one `thing` is required, so `curry` knows when to trigger. +- **Macros are the nuclear option**: Only make a macro when a regular function can't do the job. Prefer a pure-Python core with a thin macro layer for UX. +- **Macro `**kw` passing**: Use `dyn` (dynamic variables) to pass `mcpyrate` `**kw` arguments through to syntax transformers, rather than threading them through parameter lists. +- **Line width** ~110 characters. Docstrings in reStructuredText. +- **Module size target**: ~100–300 SLOC, rough max ~700 lines. +- **Dependencies**: Avoid external dependencies. `mcpyrate` is the only allowed external dep and must remain strictly optional for the pure-Python layer. + +## Key cross-cutting concerns + +- `curry` has cross-cutting behavior — grep for it when investigating interactions. +- `@generic` (multiple dispatch) similarly has cross-cutting concerns. +- The `lazify` macro: also grep for `passthrough_lazy_args` and `maybe_force_args`. +- The `continuations` macro builds on `tco` — read `tco` first when studying continuations. +- `unpythonic.syntax.scopeanalyzer` implements lexical scope analysis for macros that interact with Python's scoping rules (notably `let`). From cf7dcd7758ac5672b940d1bc88353cfba104d82e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 4 Feb 2026 13:10:43 +0200 Subject: [PATCH 786/832] udpate gitignore --- .gitignore | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.gitignore b/.gitignore index 2a6cbf88..ac831acf 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,15 @@ +00_stuff +__pycache__ *~ *.pyc *.c build dist +MANIFEST +pdm.lock +.pdm-python .spyproject +:venv *.egg-info +*.mypy_cache +.python-version From 6c4330d7cccfb22583e1ff6ff82bcafab1c7b86f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 19 Feb 2026 14:14:48 +0200 Subject: [PATCH 787/832] update flake8rc as per Claude's recommendations (PEP8 Knuth style, Black/Ruff style) --- flake8rc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/flake8rc b/flake8rc index 23a1ff37..3e3fe033 100644 --- a/flake8rc +++ b/flake8rc @@ -25,6 +25,10 @@ ignore = E704, # do not assign a lambda expression, use a def (because autopep8 applies it blindly) E731, + # whitespace before ':' (false positive on alignment and slices; Black/Ruff agree) + E203, + # line break before binary operator (PEP 8 recommends Knuth's style, i.e. break before) + W503, # line break after binary operator W504 exclude = .git,__pycache__,docs/source/conf.py,old,build,dist,node_modules,instance,00_stuff,00_old From f43de91cca96018f9e46f3bc7b044ae890d19d08 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 21 Feb 2026 09:41:15 +0200 Subject: [PATCH 788/832] =?UTF-8?q?bump=20version=20to=201.0.0=20=E2=80=94?= =?UTF-8?q?=20re-release=20of=200.15.5=20as=20stable?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The library has been stable and in light maintenance mode for years; the version number now reflects this de facto status quo. Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 7 +++++++ unpythonic/__init__.py | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 476b86b4..09c1ee8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +**1.0.0** (21 February 2026) — *"Same supercharger, new badge"* edition: + +Re-release of 0.15.5 as 1.0.0. No code changes. The library has been stable and in light maintenance mode for years; the version number now reflects this de facto status quo. + + +--- + **0.15.5** (16 April 2025) - hotfix: **Changed**: diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 9b5f4dc2..7a949b9e 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '0.15.5' +__version__ = '1.0.0' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From cad8576350894eda51a9180ef51c062367dede86 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 21 Feb 2026 09:45:03 +0200 Subject: [PATCH 789/832] add Dependabot for GitHub Actions, bump action versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add weekly Dependabot updates for the github-actions ecosystem. Bump checkout v2→v4, setup-python v2→v5, codecov-action v1→v5. Co-Authored-By: Claude Opus 4.6 --- .github/dependabot.yml | 6 ++++++ .github/workflows/coverage.yml | 6 +++--- .github/workflows/python-package.yml | 4 ++-- 3 files changed, 11 insertions(+), 5 deletions(-) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 00000000..5ace4600 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,6 @@ +version: 2 +updates: + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index e5559af3..f10f23cb 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -18,9 +18,9 @@ jobs: python-version: ["3.10"] steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Install tools in CI virtualenv @@ -47,7 +47,7 @@ jobs: python -m coverage run --source=. -m runtests python -m coverage xml - name: Upload coverage to Codecov - uses: codecov/codecov-action@v1 + uses: codecov/codecov-action@v5 with: token: ${{ secrets.CODECOV_TOKEN }} file: ./coverage.xml diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 69417f72..7c575a43 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -20,9 +20,9 @@ jobs: python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", pypy-3.8, pypy-3.9, pypy-3.10] steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Install tools in CI venv From fe012c2450096e800524887a7530ce76020353d5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 21 Feb 2026 07:46:13 +0000 Subject: [PATCH 790/832] Bump actions/setup-python from 5 to 6 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/coverage.yml | 2 +- .github/workflows/python-package.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index f10f23cb..d96aa93e 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -20,7 +20,7 @@ jobs: steps: - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-version }} - name: Install tools in CI virtualenv diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 7c575a43..1f8f2f38 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -22,7 +22,7 @@ jobs: steps: - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-version }} - name: Install tools in CI venv From 4b2681f6575ed03f639f3cc0c875ad26a1f96abf Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 21 Feb 2026 09:48:13 +0200 Subject: [PATCH 791/832] add Claude as AI pair programmer in AUTHORS.md Co-Authored-By: Claude Opus 4.6 --- AUTHORS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AUTHORS.md b/AUTHORS.md index 584a1c65..e0405378 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -2,6 +2,7 @@ - Juha Jeronen (@Technologicat) - original author - @aisha-w - documentation improvements +- @Technologicat with Claude (Anthropic) as AI pair programmer - CI modernization **Design inspiration from the internet**: From 06a188dce387b8ba978d3e5462c824fa5c90615f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 21 Feb 2026 07:50:29 +0000 Subject: [PATCH 792/832] Bump actions/checkout from 4 to 6 Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/coverage.yml | 2 +- .github/workflows/python-package.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index d96aa93e..e8f27c59 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -18,7 +18,7 @@ jobs: python-version: ["3.10"] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v6 with: diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 1f8f2f38..5b41d327 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -20,7 +20,7 @@ jobs: python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", pypy-3.8, pypy-3.9, pypy-3.10] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v6 with: From de5902e6a24d9f24370b85003dda1735f9ccce65 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Sat, 21 Feb 2026 09:58:02 +0200 Subject: [PATCH 793/832] note semantic versioning in README Co-Authored-By: Claude Opus 4.6 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 622ea93a..e373a3fb 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing f ![version on PyPI](https://img.shields.io/pypi/v/unpythonic) ![PyPI package format](https://img.shields.io/pypi/format/unpythonic) ![dependency status](https://img.shields.io/librariesio/github/Technologicat/unpythonic) ![license: BSD](https://img.shields.io/pypi/l/unpythonic) ![open issues](https://img.shields.io/github/issues/Technologicat/unpythonic) [![PRs welcome](https://img.shields.io/badge/PRs-welcome-brightgreen)](http://makeapullrequest.com/) +We use [semantic versioning](https://semver.org/). + *Some hypertext features of this README, such as local links to detailed documentation, and expandable example highlights, are not supported when viewed on PyPI; [view on GitHub](https://github.com/Technologicat/unpythonic) to have those work properly.* From c35f7e94814ab3155b0a7d80a9ec51b769f619d3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 3 Mar 2026 11:37:23 +0200 Subject: [PATCH 794/832] Windows support: termios is not available, make ptyproxy optional This is a quick workaround that essentially disables the REPL server when running in a MS Windows environment. --- unpythonic/net/__init__.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/unpythonic/net/__init__.py b/unpythonic/net/__init__.py index 2b3ca9c9..a16495b0 100644 --- a/unpythonic/net/__init__.py +++ b/unpythonic/net/__init__.py @@ -10,5 +10,12 @@ """ from .msg import * -from .ptyproxy import * +try: + from .ptyproxy import * +except ModuleNotFoundError: + import logging + logging.basicConfig(level=logging.INFO) + logger = logging.getLogger(__name__) + logger.info("`unpythonic.net.ptyproxy` could not be loaded, the REPL server will not be available. Usually this is harmless; most applications do not need the REPL server.") + PTYSocketProxy = None from .util import * From ce44b3376a466bd3ada1c2101b5b6ceac47bdfb6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 3 Mar 2026 11:37:50 +0200 Subject: [PATCH 795/832] pre-emptive version bump --- unpythonic/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 7a949b9e..6703e4b9 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '1.0.0' +__version__ = '1.0.1' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From 49baf99f4f41b51a5d68dcae87b4fd854d2691c8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 3 Mar 2026 11:41:44 +0200 Subject: [PATCH 796/832] report bugfix in changelog --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 09c1ee8f..c37d9e80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +**1.0.1** (March 2026, in progress) — *"Same supercharger, new badge"* edition: + +**Fixed**: + +- MS Windows: `unpythonic.net.util` failed to load, due to missing `termios` module (which is *nix only) being loaded by `unpythonic.net.__init__` when it imports `unpythonic.net.ptyproxy`. + - Fixed by catching `ModuleNotFoundError`, disabling `ptyproxy` on MS Windows systems. + - This means that the live REPL server is not available on MS Windows. This is usually harmless, as most applications using `unpythonic` do not need it. + + +--- + **1.0.0** (21 February 2026) — *"Same supercharger, new badge"* edition: Re-release of 0.15.5 as 1.0.0. No code changes. The library has been stable and in light maintenance mode for years; the version number now reflects this de facto status quo. From e91d3ddb8d0a367908cbd3bcfab5756d9e10e9d8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Tue, 3 Mar 2026 11:42:24 +0200 Subject: [PATCH 797/832] gah, fix version title in changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c37d9e80..61ecba88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -**1.0.1** (March 2026, in progress) — *"Same supercharger, new badge"* edition: +**1.0.1** (March 2026, in progress) — hotfix: **Fixed**: From 87d2076bec808634254e471b45d31202757ab66e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Wed, 4 Mar 2026 16:39:43 +0200 Subject: [PATCH 798/832] update CLAUDE.md --- CLAUDE.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 096edda5..c109c963 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,9 +10,13 @@ A Python library providing language extensions and utilities inspired by Lisp, H 2. **Macro layer** (`unpythonic/syntax/`): Syntactic macros via `mcpyrate` providing cleaner syntax for let-bindings, autocurry, lazify, TCO, continuations, etc. 3. **Dialect layer** (`unpythonic/dialects/`): Full language variants (Lispython, Listhell, Pytkell) built on the macro layer. +## API stability + +Released as 1.0.0 in February 2026, signalling API stability. The public API (everything in `__all__`) should remain backward-compatible. If backward-incompatible changes become necessary (e.g. due to Python 3.13/3.14 compat), they warrant a 2.0.0 release. Prefer non-breaking solutions when possible. + ## Build and development -Uses PDM with `pdm-backend`. Python 3.8–3.12, also PyPy 3.8–3.10. +Uses PDM with `pdm-backend`. Python 3.8–3.12, also PyPy 3.8–3.10. Version 3.13/3.14 compatibility update pending (will be released as 1.1.0). ```bash # Set up development environment @@ -25,7 +29,7 @@ source .venv/bin/activate ## Running tests -Custom test framework (not pytest). Tests use macros (`test[]`, `test_raises[]`) and conditions/restarts for reporting. The test runner does not need the `macropython` wrapper—it activates macros via `import mcpyrate.activate`. +Custom test framework (`unpythonic.test.fixtures`, not pytest). Tests use macros (`test[]`, `test_raises[]`) and conditions/restarts for reporting. The test runner does not need the `macropython` wrapper—it activates macros via `import mcpyrate.activate`. Note: test *framework* is at `unpythonic/test/` (singular); actual *tests* are in `tests/` (plural) subdirectories. ```bash # Run all tests (from repo root, with venv activated) @@ -67,7 +71,7 @@ flake8 . --config=flake8rc --exit-zero --max-line-length=127 - **Macros are the nuclear option**: Only make a macro when a regular function can't do the job. Prefer a pure-Python core with a thin macro layer for UX. - **Macro `**kw` passing**: Use `dyn` (dynamic variables) to pass `mcpyrate` `**kw` arguments through to syntax transformers, rather than threading them through parameter lists. - **Line width** ~110 characters. Docstrings in reStructuredText. -- **Module size target**: ~100–300 SLOC, rough max ~700 lines. +- **Module size target**: ~100–300 SLOC, rough max ~700 lines. Some modules are longer when appropriate (e.g. `syntax/tailtools.py` at ~1600 lines). Never split just because the line count was exceeded. - **Dependencies**: Avoid external dependencies. `mcpyrate` is the only allowed external dep and must remain strictly optional for the pure-Python layer. ## Key cross-cutting concerns From dd1b926f6bd5828194a60e428bb0a6f6e90463e8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 5 Mar 2026 10:46:36 +0200 Subject: [PATCH 799/832] fix .gitignore --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index ac831acf..24af6940 100644 --- a/.gitignore +++ b/.gitignore @@ -9,7 +9,7 @@ MANIFEST pdm.lock .pdm-python .spyproject -:venv +.venv *.egg-info *.mypy_cache .python-version From 703edd69aeb577126b66655a00a3729deac5729b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 10:26:07 +0200 Subject: [PATCH 800/832] Phase 2: bump floor to Python 3.10, version to 2.0.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop Python 3.8/3.9 support. Remove all dead version guards (18 sites): ast.Index wrappers (3.9), posonlyargs hasattr checks (3.8), CodeType positional construction (3.8), types.UnionType guard (3.10). Include posonlyargs directly in arguments() constructors. Remove walrus-inside-subscript parens (now 3.10+). Update TODO comments: parenthesis syntax for macro arguments is now deprecated; kept for backward compatibility. Update metadata: requires-python >=3.10,<3.15, mcpyrate dev dep, classifiers (drop 3.8/3.9, add 3.13/3.14, Production/Stable). CI matrix: 3.10–3.14 + pypy-3.11. Coverage on 3.12. Update CLAUDE.md, README.md, CONTRIBUTING.md. Co-Authored-By: Claude Opus 4.6 --- .github/workflows/coverage.yml | 2 +- .github/workflows/python-package.yml | 2 +- CLAUDE.md | 4 +- CONTRIBUTING.md | 2 +- README.md | 2 +- pyproject.toml | 10 ++--- unpythonic/__init__.py | 2 +- unpythonic/misc.py | 25 +---------- unpythonic/syntax/__init__.py | 3 +- unpythonic/syntax/lambdatools.py | 5 +-- unpythonic/syntax/letdo.py | 11 +---- unpythonic/syntax/letdoutil.py | 17 +++----- unpythonic/syntax/letsyntax.py | 12 ++---- unpythonic/syntax/nameutil.py | 9 +--- unpythonic/syntax/prefix.py | 7 +-- unpythonic/syntax/scopeanalyzer.py | 4 +- unpythonic/syntax/tailtools.py | 4 +- unpythonic/syntax/testingtools.py | 6 +-- .../syntax/tests/test_conts_multishot.py | 5 +-- unpythonic/syntax/tests/test_letdo.py | 14 +++--- unpythonic/syntax/tests/test_letdoutil.py | 43 +++++-------------- unpythonic/typecheck.py | 8 +--- 22 files changed, 52 insertions(+), 145 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index e8f27c59..e54e686d 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.10"] + python-version: ["3.12"] steps: - uses: actions/checkout@v6 diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 5b41d327..f6f59be3 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", pypy-3.8, pypy-3.9, pypy-3.10] + python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "pypy-3.11"] steps: - uses: actions/checkout@v6 diff --git a/CLAUDE.md b/CLAUDE.md index c109c963..c6165d3e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,11 +12,11 @@ A Python library providing language extensions and utilities inspired by Lisp, H ## API stability -Released as 1.0.0 in February 2026, signalling API stability. The public API (everything in `__all__`) should remain backward-compatible. If backward-incompatible changes become necessary (e.g. due to Python 3.13/3.14 compat), they warrant a 2.0.0 release. Prefer non-breaking solutions when possible. +Released as 2.0.0 in March 2026 (floor bump + mcpyrate 4.0.0 dependency). The public API (everything in `__all__`) should remain backward-compatible. Prefer non-breaking solutions when possible. ## Build and development -Uses PDM with `pdm-backend`. Python 3.8–3.12, also PyPy 3.8–3.10. Version 3.13/3.14 compatibility update pending (will be released as 1.1.0). +Uses PDM with `pdm-backend`. Python 3.10–3.14, also PyPy 3.11. ```bash # Set up development environment diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e23e569e..e791c09d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -118,7 +118,7 @@ The `lazify` and `continuations` macros are the most complex (and perhaps fearso `unpythonic.syntax.scopeanalyzer` is a unfortunate artifact that is needed to implement macros that interact with Python's scoping rules, notably `let`. Fortunately, [the language reference explicitly documents](https://docs.python.org/3/reference/executionmodel.html#naming-and-binding) what is needed for a lexical scope analysis for Python. So we have just implemented that (better, as an AST analysis, rather than scanning the surface syntax text). -As of the first half of 2021, the main target platforms are **CPython 3.8** and **PyPy3 3.7** (since as of April 2021, PyPy3 does not have 3.8 yet). The code should run on 3.6 or any later Python. We have [a GitHub workflow](https://github.com/Technologicat/unpythonic/actions?query=workflow%3A%22Python+package%22) that runs the test suite on CPython 3.6 through 3.9, and on PyPy3. +As of v2.0.0, the main target platforms are **CPython 3.10** through **3.14**, and **PyPy3** (language version 3.11). We have [a GitHub workflow](https://github.com/Technologicat/unpythonic/actions?query=workflow%3A%22Python+package%22) that runs the test suite on these platforms. ## Style guide diff --git a/README.md b/README.md index e373a3fb..f124a4ad 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ None required. - [`mcpyrate`](https://github.com/Technologicat/mcpyrate) optional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects. -As of v0.15.3, `unpythonic` runs on CPython 3.8, 3.9 and 3.10, 3.11, 3.12, and PyPy3 (language versions 3.8, 3.9, 3.10); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. New Python versions are added and old ones are removed following the [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). +As of v2.0.0, `unpythonic` runs on CPython 3.10, 3.11, 3.12, 3.13, 3.14, and PyPy3 (language version 3.11); the [CI](https://en.wikipedia.org/wiki/Continuous_integration) process verifies the tests pass on those platforms. New Python versions are added and old ones are removed following the [Long-term support roadmap](https://github.com/Technologicat/unpythonic/issues/1). ### Documentation diff --git a/pyproject.toml b/pyproject.toml index 21abd190..e6808283 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ description = "Supercharge your Python with parts of Lisp and Haskell." authors = [ { name = "Juha Jeronen", email = "juha.m.jeronen@gmail.com" }, ] -requires-python = ">=3.8,<3.13" +requires-python = ">=3.10,<3.15" # the `read` function and long_description_content_type from setup.py are no longer needed, # modern build tools like pdm/hatch already know how to handle markdown if you point them at a .md file @@ -19,25 +19,25 @@ license = { text = "BSD" } dynamic = ["version"] dependencies = [ - "mcpyrate>=3.6.4", + "mcpyrate @ file:///home/jje/Documents/koodit/mcpyrate", "sympy>=1.13" ] keywords=["functional-programming", "language-extension", "syntactic-macros", "tail-call-optimization", "tco", "continuations", "currying", "lazy-evaluation", "dynamic-variable", "macros", "lisp", "scheme", "racket", "haskell"] classifiers = [ - "Development Status :: 4 - Beta", + "Development Status :: 5 - Production/Stable", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Operating System :: POSIX :: Linux", "Programming Language :: Python", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.8", - "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Software Development :: Libraries", diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 6703e4b9..48e202e4 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '1.0.1' +__version__ = '2.0.0' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 diff --git a/unpythonic/misc.py b/unpythonic/misc.py index 2e765a77..b259598f 100644 --- a/unpythonic/misc.py +++ b/unpythonic/misc.py @@ -15,9 +15,8 @@ from itertools import count import inspect from queue import Empty -from sys import version_info from time import monotonic -from types import CodeType, FunctionType, LambdaType +from types import FunctionType, LambdaType from .regutil import register_decorator @@ -98,27 +97,7 @@ def rename(f): f.__name__ = name idx = f.__qualname__.rfind('.') f.__qualname__ = f"{f.__qualname__[:idx]}.{name}" if idx != -1 else name - # __code__.co_name is read-only, but there's a types.CodeType constructor - # that we can use to re-create the code object with the new name. - # (This is no worse than what the stdlib's Lib/modulefinder.py already does.) - co = f.__code__ - # https://github.com/ipython/ipython/blob/master/IPython/core/interactiveshell.py - # https://www.python.org/dev/peps/pep-0570/ - # https://docs.python.org/3/library/types.html#types.CodeType - # https://docs.python.org/3/library/inspect.html#types-and-members - if version_info >= (3, 8, 0): # Python 3.8+: positional-only parameters - # In Python 3.8+, `CodeType` has the convenient `replace()` method to functionally update it. - # In Python 3.10, we must actually use it to avoid losing the line number info, - # or `inspect.stack()` will crash in the unit tests for `callsite_filename()`. - f.__code__ = f.__code__.replace(co_name=name) - else: - f.__code__ = CodeType(co.co_argcount, co.co_kwonlyargcount, - co.co_nlocals, co.co_stacksize, co.co_flags, - co.co_code, co.co_consts, co.co_names, - co.co_varnames, co.co_filename, - name, - co.co_firstlineno, co.co_lnotab, co.co_freevars, - co.co_cellvars) + f.__code__ = f.__code__.replace(co_name=name) return f return rename diff --git a/unpythonic/syntax/__init__.py b/unpythonic/syntax/__init__.py index 4d3f494b..e4e9257d 100644 --- a/unpythonic/syntax/__init__.py +++ b/unpythonic/syntax/__init__.py @@ -81,7 +81,8 @@ # TODO: 0.16: AST pattern matching for `mcpyrate`? Would make destructuring easier. A writable representation (auto-viewify) is a pain to build, though... -# TODO: Far future: Change decorator macro invocations to use [] instead of () to pass macro arguments. Requires Python 3.9, so the earliest time to do this is when 3.9 becomes the minimum Python version for `unpythonic`. +# Parenthesis syntax for decorator macro arguments is deprecated; bracket syntax is preferred. +# Parenthesis syntax is kept for backward compatibility. from .autocurry import * # noqa: F401, F403 from .autoref import * # noqa: F401, F403 diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 4b12fd21..58ff6336 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -451,10 +451,7 @@ def _envify(block_body): # second pass, inside-out def getargs(tree): # tree: FunctionDef, AsyncFunctionDef, Lambda a = tree.args - if hasattr(a, "posonlyargs"): # Python 3.8+: positional-only parameters - allargs = a.posonlyargs + a.args + a.kwonlyargs - else: - allargs = a.args + a.kwonlyargs + allargs = a.posonlyargs + a.args + a.kwonlyargs argnames = [x.arg for x in allargs] if a.vararg: argnames.append(a.vararg.arg) diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 5bda9cf7..425363e7 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -27,7 +27,6 @@ AsyncFunctionDef, arguments, arg, Load) -import sys from mcpyrate.quotes import macros, q, u, n, a, t, h # noqa: F401 @@ -590,10 +589,8 @@ def _dletseq_impl(bindings, body, kind): userargs = body.args # original arguments to the def fname = body.name - noargs = arguments(args=[], kwonlyargs=[], vararg=None, kwarg=None, + noargs = arguments(args=[], posonlyargs=[], kwonlyargs=[], vararg=None, kwarg=None, defaults=[], kw_defaults=[]) - if sys.version_info >= (3, 8, 0): # Python 3.8+: positional-only arguments - noargs.posonlyargs = [] iname = gensym(f"{fname}_inner") body.args = noargs body.name = iname @@ -927,11 +924,7 @@ def _do0(tree): elts = tree.elts # Use `local[]` and `do[]` as hygienically captured macros. # - # Python 3.8 and Python 3.9 require the parens around the walrus when used inside a subscript. - # TODO: Remove the parens when we bump minimum Python to 3.10. - # From https://docs.python.org/3/whatsnew/3.10.html: - # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). - newelts = [q[a[_our_local][(_do0_result := a[elts[0]])]], # noqa: F821, local[] defines it inside the do[]. + newelts = [q[a[_our_local][_do0_result := a[elts[0]]]], # noqa: F821, local[] defines it inside the do[]. *elts[1:], q[_do0_result]] # noqa: F821 return q[a[_our_do][t[newelts]]] # do0[] is also just a do[] diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index d286d0e7..ac743c84 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -8,7 +8,6 @@ from ast import (Call, Name, Subscript, Compare, In, Tuple, List, Constant, BinOp, LShift, Lambda) -import sys from mcpyrate import unparse from mcpyrate.astcompat import getconstant, Str, NamedExpr @@ -22,14 +21,10 @@ def _get_subscript_slice(tree): assert type(tree) is Subscript - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - return tree.slice - return tree.slice.value + return tree.slice def _set_subscript_slice(tree, newslice): # newslice: AST assert type(tree) is Subscript - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - tree.slice = newslice - tree.slice.value = newslice + tree.slice = newslice def _canonize_macroargs_node(macroargs): # We do this like `mcpyrate.expander.destructure_candidate` does, # except that we also destructure a list. @@ -197,7 +192,7 @@ def islet(tree, expanded=True): s = tree.value.id if any(s == x for x in deconames): return ("decorator", s) - if type(tree) is Call and type(tree.func) is Name: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + if type(tree) is Call and type(tree.func) is Name: # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) s = tree.func.id if any(s == x for x in deconames): return ("decorator", s) @@ -214,7 +209,7 @@ def islet(tree, expanded=True): s = macro.value.id if any(s == x for x in exprnames): return ("lispy_expr", s) - elif type(macro) is Call and type(macro.func) is Name: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + elif type(macro) is Call and type(macro.func) is Name: # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) s = macro.func.id if any(s == x for x in exprnames): return ("lispy_expr", s) @@ -497,7 +492,7 @@ def _getbindings(self): # ^^^^^^^^^^ thetree = self._tree.value - if type(thetree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + if type(thetree) is Call: # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) return canonize_bindings(thetree.args) # Subscript theargs = _get_subscript_slice(thetree) @@ -526,7 +521,7 @@ def _setbindings(self, newbindings): # ^^^^^^^^^^ thetree = self._tree.value - if type(thetree) is Call: # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + if type(thetree) is Call: # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) thetree.args = newbindings return _set_subscript_slice(thetree, Tuple(elts=newbindings)) diff --git a/unpythonic/syntax/letsyntax.py b/unpythonic/syntax/letsyntax.py index 1a52807d..3ea73c3f 100644 --- a/unpythonic/syntax/letsyntax.py +++ b/unpythonic/syntax/letsyntax.py @@ -21,7 +21,6 @@ from ast import Name, Call, Subscript, Tuple, Starred, Expr, With from copy import deepcopy from functools import partial -import sys from mcpyrate import parametricmacro from mcpyrate.quotes import is_captured_value @@ -328,7 +327,7 @@ def isbinding(tree): if type(ctxmanager) is Subscript and type(ctxmanager.value) is Name and ctxmanager.value.id == mode: return mode, "template" # expr(...), block(...) - # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) if type(ctxmanager) is Call and type(ctxmanager.func) is Name and ctxmanager.func.id == mode: return mode, "template" return False @@ -369,10 +368,7 @@ def isbinding(tree): # ----------------------------------------------------------------------------- def _get_subscript_args(tree): - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - theslice = tree.slice - else: - theslice = tree.slice.value + theslice = tree.slice if type(theslice) is Tuple: args = theslice.elts else: @@ -389,7 +385,7 @@ def _analyze_lhs(tree): elif type(tree) is Subscript and type(tree.value) is Name: # template f[x, ...] name = tree.value.id args = [a.id for a in _get_subscript_args(tree)] - # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) elif type(tree) is Call and type(tree.func) is Name: # template f(x, ...) name = tree.func.id if any(type(a) is Starred for a in tree.args): # *args (Python 3.5+) @@ -445,7 +441,7 @@ def _substitute_templates(templates, tree): def isthisfunc(tree): if type(tree) is Subscript and type(tree.value) is Name and tree.value.id == name: return True - # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) if type(tree) is Call and type(tree.func) is Name and tree.func.id == name: return True return False diff --git a/unpythonic/syntax/nameutil.py b/unpythonic/syntax/nameutil.py index caf43993..1df61e98 100644 --- a/unpythonic/syntax/nameutil.py +++ b/unpythonic/syntax/nameutil.py @@ -9,7 +9,6 @@ "is_unexpanded_expr_macro", "is_unexpanded_block_macro"] from ast import Name, Attribute, Subscript, Call, With -import sys from mcpyrate.core import Done from mcpyrate.quotes import is_captured_macro, is_captured_value, lookup_macro @@ -124,11 +123,7 @@ def is_unexpanded_expr_macro(macrofunction, expander, tree): # extract the expr macro = expander.isbound(name_node.id) if macro is macrofunction: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - body = tree.slice - else: - body = tree.slice.value - return body + return tree.slice return False @@ -154,7 +149,7 @@ def is_unexpanded_block_macro(macrofunction, expander, tree): # discard args if any if type(maybemacro) is Subscript: maybemacro = maybemacro.value - # parenthesis syntax for macro arguments TODO: Python 3.9+: remove once we bump minimum Python to 3.9 + # Parenthesis syntax for macro arguments (deprecated; kept for backward compatibility) elif type(maybemacro) is Call: maybemacro = maybemacro.func diff --git a/unpythonic/syntax/prefix.py b/unpythonic/syntax/prefix.py index 671358f0..b5dcfc6a 100644 --- a/unpythonic/syntax/prefix.py +++ b/unpythonic/syntax/prefix.py @@ -7,7 +7,6 @@ __all__ = ["prefix", "q", "u", "kw"] from ast import Call, Starred, Tuple, Load, Subscript -import sys from mcpyrate.quotes import macros, q, u, a, t # noqa: F811, F401 @@ -194,11 +193,7 @@ def transform(self, tree): # Expr # Subscript if type(tree) is Subscript: - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - body = tree.slice - else: - body = tree.slice.value - + body = tree.slice if type(body) is Tuple: # Skip the transformation of the expr tuple itself, but transform its elements. # This skips the transformation of the macro argument tuple, too, because diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 27d7a8ce..a9c9827c 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -424,9 +424,7 @@ def extract_args(tree): if type(tree) not in (Lambda, FunctionDef, AsyncFunctionDef): raise ValueError(f"Expected a function definition AST node, got {tree}") a = tree.args - allargs = a.args + a.kwonlyargs - if hasattr(a, "posonlyargs"): # Python 3.8+: positional-only arguments - allargs += a.posonlyargs + allargs = a.posonlyargs + a.args + a.kwonlyargs argnames = [x.arg for x in allargs] if a.vararg: argnames.append(a.vararg.arg) diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index cb5f5e41..6221ac10 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -17,7 +17,6 @@ With, AsyncWith, If, IfExp, Try, Assign, Return, Expr, Await, copy_location) -import sys from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 @@ -1110,13 +1109,12 @@ def prepare_call(tree): body=q[n["cc"]], orelse=non) contarguments = arguments(args=[arg(arg=x) for x in targets], + posonlyargs=[], kwonlyargs=[arg(arg="cc"), arg(arg="_pcc")], vararg=(arg(arg=starget) if starget else None), kwarg=None, defaults=posargdefaults, kw_defaults=[q[h[identity]], maybe_capture]) - if sys.version_info >= (3, 8, 0): # Python 3.8+: positional-only arguments - contarguments.posonlyargs = [] funcdef = FDef(name=contname, args=contarguments, body=contbody, diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 73fcad01..9cef122c 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -20,7 +20,6 @@ from mcpyrate.walkers import ASTTransformer from ast import Tuple, Subscript, Name, Call, copy_location, Compare, arg, Return, parse, Expr, AST -import sys from ..dynassign import dyn from ..env import env @@ -880,10 +879,7 @@ def transform(self, tree): if isunexpandedtestmacro(tree): return tree elif _is_important_subexpr_mark(tree): - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - thing = tree.slice - else: - thing = tree.slice.value + thing = tree.slice self.collect(thing) # or anything really; value not used, we just count them. # Handle any nested the[] subexpressions subtree = self.visit(thing) diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index db77e591..c37bf15f 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -34,7 +34,6 @@ import ast from functools import partial - import sys from mcpyrate.quotes import macros, q, n, a, h # noqa: F811 from unpythonic.misc import safeissubclass @@ -191,9 +190,7 @@ def is_myield_name(node): def is_myield_expr(node): return type(node) is ast.Subscript and is_myield_name(node.value) def getslice(subscript_node): - if sys.version_info >= (3, 9, 0): # Python 3.9+: no ast.Index wrapper - return subscript_node.slice - return subscript_node.slice.value + return subscript_node.slice class MultishotYieldTransformer(ASTTransformer): def transform(self, tree): if is_captured_value(tree): # do not recurse into hygienic captures diff --git a/unpythonic/syntax/tests/test_letdo.py b/unpythonic/syntax/tests/test_letdo.py index 97ebf0ca..0ea8dd0b 100644 --- a/unpythonic/syntax/tests/test_letdo.py +++ b/unpythonic/syntax/tests/test_letdo.py @@ -1,8 +1,8 @@ # -*- coding: utf-8 -*- """Let constructs; do (imperative code in expression position).""" -# TODO: Update the @dlet, @dletseq, @dletrec, @blet, @bletseq, @bletrec examples -# TODO: to pass macro arguments using brackets once we bump to minimum Python 3.9. +# NOTE: Decorator macro arguments use parenthesis syntax in some examples below. +# Bracket syntax is preferred for new code; parenthesis syntax is deprecated but kept for backward compatibility. from ...syntax import macros, test, test_raises # noqa: F401 from ...test.fixtures import session, testset @@ -27,11 +27,7 @@ def runtests(): # - No need for ``lambda e: ...`` wrappers. Inserted automatically, # so the lines are only evaluated as the underlying seq.do() runs. # - # Python 3.8 and Python 3.9 require the parens around the walrus when used inside a subscript. - # TODO: Remove the parens (in all walrus-inside-subscript instances in this file) when we bump minimum Python to 3.10. - # From https://docs.python.org/3/whatsnew/3.10.html: - # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). - d1 = do[local[(x := 17)], + d1 = do[local[x := 17], print(x), (x := 23), x] @@ -42,7 +38,7 @@ def runtests(): # v0.14.0: do[] now supports deleting previously defined local names with delete[] a = 5 - d = do[local[(a := 17)], # noqa: F841, yes, d is unused. + d = do[local[a := 17], # noqa: F841, yes, d is unused. test[a == 17], delete[a], test[a == 5], # lexical scoping @@ -51,7 +47,7 @@ def runtests(): test_raises[KeyError, do[delete[a], ], "should have complained about deleting nonexistent local 'a'"] # do0[]: like do[], but return the value of the **first** expression - d2 = do0[local[(y := 5)], # noqa: F821, `local` defines the name on the LHS of the `<<`. + d2 = do0[local[y := 5], # noqa: F821, `local` defines the name on the LHS of the `<<`. print("hi there, y =", y), # noqa: F821 42] # evaluated but not used test[d2 == 5] diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index c6d2fc18..cd3c8b23 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -13,7 +13,6 @@ autocurry) from ast import Tuple, Name, Constant, Lambda, BinOp, Attribute, Call -import sys from mcpyrate import unparse @@ -38,10 +37,6 @@ def validate(lst): if type(k) is not Name: return False # pragma: no cover, only reached if the test fails. return True - # Python 3.8 and Python 3.9 require the parens around the walrus when used inside a subscript. - # TODO: Remove the parens (in all walrus-inside-subscript instances in this file) when we bump minimum Python to 3.10. - # From https://docs.python.org/3/whatsnew/3.10.html: - # Assignment expressions can now be used unparenthesized within set literals and set comprehensions, as well as in sequence indexes (but not slices). test[validate(the[canonize_bindings(q[k0, v0].elts)])] # noqa: F821, it's quoted. test[validate(the[canonize_bindings(q[((k0, v0),)].elts)])] # noqa: F821 test[validate(the[canonize_bindings(q[(k0, v0), (k1, v1)].elts)])] # noqa: F821 @@ -603,38 +598,29 @@ def f8(): # Destructuring - unexpanded do with testset("do destructuring (unexpanded) (new env-assign syntax v0.15.3+)"): - testdata = q[do[local[(x := 21)], # noqa: F821 + testdata = q[do[local[x := 21], # noqa: F821 2 * x]] # noqa: F821 view = UnexpandedDoView(testdata) # read thebody = view.body - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - thing = thebody[0].slice - else: - thing = thebody[0].slice.value + thing = thebody[0].slice test[isenvassign(the[thing])] # write # This mutates the original, but we have to assign `view.body` to trigger the setter. - thebody[0] = q[local[(x := 9001)]] # noqa: F821 + thebody[0] = q[local[x := 9001]] # noqa: F821 view.body = thebody # implicit do, a.k.a. extra bracket syntax - testdata = q[let[[local[(x := 21)], # noqa: F821 + testdata = q[let[[local[x := 21], # noqa: F821 2 * x]]] # noqa: F821 - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - theimplicitdo = testdata.slice - else: - theimplicitdo = testdata.slice.value + theimplicitdo = testdata.slice view = UnexpandedDoView(theimplicitdo) # read thebody = view.body - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - thing = thebody[0].slice - else: - thing = thebody[0].slice.value + thing = thebody[0].slice test[isenvassign(the[thing])] # write - thebody[0] = q[local[(x := 9001)]] # noqa: F821 + thebody[0] = q[local[x := 9001]] # noqa: F821 view.body = thebody test_raises[TypeError, @@ -647,10 +633,7 @@ def f8(): view = UnexpandedDoView(testdata) # read thebody = view.body - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - thing = thebody[0].slice - else: - thing = thebody[0].slice.value + thing = thebody[0].slice test[isenvassign(the[thing])] # write # This mutates the original, but we have to assign `view.body` to trigger the setter. @@ -660,17 +643,11 @@ def f8(): # implicit do, a.k.a. extra bracket syntax testdata = q[let[[local[x << 21], # noqa: F821 2 * x]]] # noqa: F821 - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - theimplicitdo = testdata.slice - else: - theimplicitdo = testdata.slice.value + theimplicitdo = testdata.slice view = UnexpandedDoView(theimplicitdo) # read thebody = view.body - if sys.version_info >= (3, 9, 0): # Python 3.9+: the Index wrapper is gone. - thing = thebody[0].slice - else: - thing = thebody[0].slice.value + thing = thebody[0].slice test[isenvassign(the[thing])] # write thebody[0] = q[local[x << 9001]] # noqa: F821 diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 228c9dd4..1c2f3542 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -15,7 +15,6 @@ """ import collections -import sys import typing _MyGenericAlias = typing._GenericAlias # Python 3.7+ @@ -183,12 +182,7 @@ def get_origin(tp): return False # pragma: no cover, Python 3.7+ only. def isNewType(T): - # In Python 3.10, an instance of `typing.NewType` is now actually such and not just a function. Nice! - if sys.version_info >= (3, 10, 0): - return isinstance(T, typing.NewType) - # Python 3.6, Python 3.7, Python 3.8, Python 3.9 - # TODO: in Python 3.7+, what is the mysterious callable that doesn't have a `__qualname__`? - return callable(T) and hasattr(T, "__qualname__") and T.__qualname__ == "NewType..new_type" + return isinstance(T, typing.NewType) if isNewType(T): # This is the best we can do, because the static types created by `typing.NewType` # have a constructor that discards the type information at runtime: From 1ae00ef520ac965466ca7736ab60c947320fd6f3 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 11:07:23 +0200 Subject: [PATCH 801/832] update CLAUDE.md: pyc cache warning, test result guide, naming conventions - Never py_compile macro-enabled code; macropython -c to fix stale caches - How to read test framework output (Pass/Fail/Error, nested testsets) - Variable naming: descriptive but compact; avoid the-prefixed names in test code using the[] macro - Use descriptive but compact variable names Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index c6165d3e..16ccda0a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,7 +25,7 @@ pdm use --venv in-project source .venv/bin/activate ``` -**Critical**: When installing from source, never use `--compile` / precompilation. Precompiled bytecode without macro support breaks macro imports like `from unpythonic.syntax import macros, let`. +**Critical**: Never compile `.py` files in this project using `py_compile`, `python -m compileall`, `--compile`, or any other mechanism that bypasses the macro expander. Stale `.pyc` files compiled without macro support will break macro imports (symptom: `ImportError: cannot import name 'macros' from 'mcpyrate.quotes'`). If this happens, clean the caches with `macropython -c unpythonic` and re-run. ## Running tests @@ -50,6 +50,8 @@ Test suites discovered by `runtests.py`: Each test module exports a `runtests()` function. Tests are grouped with `testset()` context managers. +**Reading test results**: The framework reports Pass/Fail/Error/Total per testset. "Error" means an unexpected exception inside a `test[]` expression — this includes intentional skip-with-message patterns (e.g. "SymPy not installed"), so a few errors from optional-dependency tests are normal. Look at the actual error messages, not just the count. Nested testsets show hierarchy with indentation and asterisk depth (`**`, `****`, `******`, etc.). + ## Linting ```bash @@ -70,6 +72,7 @@ flake8 . --config=flake8rc --exit-zero --max-line-length=127 - **Curry-friendly signatures**: Parameters that change least often go on the left. Use `def f(func, thing0, *things)` (not `def f(func, *things)`) when at least one `thing` is required, so `curry` knows when to trigger. - **Macros are the nuclear option**: Only make a macro when a regular function can't do the job. Prefer a pure-Python core with a thin macro layer for UX. - **Macro `**kw` passing**: Use `dyn` (dynamic variables) to pass `mcpyrate` `**kw` arguments through to syntax transformers, rather than threading them through parameter lists. +- **Variable names**: Descriptive but compact. Prefer `theconstant` over `node` when the type matters, `thebody` over `b` when scope is more than a few lines. Avoid generic names like `tmp`, `data`, `x` unless scope is trivially small. In test code using the `the[]` macro, avoid `the`-prefixed names — `the[theconstant]` isn't English. Use e.g. `constant_node` instead. - **Line width** ~110 characters. Docstrings in reStructuredText. - **Module size target**: ~100–300 SLOC, rough max ~700 lines. Some modules are longer when appropriate (e.g. `syntax/tailtools.py` at ~1600 lines). Never split just because the line count was exceeded. - **Dependencies**: Avoid external dependencies. `mcpyrate` is the only allowed external dep and must remain strictly optional for the pure-Python layer. From 4b92d7910a75b44a9b33a8ab5d294315147eaeec Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 11:07:31 +0200 Subject: [PATCH 802/832] Phase 3: adapt to mcpyrate 4.0.0 API changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace getconstant(node) with node.value at all 14 call sites. Remove Str, Num, NameConstant from imports — collapse type checks like type(x) in (Constant, Str) to type(x) is Constant. Remove entire mcpyrate.astcompat imports from autoref.py and util.py (no longer needed). Update error message in letdoutil.py to reference only ast.Constant. In test files, use intermediate variables (constant_node) to avoid .value.value chains. Co-Authored-By: Claude Opus 4.6 --- unpythonic/syntax/autoref.py | 3 +-- unpythonic/syntax/lambdatools.py | 6 +++--- unpythonic/syntax/letdoutil.py | 12 ++++++------ unpythonic/syntax/tailtools.py | 6 +++--- unpythonic/syntax/tests/test_letdoutil.py | 15 +++++++++------ unpythonic/syntax/tests/test_util.py | 16 +++++++++------- unpythonic/syntax/util.py | 19 +++++++------------ 7 files changed, 38 insertions(+), 39 deletions(-) diff --git a/unpythonic/syntax/autoref.py b/unpythonic/syntax/autoref.py index 739b22d1..0d4e9723 100644 --- a/unpythonic/syntax/autoref.py +++ b/unpythonic/syntax/autoref.py @@ -9,7 +9,6 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym, parametricmacro -from mcpyrate.astcompat import getconstant from mcpyrate.astfixers import fix_ctx from mcpyrate.quotes import is_captured_value from mcpyrate.walkers import ASTTransformer @@ -234,7 +233,7 @@ def transform(self, tree): elif isinstance(tree, ExpandedAutorefMarker): self.generic_withstate(tree, referents=referents + [tree.varname]) elif isautoreference(tree): # generated by an inner already expanded autoref block - thename = getconstant(get_resolver_list(tree)[-1]) + thename = get_resolver_list(tree)[-1].value if thename in referents: # This case is tricky to trigger, so let's document it here. This code: # diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 58ff6336..9e374a6c 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -14,7 +14,7 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym -from mcpyrate.astcompat import getconstant, Str, NamedExpr +from mcpyrate.astcompat import NamedExpr from mcpyrate.expander import MacroExpander from mcpyrate.quotes import is_captured_value from mcpyrate.splicing import splice_expression @@ -368,8 +368,8 @@ def transform(self, tree): if k is None: # {..., **d, ...} tree.values[j] = self.visit(v) else: - if type(k) in (Constant, Str): # Python 3.8+: ast.Constant - thename = getconstant(k) + if type(k) is Constant: + thename = k.value tree.values[j], thelambda, match = nameit(thename, v) if match: thelambda.body = self.visit(thelambda.body) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index ac743c84..c3f7838d 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -10,7 +10,7 @@ Tuple, List, Constant, BinOp, LShift, Lambda) from mcpyrate import unparse -from mcpyrate.astcompat import getconstant, Str, NamedExpr +from mcpyrate.astcompat import NamedExpr from mcpyrate.core import Done from .nameutil import isx, getname @@ -178,8 +178,8 @@ def islet(tree, expanded=True): elif not isx(tree.func, letf_name): return False mode = [kw.value for kw in tree.keywords if kw.arg == "mode"] - assert len(mode) == 1 and type(mode[0]) in (Constant, Str) - mode = getconstant(mode[0]) + assert len(mode) == 1 and type(mode[0]) is Constant + mode = mode[0].value kwnames = [kw.arg for kw in tree.keywords] if "_envname" in kwnames: return (f"{kind}_decorator", mode) # this call was generated by _let_decorator_impl @@ -723,8 +723,8 @@ def _setbindings(self, newbindings): raise NotImplementedError("changing the number of items currently not supported by this view (do that before the let[] expands)") # pragma: no cover for newb in newbindings.elts: newk, newv = newb.elts - if type(newk) not in (Constant, Str): # Python 3.8+: ast.Constant - raise TypeError("ExpandedLetView: let: each key must be an ast.Constant or an ast.Str") # pragma: no cover + if type(newk) is not Constant: + raise TypeError("ExpandedLetView: let: each key must be an ast.Constant") # pragma: no cover # Abstract away the namelambda(...). We support both "with autocurry" and bare formats: # currycall(letter, bindings, currycall(currycall(namelambda, "let_body"), curryf(lambda e: ...))) # letter(bindings, namelambda("let_body")(lambda e: ...)) @@ -736,7 +736,7 @@ def _setbindings(self, newbindings): for oldb, newb in zip(thebindings.elts, newbindings.elts): oldk, thev = oldb.elts newk, newv = newb.elts - newk_string = getconstant(newk) # Python 3.8+: ast.Constant + newk_string = newk.value if type(newv) is not Lambda: raise TypeError("ExpandedLetView: letrec: each value must be of the form `lambda e: ...`") # pragma: no cover if curried: diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 6221ac10..47b3f9fb 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -21,7 +21,7 @@ from mcpyrate.quotes import macros, q, u, n, a, h # noqa: F401 from mcpyrate import gensym -from mcpyrate.astcompat import getconstant, NameConstant, TryStar +from mcpyrate.astcompat import TryStar from mcpyrate.quotes import capture_as_macro, is_captured_value from mcpyrate.utils import NestingLevelTracker from mcpyrate.walkers import ASTTransformer, ASTVisitor @@ -1034,12 +1034,12 @@ def maybe_starred(expr): # return [expr.id] or set starget if not isinstance(stmt.value, CallCcMarker): # both Assign and Expr have a .value assert False # we should get only valid call_cc[] invocations that pass the `iscallccstatement` test # pragma: no cover theexpr = stmt.value.body # discard the AST marker - if not (type(theexpr) in (Call, IfExp) or (type(theexpr) in (Constant, NameConstant) and getconstant(theexpr) is None)): + if not (type(theexpr) in (Call, IfExp) or (type(theexpr) is Constant and theexpr.value is None)): raise SyntaxError("the bracketed expression in call_cc[...] must be a function call, an if-expression, or None") # pragma: no cover def extract_call(tree): if type(tree) is Call: return tree - elif type(tree) in (Constant, NameConstant) and getconstant(tree) is None: + elif type(tree) is Constant and tree.value is None: return None else: raise SyntaxError("call_cc[...]: expected a function call or None") # pragma: no cover diff --git a/unpythonic/syntax/tests/test_letdoutil.py b/unpythonic/syntax/tests/test_letdoutil.py index cd3c8b23..f30d844a 100644 --- a/unpythonic/syntax/tests/test_letdoutil.py +++ b/unpythonic/syntax/tests/test_letdoutil.py @@ -4,7 +4,6 @@ from ...syntax import macros, test, test_raises, warn, the # noqa: F401 from ...test.fixtures import session, testset -from mcpyrate.astcompat import getconstant, Num from mcpyrate.quotes import macros, q, n # noqa: F401, F811 from mcpyrate.metatools import macros, expandrq # noqa: F811 @@ -245,13 +244,15 @@ def f5(): # read test[view.name == "x"] - test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 42] # Python 3.8: ast.Constant + constant_node = view.value + test[type(the[constant_node]) is Constant and constant_node.value == 42] # write view.name = "y" view.value = q[23] test[view.name == "y"] - test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 23] # Python 3.8: ast.Constant + constant_node = view.value + test[type(the[constant_node]) is Constant and constant_node.value == 23] # it's a live view test[unparse(testdata) == "(y := 23)"] # syntax type `:=` vs. `<<` is preserved @@ -269,13 +270,15 @@ def f5(): # read test[view.name == "x"] - test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 42] # Python 3.8: ast.Constant + constant_node = view.value + test[type(the[constant_node]) is Constant and constant_node.value == 42] # write view.name = "y" view.value = q[23] test[view.name == "y"] - test[type(the[view.value]) in (Constant, Num) and getconstant(view.value) == 23] # Python 3.8: ast.Constant + constant_node = view.value + test[type(the[constant_node]) is Constant and constant_node.value == 23] # it's a live view test[unparse(testdata) == "(y << 23)"] # syntax type `:=` vs. `<<` is preserved @@ -520,7 +523,7 @@ def testbindings(*expected): test[the[unparse(bk)] == the[f"'{k}'"]] test[type(the[lam]) is Lambda] lambody = lam.body - test[type(the[lambody]) in (Constant, Num) and getconstant(lambody) == the[v]] # Python 3.8: ast.Constant + test[type(the[lambody]) is Constant and lambody.value == the[v]] # read test[len(view.bindings.elts) == 2] diff --git a/unpythonic/syntax/tests/test_util.py b/unpythonic/syntax/tests/test_util.py index 807c5da7..1ef3f7c8 100644 --- a/unpythonic/syntax/tests/test_util.py +++ b/unpythonic/syntax/tests/test_util.py @@ -4,7 +4,6 @@ from ...syntax import macros, do, local, test, test_raises, fail, the # noqa: F401 from ...test.fixtures import session, testset -from mcpyrate.astcompat import getconstant, Num, Str from mcpyrate.quotes import macros, q, n, h # noqa: F401, F811 from mcpyrate.metatools import macros, expandrq # noqa: F401, F811 @@ -156,8 +155,8 @@ def architectural(): test[len(decos) == 3] test[all(type(node) is Call and type(node.func) is Name for node in decos)] test[[node.func.id for node in decos] == ["memoize", "trampolined", "curry"]] - test[type(lam.body) in (Constant, Num)] # Python 3.8+: ast.Constant - test[getconstant(lam.body) == 42] # Python 3.8+: ast.Constant + test[type(lam.body) is Constant] + test[lam.body.value == 42] def test_sort_lambda_decorators(testdata): sort_lambda_decorators(testdata) @@ -185,15 +184,18 @@ def myfunction(x): "finally" collected = [] def collectstrings(tree): - if type(tree) is Expr and type(tree.value) in (Constant, Str): # Python 3.8+: ast.Constant - collected.append(getconstant(tree.value)) + if type(tree) is Expr and type(tree.value) is Constant: + constant_node = tree.value + collected.append(constant_node.value) return [tree] transform_statements(collectstrings, transform_statements_testdata) test[set(collected) == {"function body", "try", "if body", "if else", "finally", "except"}] def ishello(tree): - # Python 3.8+: ast.Constant - return type(tree) is Expr and type(tree.value) in (Constant, Str) and getconstant(tree.value) == "hello" + if type(tree) is Expr and type(tree.value) is Constant: + constant_node = tree.value + return constant_node.value == "hello" + return False # numeric with q as eliminate_ifones_testdata1: diff --git a/unpythonic/syntax/util.py b/unpythonic/syntax/util.py index 721b8b30..15cfa8ea 100644 --- a/unpythonic/syntax/util.py +++ b/unpythonic/syntax/util.py @@ -16,9 +16,8 @@ from functools import partial -from ast import Call, Lambda, FunctionDef, AsyncFunctionDef, If, stmt +from ast import Call, Constant, Lambda, FunctionDef, AsyncFunctionDef, If, stmt -from mcpyrate.astcompat import getconstant from mcpyrate.core import add_postprocessor from mcpyrate.markers import ASTMarker, delete_markers from mcpyrate.quotes import is_captured_value @@ -353,16 +352,12 @@ def eliminate_ifones(body): include a ``call_cc`` (see the example in test_conts_gen.py)... """ def isifone(tree): - if type(tree) is If: - try: - value = getconstant(tree.test) - except TypeError: - pass - else: - if value in (1, True): - return "then" - elif value in (0, False, None): - return "else" + if type(tree) is If and type(tree.test) is Constant: + value = tree.test.value + if value in (1, True): + return "then" + elif value in (0, False, None): + return "else" return False def optimize(tree): # stmt -> list of stmts From 33cfa203a39f078236e2563a302aed1781d90629 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 11:34:27 +0200 Subject: [PATCH 803/832] Phase 4: Python 3.13/3.14 compatibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix hasattr checks for 3.13+ AST field defaults: fields like ctx and lineno now always exist (defaulting to Load() or None) instead of being absent. Critical fixes in letdo.py (envify) and letdoutil.py (location propagation), plus test_conts_multishot.py. Cleanup fixes in scopeanalyzer.py, lambdatools.py, testingtools.py, dbg.py — use getattr with default instead of hasattr. Fix typing.Union detection on 3.14: replace local get_origin copy (broken on 3.14 where Union is no longer a _GenericAlias) with typing.get_origin. Add types.UnionType support for the X | Y syntax. Remove stale _MyGenericAlias and _MySupportsIndex aliases. Co-Authored-By: Claude Opus 4.6 --- unpythonic/syntax/dbg.py | 4 +- unpythonic/syntax/lambdatools.py | 5 +- unpythonic/syntax/letdo.py | 8 +- unpythonic/syntax/letdoutil.py | 2 +- unpythonic/syntax/scopeanalyzer.py | 4 +- unpythonic/syntax/testingtools.py | 8 +- .../syntax/tests/test_conts_multishot.py | 2 +- unpythonic/typecheck.py | 75 ++++--------------- 8 files changed, 29 insertions(+), 79 deletions(-) diff --git a/unpythonic/syntax/dbg.py b/unpythonic/syntax/dbg.py index 0eb10d72..68abff2c 100644 --- a/unpythonic/syntax/dbg.py +++ b/unpythonic/syntax/dbg.py @@ -226,7 +226,7 @@ def transform(self, tree): values = q[t[tree.args]] tree.args = [names, values] # can't use inspect.stack in the printer itself because we want the line number *before macro expansion*. - lineno = tree.lineno if hasattr(tree, "lineno") else None + lineno = getattr(tree, "lineno", None) # may be absent on 3.10–3.12; None on 3.13+ tree.keywords += [keyword(arg="filename", value=q[h[callsite_filename]()]), keyword(arg="lineno", value=q[u[lineno]])] tree.func = pfunc @@ -237,7 +237,7 @@ def _dbg_expr(tree): # TODO: Do we really need to expand inside-out here? tree = dyn._macro_expander.visit_recursively(tree) - ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] + ln = q[u[getattr(tree, "lineno", None)]] filename = q[h[callsite_filename]()] # Careful here! We must `h[]` the `dyn`, but not `dbgprint_expr` itself, # because we want to look up that attribute dynamically. diff --git a/unpythonic/syntax/lambdatools.py b/unpythonic/syntax/lambdatools.py index 9e374a6c..754767e0 100644 --- a/unpythonic/syntax/lambdatools.py +++ b/unpythonic/syntax/lambdatools.py @@ -531,9 +531,8 @@ def isourupdate(thecall): # because the gensymmed environment name won't be in our bindings, and the "x" # has become the `attr` in an `Attribute` node. elif type(tree) is Name and tree.id in bindings.keys(): - # We must be careful to preserve the Load/Store/Del context of the name. - # The default lets `mcpyrate` fix it later. - ctx = tree.ctx if hasattr(tree, "ctx") else None + # Preserve the Load/Store/Del context of the name. + ctx = getattr(tree, "ctx", None) out = deepcopy(bindings[tree.id]) out.ctx = ctx return out diff --git a/unpythonic/syntax/letdo.py b/unpythonic/syntax/letdo.py index 425363e7..42ea4dc5 100644 --- a/unpythonic/syntax/letdo.py +++ b/unpythonic/syntax/letdo.py @@ -26,7 +26,7 @@ FunctionDef, Return, AsyncFunctionDef, arguments, arg, - Load) + Store, Del) from mcpyrate.quotes import macros, q, u, n, a, t, h # noqa: F401 @@ -474,12 +474,10 @@ def transform(tree, names_in_scope): # in those parts of code where it is used, so an outer let will # leave it alone. if type(tree) is Name and tree.id in rhsnames and tree.id not in names_in_scope: - hasctx = hasattr(tree, "ctx") # Macro-created nodes might not have a ctx. - if hasctx and type(tree.ctx) is not Load: # Ignore assignments and deletes. + if type(getattr(tree, "ctx", None)) in (Store, Del): # Skip assignments and deletes. return tree attr_node = q[n[f"{envname}.{tree.id}"]] - if hasctx: - attr_node.ctx = tree.ctx + attr_node.ctx = getattr(tree, "ctx", None) return attr_node return tree return scoped_transform(tree, callback=transform) diff --git a/unpythonic/syntax/letdoutil.py b/unpythonic/syntax/letdoutil.py index c3f7838d..2af69541 100644 --- a/unpythonic/syntax/letdoutil.py +++ b/unpythonic/syntax/letdoutil.py @@ -755,7 +755,7 @@ def _setbindings(self, newbindings): # Macro-generated nodes may be missing source location information, # in which case we let `mcpyrate` fix it later. # This is mainly an issue for the unit tests of this module, which macro-generate the "old" data. - if hasattr(oldb, "lineno") and hasattr(oldb, "col_offset"): + if getattr(oldb, "lineno", None) is not None and getattr(oldb, "col_offset", None) is not None: newelts.append(Tuple(elts=[newk, thev], lineno=oldb.lineno, col_offset=oldb.col_offset)) else: newelts.append(Tuple(elts=[newk, thev])) diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index a9c9827c..249a8430 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -386,7 +386,7 @@ def examine(self, tree): # if item.optional_vars is not None: # self._collect_name_or_list(item.optional_vars) # macro-created nodes might not have a ctx, but our macros don't create lexical assignments. - if type(tree) is Name and hasattr(tree, "ctx") and type(tree.ctx) is Store: + if type(tree) is Name and type(getattr(tree, "ctx", None)) is Store: self.collect(tree.id) if not isnewscope(tree): self.generic_visit(tree) @@ -408,7 +408,7 @@ def examine(self, tree): # We don't currently care about "del myobj.x" or "del mydict['x']" (these old examples in Python 3.6): # Delete(targets=[Attribute(value=Name(id='myobj', ctx=Load()), attr='x', ctx=Del()),]) # Delete(targets=[Subscript(value=Name(id='mydict', ctx=Load()), slice=Index(value=Str(s='x')), ctx=Del()),]) - if type(tree) is Name and hasattr(tree, "ctx") and type(tree.ctx) is Del: + if type(tree) is Name and type(getattr(tree, "ctx", None)) is Del: self.collect(tree.id) if not isnewscope(tree): self.generic_visit(tree) diff --git a/unpythonic/syntax/testingtools.py b/unpythonic/syntax/testingtools.py index 9cef122c..e3d8cc0f 100644 --- a/unpythonic/syntax/testingtools.py +++ b/unpythonic/syntax/testingtools.py @@ -799,7 +799,7 @@ def _warn_expr(tree): def _test_expr(tree): # Note we want the line number *before macro expansion*, so we capture it now. - ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] + ln = q[u[getattr(tree, "lineno", None)]] # may be absent on 3.10–3.12; None on 3.13+ filename = q[h[callsite_filename]()] asserter = q[h[unpythonic_assert]] @@ -897,7 +897,7 @@ def _test_expr_raises(tree): return _test_expr_signals_or_raises(tree, "test_raises", q[h[unpythonic_assert_raises]]) def _test_expr_signals_or_raises(tree, syntaxname, asserter): - ln = q[u[tree.lineno]] if hasattr(tree, "lineno") else q[None] + ln = q[u[getattr(tree, "lineno", None)]] # may be absent on 3.10–3.12; None on 3.13+ filename = q[h[callsite_filename]()] # test_signals[exctype, expr, message] @@ -934,7 +934,7 @@ def _test_block(block_body, args): first_stmt = block_body[0] # Note we want the line number *before macro expansion*, so we capture it now. - ln = q[u[first_stmt.lineno]] if hasattr(first_stmt, "lineno") else q[None] + ln = q[u[getattr(first_stmt, "lineno", None)]] # may be absent on 3.10–3.12; None on 3.13+ filename = q[h[callsite_filename]()] asserter = q[h[unpythonic_assert]] @@ -1006,7 +1006,7 @@ def _test_block_signals_or_raises(block_body, args, syntaxname, asserter): first_stmt = block_body[0] # Note we want the line number *before macro expansion*, so we capture it now. - ln = q[u[first_stmt.lineno]] if hasattr(first_stmt, "lineno") else q[None] + ln = q[u[getattr(first_stmt, "lineno", None)]] # may be absent on 3.10–3.12; None on 3.13+ filename = q[h[callsite_filename]()] # with test_raises[exctype, message]: diff --git a/unpythonic/syntax/tests/test_conts_multishot.py b/unpythonic/syntax/tests/test_conts_multishot.py index c37bf15f..fc719a08 100644 --- a/unpythonic/syntax/tests/test_conts_multishot.py +++ b/unpythonic/syntax/tests/test_conts_multishot.py @@ -64,7 +64,7 @@ def myield_function(tree, syntax, **kw): # syntax error, because that `myield` is not inside a `@multishot` generator. # # We hack around it, by allowing `myield` anywhere as long as the context is not a `Load`. - if hasattr(tree, "ctx") and type(tree.ctx) is not ast.Load: + if type(getattr(tree, "ctx", None)) in (ast.Store, ast.Del): return tree # `myield` is not really a macro, but a pattern that `multishot` looks for and compiles away. diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 1c2f3542..bf5dbf09 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -15,11 +15,9 @@ """ import collections +import types import typing -_MyGenericAlias = typing._GenericAlias # Python 3.7+ -_MySupportsIndex = typing.SupportsIndex # Python 3.8+ - from .misc import safeissubclass __all__ = ["isoftype"] @@ -125,61 +123,16 @@ def isoftype(value, T): # in `unpythonic.dispatch`. And see: # https://docs.python.org/3/library/typing.html#typing.get_type_hints - # TODO: Python 3.8 adds `typing.get_origin` and `typing.get_args`: - # https://docs.python.org/3/library/typing.html#typing.get_origin - # TODO: We replicate them here so that we can use them in 3.7. - # TODO: Delete the local copies once we start requiring Python 3.8. - # - # Used under the PSF license. Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, - # 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Python Software Foundation; All Rights Reserved - # https://github.com/python/cpython/blob/3.8/LICENSE - def get_origin(tp): - """Get the unsubscripted version of a type. - This supports generic types, Callable, Tuple, Union, Literal, Final and ClassVar. - Return None for unsupported types. Examples:: - get_origin(Literal[42]) is Literal - get_origin(int) is None - get_origin(ClassVar[int]) is ClassVar - get_origin(Generic) is Generic - get_origin(Generic[T]) is Generic - get_origin(Union[T, int]) is Union - get_origin(List[Tuple[T, T]][int]) == list - """ - if isinstance(tp, _MyGenericAlias): - return tp.__origin__ - if tp is typing.Generic: - return typing.Generic - return None - # def get_args(tp): - # """Get type arguments with all substitutions performed. - # For unions, basic simplifications used by Union constructor are performed. - # Examples:: - # get_args(Dict[str, int]) == (str, int) - # get_args(int) == () - # get_args(Union[int, Union[T, int], str][int]) == (int, str) - # get_args(Union[int, Tuple[T, int]][str]) == (int, Tuple[str, int]) - # get_args(Callable[[], T][int]) == ([], int) - # """ - # if isinstance(tp, _MyGenericAlias) and not tp._special: - # res = tp.__args__ - # if get_origin(tp) is collections.abc.Callable and res[0] is not Ellipsis: - # res = (list(res[:-1]), res[-1]) - # return res - # return () - # <--- end of local copies of get_origin and get_args. The rest is our code. - - # Optional normalizes to Union[argtype, NoneType]. - # Python 3.6 has the repr, 3.7+ use typing._GenericAlias. - if repr(T.__class__) == "typing.Union" or get_origin(T) is typing.Union: - if T.__args__ is None: # Python 3.6 bare `typing.Union`; empty, has no types in it, so no value can match. - return False + # typing.Union[X, Y] and the builtin X | Y syntax (types.UnionType, Python 3.10+). + # Optional[X] normalizes to Union[X, NoneType]. + if typing.get_origin(T) is typing.Union or isinstance(T, types.UnionType): if not any(isoftype(value, U) for U in T.__args__): return False return True - # Python 3.7+ bare typing.Union; empty, has no types in it, so no value can match. - if T is typing.Union: # isinstance(T, typing._SpecialForm) and T._name == "Union": - return False # pragma: no cover, Python 3.7+ only. + # Bare typing.Union; empty, has no types in it, so no value can match. + if T is typing.Union: + return False # pragma: no cover def isNewType(T): return isinstance(T, typing.NewType) @@ -212,7 +165,7 @@ def isNewType(T): typing.SupportsFloat, typing.SupportsComplex, typing.SupportsBytes, - _MySupportsIndex, + typing.SupportsIndex, typing.SupportsAbs, typing.SupportsRound): if U is T: @@ -224,7 +177,7 @@ def isNewType(T): return isinstance(value, str) # alias for str # Subclass test for Python 3.6 only. Python 3.7+ have typing._GenericAlias for the generics. - if safeissubclass(T, typing.Tuple) or get_origin(T) is tuple: + if safeissubclass(T, typing.Tuple) or typing.get_origin(T) is tuple: if not isinstance(value, tuple): return False # bare `typing.Tuple`, no restrictions on length or element type. @@ -261,12 +214,12 @@ def ismapping(statictype, runtimetype): for statictype, runtimetype in ((typing.Dict, dict), (typing.MutableMapping, collections.abc.MutableMapping), (typing.Mapping, collections.abc.Mapping)): - if safeissubclass(T, statictype) or get_origin(T) is runtimetype: + if safeissubclass(T, statictype) or typing.get_origin(T) is runtimetype: return ismapping(statictype, runtimetype) # ItemsView is a special-case mapping in that we must not call # `.items()` on `value`. - if safeissubclass(T, typing.ItemsView) or get_origin(T) is collections.abc.ItemsView: + if safeissubclass(T, typing.ItemsView) or typing.get_origin(T) is collections.abc.ItemsView: if not isinstance(value, collections.abc.ItemsView): return False # Python 3.9: if a generic has no args, it has no `__args__` attribute. @@ -289,7 +242,7 @@ def ismapping(statictype, runtimetype): def iscollection(statictype, runtimetype): if not isinstance(value, runtimetype): return False - if safeissubclass(statictype, typing.ByteString) or get_origin(statictype) is collections.abc.ByteString: + if safeissubclass(statictype, typing.ByteString) or typing.get_origin(statictype) is collections.abc.ByteString: # WTF? A ByteString is a Sequence[int], but only statically. # At run time, the `__args__` are actually empty - it looks # like a bare Sequence, which is invalid. HACK the special case. @@ -324,10 +277,10 @@ def iscollection(statictype, runtimetype): (typing.MutableSequence, collections.abc.MutableSequence), (typing.MappingView, collections.abc.MappingView), (typing.Sequence, collections.abc.Sequence)): - if safeissubclass(T, statictype) or get_origin(T) is runtimetype: + if safeissubclass(T, statictype) or typing.get_origin(T) is runtimetype: return iscollection(statictype, runtimetype) - if safeissubclass(T, typing.Callable) or get_origin(T) is collections.abc.Callable: + if safeissubclass(T, typing.Callable) or typing.get_origin(T) is collections.abc.Callable: if not callable(value): return False return True From 92ad920d7930737671bcad9243fb78ba21bccaf8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 11:39:13 +0200 Subject: [PATCH 804/832] clean up typecheck.py: fix TypeVar detection, remove stale comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace fragile repr(T.__class__) string matching for TypeVar with isinstance(T, typing.TypeVar). Remove stale Python 3.6/3.7/3.9 comments and hasattr patterns — use getattr with defaults instead. Drop unused _MyGenericAlias alias. Resolves deferred issues D2, D3. Co-Authored-By: Claude Opus 4.6 --- unpythonic/typecheck.py | 89 ++++++++--------------------------------- 1 file changed, 17 insertions(+), 72 deletions(-) diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index bf5dbf09..0a598169 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -60,69 +60,27 @@ def isoftype(value, T): Returns `True` if `value` matches the type specification; `False` if not. """ - # TODO: This function is one big hack. + # Many `typing` meta-utilities explicitly raise TypeError from isinstance/issubclass, + # so we identify them via typing.get_origin, isinstance checks, or identity comparisons. + # We also access some internal fields (__args__, __constraints__, __supertype__) where + # Python provides no official public API for run-time type introspection. # - # As of Python 3.6, there seems to be no consistent way to identify a type - # specification at run time. So what we have is a mess. - # - # - Many `typing` meta-utilities explicitly `raise TypeError` when one - # attempts The One Obvious Way To Do It (`isinstance`, `issubclass`). - # - # - Their `type` can be something like `typing.TypeVar`, `typing.Union`, - # ``, ``... the - # format is case-dependent. A check like `type(T) is typing.TypeVar` - # doesn't work. - # - # So, we inspect `repr(T.__class__)` to match on the names of the prickly types, - # and call `issubclass` on those that don't hate us for doing so (catching - # `TypeError`, just in case `T` is an unsupported yet prickly type). - # - # Obviously, this won't work if someone subclasses one of the prickly types. - # `issubclass` would be The Right Thing, but since it's explicitly blocked, - # there's not much we can do. - - # TODO: Right now we're accessing internal fields to get what we need. - # TODO: Would be nice to rewrite this if Python, at some point, adds an - # TODO: official API to access the static type information at run time. + # Unsupported typing features: + # NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, + # IO, TextIO, BinaryIO, Pattern, Match, Generic, Type, + # Awaitable, Coroutine, AsyncIterable, AsyncIterator, + # ContextManager, AsyncContextManager, Generator, AsyncGenerator, + # NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef if T is typing.Any: return True # AnyStr normalizes to TypeVar("AnyStr", str, bytes) - # Python 3.6 has "typing.TypeVar" as the repr, but Python 3.7+ adds the "" around it. - if repr(T.__class__) == "typing.TypeVar" or repr(T.__class__) == "": + if isinstance(T, typing.TypeVar): if not T.__constraints__: # just an abstract type name return True return any(isoftype(value, U) for U in T.__constraints__) - # TODO: Here is THE FULL LIST of `typing` features we **don't** currently support, - # TODO: as of Python 3.8 (March 2020). https://docs.python.org/3/library/typing.html - # TODO: If you add a feature to the type checker, please update this list. - # - # TODO: Update this list for Python 3.9 - # TODO: Update this list for Python 3.10 - # TODO: Update this list for Python 3.11 - # TODO: Update this list for Python 3.12 - # - # Python 3.6+: - # NamedTuple, DefaultDict, Counter, ChainMap, - # IO, TextIO, BinaryIO, - # Pattern, Match, (regular expressions) - # Generic, Type, - # Awaitable, Coroutine, AsyncIterable, AsyncIterator, - # ContextManager, AsyncContextManager, - # Generator, AsyncGenerator, - # NoReturn (callable return value only), - # ClassVar, Final - # - # Python 3.7+: OrderedDict - # Python 3.8+: Protocol, TypedDict, Literal - # - # TODO: Do we need to support `typing.ForwardRef`? - # No, if `get_type_hints` already resolves that. Consider our main use case, - # in `unpythonic.dispatch`. And see: - # https://docs.python.org/3/library/typing.html#typing.get_type_hints - # typing.Union[X, Y] and the builtin X | Y syntax (types.UnionType, Python 3.10+). # Optional[X] normalizes to Union[X, NoneType]. if typing.get_origin(T) is typing.Union or isinstance(T, types.UnionType): @@ -155,9 +113,6 @@ def isNewType(T): return isinstance(value, U) if T is typing.Reversible: # can't non-destructively check element type - # We don't isinstance(), because in Python 3.5, typing.Reversible used to be just a protocol, - # and ": Protocols cannot be used with isinstance()." - # https://docs.python.org/3/library/collections.abc.html#module-collections.abc return hasattr(value, "__reversed__") # "Protocols cannot be used with isinstance()", so: @@ -176,13 +131,11 @@ def isNewType(T): if safeissubclass(T, typing.Text): # https://docs.python.org/3/library/typing.html#typing.Text return isinstance(value, str) # alias for str - # Subclass test for Python 3.6 only. Python 3.7+ have typing._GenericAlias for the generics. if safeissubclass(T, typing.Tuple) or typing.get_origin(T) is tuple: if not isinstance(value, tuple): return False # bare `typing.Tuple`, no restrictions on length or element type. - # Python 3.9: if a generic has no args, it has no `__args__` attribute. - if not hasattr(T, "__args__") or not T.__args__: + if not getattr(T, "__args__", None): return True # homogeneous element type, arbitrary length if len(T.__args__) == 2 and T.__args__[1] is Ellipsis: @@ -201,11 +154,9 @@ def isNewType(T): def ismapping(statictype, runtimetype): if not isinstance(value, runtimetype): return False - # Python 3.9: if a generic has no args, it has no `__args__` attribute. - if not hasattr(T, "__args__") or T.__args__ is None: # Python 3.6: consistent behavior with 3.7+, which use unconstrained TypeVar KT, VT. + args = getattr(T, "__args__", None) + if args is None: args = (typing.TypeVar("KT"), typing.TypeVar("VT")) - else: - args = T.__args__ assert len(args) == 2 if not value: # An empty dict has no key and value types. return False @@ -222,11 +173,9 @@ def ismapping(statictype, runtimetype): if safeissubclass(T, typing.ItemsView) or typing.get_origin(T) is collections.abc.ItemsView: if not isinstance(value, collections.abc.ItemsView): return False - # Python 3.9: if a generic has no args, it has no `__args__` attribute. - if not hasattr(T, "__args__") or T.__args__ is None: # Python 3.6: consistent behavior with 3.7+, which use unconstrained TypeVar KT, VT. + args = getattr(T, "__args__", None) + if args is None: args = (typing.TypeVar("KT"), typing.TypeVar("VT")) - else: - args = T.__args__ assert len(args) == 2 if not value: # An empty dict has no key and value types. return False @@ -247,12 +196,8 @@ def iscollection(statictype, runtimetype): # At run time, the `__args__` are actually empty - it looks # like a bare Sequence, which is invalid. HACK the special case. typeargs = (int,) - # Python 3.9: if a generic has no args, it has no `__args__` attribute. - elif hasattr(T, "__args__"): - typeargs = T.__args__ else: - typeargs = None - # Python 3.6: consistent behavior with 3.7+, which use an unconstrained TypeVar T. + typeargs = getattr(T, "__args__", None) if typeargs is None: typeargs = (typing.TypeVar("T"),) # Judging by the docs, List takes one type argument. The rest are similar. From 443a9c14d1b0cf9608f1cec7dc83eb748a1e86ca Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 11:41:30 +0200 Subject: [PATCH 805/832] typecheck.py: use isinstance for typing.Reversible The hasattr(__reversed__) check was a workaround for Python 3.5 where typing.Reversible was a protocol that rejected isinstance. Works fine on 3.10+. Co-Authored-By: Claude Opus 4.6 --- unpythonic/typecheck.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 0a598169..0e6becf0 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -113,7 +113,7 @@ def isNewType(T): return isinstance(value, U) if T is typing.Reversible: # can't non-destructively check element type - return hasattr(value, "__reversed__") + return isinstance(value, typing.Reversible) # "Protocols cannot be used with isinstance()", so: for U in (typing.SupportsInt, From fa847a8581c771cb12660712539db6549f9a6881 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 11:51:39 +0200 Subject: [PATCH 806/832] typecheck.py: remove redundant safeissubclass checks for generic types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On Python 3.10+, typing.get_origin handles both bare and parameterized generics (e.g. typing.Tuple → tuple, typing.Tuple[int, ...] → tuple), making the safeissubclass fallback unnecessary. Also removes the now-unused statictype parameter from ismapping. safeissubclass import retained for Supports* protocols and typing.Text. Co-Authored-By: Claude Opus 4.6 --- unpythonic/typecheck.py | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 0e6becf0..de58c805 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -131,7 +131,7 @@ def isNewType(T): if safeissubclass(T, typing.Text): # https://docs.python.org/3/library/typing.html#typing.Text return isinstance(value, str) # alias for str - if safeissubclass(T, typing.Tuple) or typing.get_origin(T) is tuple: + if typing.get_origin(T) is tuple: if not isinstance(value, tuple): return False # bare `typing.Tuple`, no restrictions on length or element type. @@ -151,7 +151,7 @@ def isNewType(T): return all(isoftype(elt, U) for elt, U in zip(value, T.__args__)) # Check mapping types that allow non-destructive iteration. - def ismapping(statictype, runtimetype): + def ismapping(runtimetype): if not isinstance(value, runtimetype): return False args = getattr(T, "__args__", None) @@ -162,15 +162,13 @@ def ismapping(statictype, runtimetype): return False K, V = args return all(isoftype(k, K) and isoftype(v, V) for k, v in value.items()) - for statictype, runtimetype in ((typing.Dict, dict), - (typing.MutableMapping, collections.abc.MutableMapping), - (typing.Mapping, collections.abc.Mapping)): - if safeissubclass(T, statictype) or typing.get_origin(T) is runtimetype: - return ismapping(statictype, runtimetype) + for runtimetype in (dict, collections.abc.MutableMapping, collections.abc.Mapping): + if typing.get_origin(T) is runtimetype: + return ismapping(runtimetype) # ItemsView is a special-case mapping in that we must not call # `.items()` on `value`. - if safeissubclass(T, typing.ItemsView) or typing.get_origin(T) is collections.abc.ItemsView: + if typing.get_origin(T) is collections.abc.ItemsView: if not isinstance(value, collections.abc.ItemsView): return False args = getattr(T, "__args__", None) @@ -191,7 +189,7 @@ def ismapping(statictype, runtimetype): def iscollection(statictype, runtimetype): if not isinstance(value, runtimetype): return False - if safeissubclass(statictype, typing.ByteString) or typing.get_origin(statictype) is collections.abc.ByteString: + if typing.get_origin(statictype) is collections.abc.ByteString: # WTF? A ByteString is a Sequence[int], but only statically. # At run time, the `__args__` are actually empty - it looks # like a bare Sequence, which is invalid. HACK the special case. @@ -222,10 +220,10 @@ def iscollection(statictype, runtimetype): (typing.MutableSequence, collections.abc.MutableSequence), (typing.MappingView, collections.abc.MappingView), (typing.Sequence, collections.abc.Sequence)): - if safeissubclass(T, statictype) or typing.get_origin(T) is runtimetype: + if typing.get_origin(T) is runtimetype: return iscollection(statictype, runtimetype) - if safeissubclass(T, typing.Callable) or typing.get_origin(T) is collections.abc.Callable: + if typing.get_origin(T) is collections.abc.Callable: if not callable(value): return False return True From ea83d4b463839e5ee386deca234c33764d55086b Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 12:17:45 +0200 Subject: [PATCH 807/832] Phase 5: autoreturn match/case, scopeanalyzer bugfix, version-gated tests - autoreturn macro now handles match/case: each case branch has its own tail position. - Fixed MatchCapturesCollector bug in scopeanalyzer: it collected class references (e.g. Point) as captures instead of actual MatchAs/MatchStar captures. Removed the collector; generic_visit + existing MatchAs/MatchStar handling covers everything except MatchMapping.rest. - Test runner supports version-suffixed modules (test_foo_3_11.py skipped on Python < 3.11). - New tests: autoreturn match/case, scopeanalyzer match/case patterns, scopeanalyzer try/except* (version-gated to 3.11+). - Changelog for 2.0.0, AUTHORS.md updated. Co-Authored-By: Claude Opus 4.6 --- AUTHORS.md | 2 +- CHANGELOG.md | 28 ++++++- TODO_DEFERRED.md | 7 ++ runtests.py | 23 +++++- unpythonic/syntax/scopeanalyzer.py | 36 +++------ unpythonic/syntax/tailtools.py | 6 +- unpythonic/syntax/tests/test_autoret.py | 38 +++++++++ unpythonic/syntax/tests/test_scopeanalyzer.py | 79 ++++++++++++++++++- .../syntax/tests/test_scopeanalyzer_3_11.py | 49 ++++++++++++ 9 files changed, 235 insertions(+), 33 deletions(-) create mode 100644 TODO_DEFERRED.md create mode 100644 unpythonic/syntax/tests/test_scopeanalyzer_3_11.py diff --git a/AUTHORS.md b/AUTHORS.md index e0405378..1853b38a 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -2,7 +2,7 @@ - Juha Jeronen (@Technologicat) - original author - @aisha-w - documentation improvements -- @Technologicat with Claude (Anthropic) as AI pair programmer - CI modernization +- @Technologicat with Claude (Anthropic) as AI pair programmer - CI modernization, Python 3.13–3.14 and mcpyrate 4.0.0 adaptation (2.0.0) **Design inspiration from the internet**: diff --git a/CHANGELOG.md b/CHANGELOG.md index 61ecba88..c765fb14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,12 +1,36 @@ # Changelog -**1.0.1** (March 2026, in progress) — hotfix: +**2.0.0** (March 2026, in progress) — *"Six impossible things before breakfast"* edition: + +**IMPORTANT**: + +- **Python version support**: 3.10–3.14 (dropped 3.8, 3.9; added 3.13, 3.14). PyPy 3.11. + - If you need `unpythonic` for Python 3.8 or 3.9, use version 1.0.0. +- **Requires mcpyrate >= 4.0.0**. + - mcpyrate 4.0.0 dropped the `Str`, `Num`, `NameConstant` AST compatibility shims and the `getconstant` helper. + +**New**: + +- **Python 3.13 and 3.14 support**. +- `autoreturn` macro now handles `match`/`case` statements. Each case branch has its own tail position. +- New scope analyzer tests for `match`/`case` patterns and `try`/`except*`. +- Test runner (`runtests.py`) now supports version-suffixed test modules (e.g. `test_foo_3_11.py` runs only on Python 3.11+). **Fixed**: +- Runtime type checker (`unpythonic.typecheck`): fixed compatibility with Python 3.14, where `typing.Union` is no longer a `_GenericAlias`. Now uses `typing.get_origin` (available since 3.8) instead of a local copy. +- Runtime type checker: fixed `TypeVar` detection to use `isinstance(T, typing.TypeVar)` instead of a fragile `repr`-based heuristic. +- Runtime type checker: `typing.Reversible` check now uses `isinstance` instead of a `hasattr("__reversed__")` workaround from the Python 3.5 era. +- Runtime type checker: removed redundant `safeissubclass` fallbacks for generic types — `typing.get_origin` handles both bare and parameterized generics on 3.10+. +- Scope analyzer: fixed `MatchCapturesCollector` bug where class references (e.g. `Point` in `case Point(x, y):`) were incorrectly collected as captured variable names. Match captures are `MatchAs`/`MatchStar` nodes with bare strings, not `Name` nodes. +- Macro layer: updated all `hasattr(tree, "ctx")` checks to use `getattr` with defaults, for correct behavior on Python 3.13+ where AST fields always exist with default values. +- Macro layer: updated `arguments()` constructor calls to always include `posonlyargs=[]`, avoiding a `DeprecationWarning` on Python 3.13 (will become an error in 3.15). - MS Windows: `unpythonic.net.util` failed to load, due to missing `termios` module (which is *nix only) being loaded by `unpythonic.net.__init__` when it imports `unpythonic.net.ptyproxy`. - Fixed by catching `ModuleNotFoundError`, disabling `ptyproxy` on MS Windows systems. - - This means that the live REPL server is not available on MS Windows. This is usually harmless, as most applications using `unpythonic` do not need it. + +**Deprecated**: + +- Parenthesis syntax for macro arguments (e.g. `let((x, 1), (y, 2))`). Use bracket syntax instead: `let[[x, 1], [y, 2]]`. The parenthesis syntax is kept for backward compatibility but may be removed in a future version. --- diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md new file mode 100644 index 00000000..4999688d --- /dev/null +++ b/TODO_DEFERRED.md @@ -0,0 +1,7 @@ +# Deferred Issues + +- **D1**: Document pyc cache pitfall and test result reading for other projects using mcpyrate/unpythonic.test.fixtures. The CLAUDE.md additions from the 2.0.0 modernization (never `py_compile` macro-enabled code; how to read test framework output with Pass/Fail/Error) are useful guidance for any project using these tools. Consider adding similar notes to mcpyrate's docs and/or unpythonic's user-facing documentation. (Discovered during Phase 3.) + +- **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features: NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, IO/TextIO/BinaryIO, Pattern/Match, Generic, Type, Awaitable, Coroutine, AsyncIterable, AsyncIterator, ContextManager, AsyncContextManager, Generator, AsyncGenerator, NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef. Would improve `unpythonic.dispatch` (multiple dispatch). (Discovered during Phase 4 cleanup.) + +- **D5**: `runtests.py` — version-suffix skip should signal `TestWarning` (via `unpythonic.conditions.signal`) instead of printing and continuing. This would make skips visible in the testset warning count, consistent with how optional dependency failures show as errors. Currently the skip message bypasses the testset reporting mechanism. (Discovered during Phase 5.) diff --git a/runtests.py b/runtests.py index e928983e..3f96c998 100644 --- a/runtests.py +++ b/runtests.py @@ -11,9 +11,12 @@ import sys from importlib import import_module -from unpythonic.test.fixtures import session, testset, tests_errored, tests_failed +from unpythonic.test.fixtures import (session, testset, maybe_colorize, + tests_errored, tests_failed, TestConfig) from unpythonic.collections import unbox +from mcpyrate.colorizer import Style + import mcpyrate.activate # noqa: F401 def listtestmodules(path): @@ -29,6 +32,17 @@ def modname(path, filename): # some/dir/mod.py --> some.dir.mod themod = re.sub(r"\.py$", r"", filename) return ".".join([modpath, themod]) +def _version_suffix(modulename): + """Parse version suffix from module name. + + E.g. ``unpythonic.syntax.tests.test_scopeanalyzer_3_11`` → ``(3, 11)``, or ``None``. + """ + # Match the final component of a dotted module name. + m = re.search(r"_(\d+)_(\d+)$", modulename) + if m: + return (int(m.group(1)), int(m.group(2))) + return None + def main(): with session(): # All folders containing unit tests are named `tests` (plural). @@ -46,6 +60,13 @@ def main(): # Wrap each module in its own testset to protect the umbrella testset # against ImportError as well as any failures at macro expansion time. with testset(m): + ver = _version_suffix(m) + if ver is not None and sys.version_info < ver: + msg = (f"Skipping '{m}' (requires Python {ver[0]}.{ver[1]}+, " + f"running {sys.version_info.major}.{sys.version_info.minor})") + TestConfig.printer(maybe_colorize(msg, Style.DIM, + TestConfig.ColorScheme.HEADING)) + continue # TODO: We're not inside a package, so we currently can't use a relative import. # TODO: So we just hope this resolves to the local `unpythonic` source code, # TODO: not to an installed copy of the library. diff --git a/unpythonic/syntax/scopeanalyzer.py b/unpythonic/syntax/scopeanalyzer.py index 249a8430..6aaa7463 100644 --- a/unpythonic/syntax/scopeanalyzer.py +++ b/unpythonic/syntax/scopeanalyzer.py @@ -80,7 +80,7 @@ Import, ImportFrom, Try, ListComp, SetComp, GeneratorExp, DictComp, Store, Del, Global, Nonlocal) -from mcpyrate.astcompat import TryStar, MatchStar, MatchMapping, MatchClass, MatchAs +from mcpyrate.astcompat import TryStar, MatchStar, MatchMapping, MatchAs from mcpyrate.core import Done from mcpyrate.walkers import ASTTransformer, ASTVisitor @@ -316,12 +316,6 @@ def get_names_in_store_context(tree): by ``get_lexical_variables`` for the nearest lexically surrounding parent tree that represents a scope. """ - class MatchCapturesCollector(ASTVisitor): # Python 3.10+: `match`/`case` - def examine(self, tree): - if type(tree) is Name: - self.collect(tree.id) - self.generic_visit(tree) - class StoreNamesCollector(ASTVisitor): # def _collect_name_or_list(self, t): # if type(t) is Name: @@ -355,28 +349,20 @@ def examine(self, tree): # TODO: `try`, even inside the `except` blocks, will be bound in the whole parent scope. for h in tree.handlers: self.collect(h.name) - # Python 3.10+: `match`/`case` uses names in `Load` context to denote captures. - # Also there are some bare strings, and sometimes `None` actually means "_" (but doesn't capture). - # So we special-case all of this. - elif type(tree) in (MatchAs, MatchStar): # a `MatchSequence` also consists of these + # Python 3.10+: `match`/`case` captures are `MatchAs(name='x')` and + # `MatchStar(name='rest')` with bare strings (not `Name` nodes). The `name` + # is `None` for `_` (wildcard, doesn't capture). `Name` nodes in patterns are + # class references (e.g. `Point` in `case Point(x, y):`), not captures. + # + # `generic_visit` handles most match patterns automatically, since `MatchAs` + # and `MatchStar` nodes appear as children. The one exception is + # `MatchMapping.rest`, which is a bare string attribute (not an AST child). + elif type(tree) in (MatchAs, MatchStar): if tree.name is not None: self.collect(tree.name) elif type(tree) is MatchMapping: - mcc = MatchCapturesCollector(tree.patterns) - mcc.visit() - for name in mcc.collected: - self.collect(name) - if tree.rest is not None: # `rest` is a capture if present + if tree.rest is not None: # `**rest` capture self.collect(tree.rest) - elif type(tree) is MatchClass: - mcc = MatchCapturesCollector(tree.patterns) - mcc.visit() - for name in mcc.collected: - self.collect(name) - mcc = MatchCapturesCollector(tree.kwd_patterns) - mcc.visit() - for name in mcc.collected: - self.collect(name) # Python 3.12+: `TypeAlias` uses a name in `Store` context on its LHS so it needs no special handling here. diff --git a/unpythonic/syntax/tailtools.py b/unpythonic/syntax/tailtools.py index 47b3f9fb..7a1d6742 100644 --- a/unpythonic/syntax/tailtools.py +++ b/unpythonic/syntax/tailtools.py @@ -14,7 +14,7 @@ List, Tuple, Call, Name, Starred, Constant, BoolOp, And, Or, - With, AsyncWith, If, IfExp, Try, Assign, Return, Expr, + With, AsyncWith, If, IfExp, Try, Match, Assign, Return, Expr, Await, copy_location) @@ -699,6 +699,10 @@ def transform(self, tree): # additionally, tail position is in each "except" handler for handler in tree.handlers: handler.body[-1] = self.visit(handler.body[-1]) + elif type(tree) is Match: # Python 3.10+: `match`/`case` + for case in tree.cases: + if case.body: + case.body[-1] = self.visit(case.body[-1]) elif type(tree) in (FunctionDef, AsyncFunctionDef, ClassDef): # v0.15.0+ # If the item in tail position is a named function definition # or a class definition, it binds a name - that of the function/class. diff --git a/unpythonic/syntax/tests/test_autoret.py b/unpythonic/syntax/tests/test_autoret.py index d0baa492..005e9654 100644 --- a/unpythonic/syntax/tests/test_autoret.py +++ b/unpythonic/syntax/tests/test_autoret.py @@ -87,6 +87,44 @@ class InnerClassDefinition: test[isinstance(classdefiner(), type)] # returned a class test[classdefiner().__name__ == "InnerClassDefinition"] + with testset("match/case"): # Python 3.10+ + with autoreturn: + def classify(x): + match x: + case 1: + "one" + case 2: + "two" + case _: + "other" + test[classify(1) == "one"] + test[classify(2) == "two"] + test[classify(42) == "other"] + + def classify_nested(x): + match x: + case (a, b): + a + b + case [a, b, *rest]: + a + b + sum(rest) + case _: + 0 + test[classify_nested((3, 4)) == 7] + test[classify_nested([1, 2, 3, 4]) == 10] + test[classify_nested("nope") == 0] + + def classify_with_guard(x): + match x: + case n if n < 0: + "negative" + case 0: + "zero" + case n if n > 0: + "positive" + test[classify_with_guard(-5) == "negative"] + test[classify_with_guard(0) == "zero"] + test[classify_with_guard(7) == "positive"] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/syntax/tests/test_scopeanalyzer.py b/unpythonic/syntax/tests/test_scopeanalyzer.py index a41a64ef..16cf0995 100644 --- a/unpythonic/syntax/tests/test_scopeanalyzer.py +++ b/unpythonic/syntax/tests/test_scopeanalyzer.py @@ -14,9 +14,6 @@ get_lexical_variables, scoped_transform) -# TODO: Add tests for `match`/`case` once we bump minimum language version to Python 3.10. -# TODO: Add tests for `try`/`except*` once we bump minimum language version to Python 3.11. - def runtests(): # test data with q as getnames_load: @@ -272,6 +269,82 @@ def f(): # noqa: F811 n["_apply_test_here_"] scoped_transform(scoped_localvar3, callback=make_checker(["f"])) # x already deleted + # Python 3.10+: `match`/`case` + with testset("match/case: get_names_in_store_context"): + # Simple capture + with q as matchcase_simple: + match x: # noqa: F821, it's only quoted. + case y: # noqa: F841, it's only quoted. + pass + test[get_names_in_store_context(matchcase_simple) == ["y"]] + + # Wildcard `_` — does NOT capture + with q as matchcase_wildcard: + match x: # noqa: F821, it's only quoted. + case _: + pass + test[get_names_in_store_context(matchcase_wildcard) == []] + + # Sequence pattern with star capture + with q as matchcase_sequence: + match x: # noqa: F821, it's only quoted. + case [a, b, *rest]: # noqa: F841, it's only quoted. + pass + test[get_names_in_store_context(matchcase_sequence) == ["a", "b", "rest"]] + + # Class pattern — captures `x` and `y`, but NOT the class reference `Point` + with q as matchcase_class: + match x: # noqa: F821, it's only quoted. + case Point(x, y): # noqa: F821, F841, it's only quoted. + pass + names = get_names_in_store_context(matchcase_class) + test["x" in names] + test["y" in names] + test["Point" not in names] # class reference, not a capture + + # Class pattern with keyword captures + with q as matchcase_class_kw: + match x: # noqa: F821, it's only quoted. + case Point(x=px, y=py): # noqa: F821, F841, it's only quoted. + pass + names = get_names_in_store_context(matchcase_class_kw) + test["px" in names] + test["py" in names] + test["Point" not in names] + + # Mapping pattern with `**rest` + with q as matchcase_mapping: + match x: # noqa: F821, it's only quoted. + case {"key": value, **rest}: # noqa: F841, it's only quoted. + pass + names = get_names_in_store_context(matchcase_mapping) + test["value" in names] + test["rest" in names] + + # Nested: mapping containing a class pattern + with q as matchcase_nested: + match x: # noqa: F821, it's only quoted. + case {"key": Point(px, py)}: # noqa: F821, F841, it's only quoted. + pass + names = get_names_in_store_context(matchcase_nested) + test["px" in names] + test["py" in names] + test["Point" not in names] # class reference, not a capture + + # OR pattern + with q as matchcase_or: + match x: # noqa: F821, it's only quoted. + case 1 | 2 | 3: + pass + test[get_names_in_store_context(matchcase_or) == []] + + # `as` pattern with guard + with q as matchcase_as: + match x: # noqa: F821, it's only quoted. + case (1 | 2) as num: # noqa: F841, it's only quoted. + pass + test[get_names_in_store_context(matchcase_as) == ["num"]] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/syntax/tests/test_scopeanalyzer_3_11.py b/unpythonic/syntax/tests/test_scopeanalyzer_3_11.py new file mode 100644 index 00000000..bd436b58 --- /dev/null +++ b/unpythonic/syntax/tests/test_scopeanalyzer_3_11.py @@ -0,0 +1,49 @@ +# -*- coding: utf-8 -*- +"""Lexical scope analysis tools — try/except* tests. + +These tests require Python 3.11+ because the ``except*`` syntax +won't parse on earlier versions. + +TODO: Merge into test_scopeanalyzer.py when floor bumps to Python 3.11+. +""" + +from ...syntax import macros, test, test_raises, the # noqa: F401 +from ...test.fixtures import session, testset + +from mcpyrate.quotes import macros, q # noqa: F401, F811 + +from ...syntax.scopeanalyzer import get_names_in_store_context + +def runtests(): + with testset("try/except*: get_names_in_store_context"): + # except* binds names just like except + with q as exceptstar_simple: + try: + pass + except* ValueError as eg: # noqa: F841, it's only quoted. + pass + test[get_names_in_store_context(exceptstar_simple) == ["eg"]] + + with q as exceptstar_multi: + try: + pass + except* ValueError as eg1: # noqa: F841, it's only quoted. + pass + except* TypeError as eg2: # noqa: F841, it's only quoted. + pass + test[get_names_in_store_context(exceptstar_multi) == ["eg1", "eg2"]] + + # Names bound inside the try body are also collected + with q as exceptstar_with_assign: + try: + x = 42 # noqa: F841, it's only quoted. + except* ValueError as eg: # noqa: F841, it's only quoted. + y = 1 # noqa: F841, it's only quoted. + names = get_names_in_store_context(exceptstar_with_assign) + test["x" in the[names]] + test["y" in the[names]] + test["eg" in the[names]] + +if __name__ == '__main__': # pragma: no cover + with session(__file__): + runtests() From bd9ab536887d308620f7c99518dd426df4f477f1 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 12:53:17 +0200 Subject: [PATCH 808/832] use warn[] instead of error[] for missing optional dependencies in tests Missing optional dependencies (sympy, mpmath) are expected in some environments, not failures. Using warn[] makes them show as warnings in the test report rather than errors, consistent with the semantic that the test is intentionally skipped. Updated documentation examples (fixtures.py docstring, README.md, doc/macros.md) to recommend the same pattern. Co-Authored-By: Claude Opus 4.6 --- README.md | 6 +++++- doc/macros.md | 6 +++++- unpythonic/syntax/tests/test_nb.py | 4 ++-- unpythonic/test/fixtures.py | 7 ++++--- unpythonic/tests/test_mathseq.py | 8 ++++---- unpythonic/tests/test_numutil.py | 4 ++-- 6 files changed, 22 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index f124a4ad..ad755d9c 100644 --- a/README.md +++ b/README.md @@ -574,10 +574,14 @@ with session("simple framework demo"): try: import blargly except ImportError: - error["blargly not installed, cannot test integration with it."] + warn["blargly not installed, skipping integration tests."] else: ... # blargly integration tests go here + # Unconditional errors and failures can be emitted with `error[]` and `fail[]`. + # with testset("not implemented"): + # fail["not implemented yet!"] + with testset(postproc=terminate): test[2 * 2 == 5] # fails, terminating the nearest dynamically enclosing `with session` test[2 * 2 == 4] # not reached diff --git a/doc/macros.md b/doc/macros.md index 8fc70043..ee4c31a0 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -1997,10 +1997,14 @@ with session("simple framework demo"): try: import blargly except ImportError: - error["blargly not installed, cannot test integration with it."] + warn["blargly not installed, skipping integration tests."] else: ... # blargly integration tests go here + # Unconditional errors and failures can be emitted with `error[]` and `fail[]`. + # with testset("not implemented"): + # fail["not implemented yet!"] + with testset(postproc=terminate): test[2 * 2 == 5] # fails, terminating the nearest dynamically enclosing `with session` test[2 * 2 == 4] # not reached diff --git a/unpythonic/syntax/tests/test_nb.py b/unpythonic/syntax/tests/test_nb.py index d4a4a78b..f83d9a0f 100644 --- a/unpythonic/syntax/tests/test_nb.py +++ b/unpythonic/syntax/tests/test_nb.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- -from ...syntax import macros, test, error # noqa: F401 +from ...syntax import macros, test, warn # noqa: F401 from ...test.fixtures import session, testset from ...syntax import macros, nb # noqa: F401, F811 @@ -18,7 +18,7 @@ def runtests(): try: from sympy import symbols, pprint except ImportError: # pragma: no cover - error["SymPy not installed in this Python, cannot test symbolic math in nb."] + warn["SymPy not installed in this Python, skipping symbolic math tests in nb."] else: with nb[pprint]: # you can specify a custom print function (first positional arg) test[_ is None] # noqa: F821 diff --git a/unpythonic/test/fixtures.py b/unpythonic/test/fixtures.py index 76ea87c8..1df3e4cc 100644 --- a/unpythonic/test/fixtures.py +++ b/unpythonic/test/fixtures.py @@ -74,17 +74,18 @@ with testset("inner 2"): test[2 + 2 == 4] - # Unconditional errors can be emitted with `error[]`. + # Warnings can be emitted with `warn[]`. # Useful e.g. if an optional dependency is missing: with testset("integration"): try: import blargly except ImportError: - error["blargly not installed, cannot test integration with it."] + warn["blargly not installed, skipping integration tests."] else: ... # blargly integration tests go here - # Similarly, unconditional errors can be emitted with `fail[]`. + # Unconditional errors can be emitted with `error[]`. + # Unconditional failures can be emitted with `fail[]`. # Useful for marking a testing TODO, or for marking a line # that should be unreachable in a code example. with testset("really fancy tests"): diff --git a/unpythonic/tests/test_mathseq.py b/unpythonic/tests/test_mathseq.py index c9328759..7e4ccd09 100644 --- a/unpythonic/tests/test_mathseq.py +++ b/unpythonic/tests/test_mathseq.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- -from ..syntax import macros, test, test_raises, error, the # noqa: F401 +from ..syntax import macros, test, test_raises, warn, the # noqa: F401 from ..test.fixtures import session, testset from operator import add, mul @@ -27,7 +27,7 @@ def runtests(): try: from sympy import symbols except ImportError: # pragma: no cover - error["SymPy not installed in this Python, cannot test symbolic input for mathseq."] + warn["SymPy not installed in this Python, skipping symbolic input tests for mathseq."] else: x = symbols("x", positive=True) test[sign(x) == +1] @@ -40,7 +40,7 @@ def runtests(): try: from sympy import symbols, exp as symbolicExp, E as NeperE except ImportError: # pragma: no cover - error["SymPy not installed in this Python, cannot test symbolic input for mathseq."] + warn["SymPy not installed in this Python, skipping symbolic input tests for mathseq."] else: test[log(NeperE**2) == 2] x = symbols("x", positive=True) @@ -328,7 +328,7 @@ def runtests(): try: from sympy import symbols except ImportError: # pragma: no cover - error["SymPy not installed in this Python, cannot test symbolic input for mathseq."] + warn["SymPy not installed in this Python, skipping symbolic input tests for mathseq."] else: x0 = symbols("x0", real=True) k = symbols("k", positive=True) # important for geometric series diff --git a/unpythonic/tests/test_numutil.py b/unpythonic/tests/test_numutil.py index 7870a4bd..0c7a09c7 100644 --- a/unpythonic/tests/test_numutil.py +++ b/unpythonic/tests/test_numutil.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- -from ..syntax import macros, test, test_raises, error, the # noqa: F401 +from ..syntax import macros, test, test_raises, warn, the # noqa: F401 from ..test.fixtures import session, testset from itertools import count, takewhile @@ -37,7 +37,7 @@ def runtests(): try: from mpmath import mpf except ImportError: # pragma: no cover - error["mpmath not installed in this Python, cannot test arbitrary precision input for mathseq."] + warn["mpmath not installed in this Python, skipping arbitrary precision input tests."] else: test[almosteq(mpf(1.0), mpf(1.0 + ulp(1.0)))] test[almosteq(1.0, mpf(1.0 + ulp(1.0)))] From 5263093eafff930c78afe3786ce0093dcbe0c8e0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 13:02:06 +0200 Subject: [PATCH 809/832] add emit_warning() to test framework, use for version-suffix skips MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New public API function `emit_warning()` in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside test[] expressions). Unlike the warn[] macro, it does not adjust tests_run since no test has been counted yet. The version-suffix skip in runtests.py now uses emit_warning() instead of printing directly, making skips visible in the testset warning count — consistent with how warn[] reports optional dependency skips. Resolves D5. Co-Authored-By: Claude Opus 4.6 --- TODO_DEFERRED.md | 1 - runtests.py | 9 +++------ unpythonic/test/fixtures.py | 17 ++++++++++++++++- 3 files changed, 19 insertions(+), 8 deletions(-) diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 4999688d..1c75ee88 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -4,4 +4,3 @@ - **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features: NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, IO/TextIO/BinaryIO, Pattern/Match, Generic, Type, Awaitable, Coroutine, AsyncIterable, AsyncIterator, ContextManager, AsyncContextManager, Generator, AsyncGenerator, NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef. Would improve `unpythonic.dispatch` (multiple dispatch). (Discovered during Phase 4 cleanup.) -- **D5**: `runtests.py` — version-suffix skip should signal `TestWarning` (via `unpythonic.conditions.signal`) instead of printing and continuing. This would make skips visible in the testset warning count, consistent with how optional dependency failures show as errors. Currently the skip message bypasses the testset reporting mechanism. (Discovered during Phase 5.) diff --git a/runtests.py b/runtests.py index 3f96c998..9a6cc63f 100644 --- a/runtests.py +++ b/runtests.py @@ -11,12 +11,10 @@ import sys from importlib import import_module -from unpythonic.test.fixtures import (session, testset, maybe_colorize, - tests_errored, tests_failed, TestConfig) +from unpythonic.test.fixtures import (session, testset, emit_warning, + tests_errored, tests_failed) from unpythonic.collections import unbox -from mcpyrate.colorizer import Style - import mcpyrate.activate # noqa: F401 def listtestmodules(path): @@ -64,8 +62,7 @@ def main(): if ver is not None and sys.version_info < ver: msg = (f"Skipping '{m}' (requires Python {ver[0]}.{ver[1]}+, " f"running {sys.version_info.major}.{sys.version_info.minor})") - TestConfig.printer(maybe_colorize(msg, Style.DIM, - TestConfig.ColorScheme.HEADING)) + emit_warning(msg) continue # TODO: We're not inside a package, so we currently can't use a relative import. # TODO: So we just hope this resolves to the local `unpythonic` source code, diff --git a/unpythonic/test/fixtures.py b/unpythonic/test/fixtures.py index 1df3e4cc..29a200be 100644 --- a/unpythonic/test/fixtures.py +++ b/unpythonic/test/fixtures.py @@ -129,13 +129,14 @@ from mcpyrate.bunch import Bunch from mcpyrate.colorizer import Fore, Style, colorize -from ..conditions import handlers, find_restart, invoke +from ..conditions import cerror, handlers, find_restart, invoke from ..collections import box, unbox from ..symbol import sym __all__ = ["session", "testset", "terminate", "returns_normally", "catch_signals", + "emit_warning", "TestConfig", "tests_run", "tests_failed", "tests_errored", "tests_warned", "TestingException", "TestFailure", "TestError", "TestWarning", @@ -183,6 +184,20 @@ def _reset(counter): with _counter_update_lock: counter << 0 +def emit_warning(msg): + """Emit a test warning from infrastructure code (outside a ``test[]`` expression). + + Use this in test runners and other infrastructure that needs to signal + a warning through the test framework without being inside a ``test[]`` + or ``warn[]`` macro. The warning will be displayed and counted by the + nearest enclosing ``testset``. + + Unlike the ``warn[]`` macro, this does not adjust ``tests_run``, + because no test has been counted for this warning to "replace". + """ + _update(tests_warned, +1) + cerror(TestWarning(msg)) + completed = sym("completed") completed.__doc__ = """TestingException `mode`: the test ran to completion normally. From ea2aa6c466d689125399fca3e97e71e2f55074e0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 13:04:33 +0200 Subject: [PATCH 810/832] changelog: document emit_warning() and warn[] for optional deps Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c765fb14..028bd968 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ - `autoreturn` macro now handles `match`/`case` statements. Each case branch has its own tail position. - New scope analyzer tests for `match`/`case` patterns and `try`/`except*`. - Test runner (`runtests.py`) now supports version-suffixed test modules (e.g. `test_foo_3_11.py` runs only on Python 3.11+). +- New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Version-suffix skips now show in the testset warning count. +- Missing optional dependencies (sympy, mpmath) in tests now emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. **Fixed**: From 3596c380b3d954f297e08678a97b774fb8d86925 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 13:05:59 +0200 Subject: [PATCH 811/832] changelog: clarify version-suffix skip wording; add D6 (installable test runner) Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 6 +++--- TODO_DEFERRED.md | 2 ++ 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 028bd968..d4e0a1ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,9 +14,9 @@ - **Python 3.13 and 3.14 support**. - `autoreturn` macro now handles `match`/`case` statements. Each case branch has its own tail position. - New scope analyzer tests for `match`/`case` patterns and `try`/`except*`. -- Test runner (`runtests.py`) now supports version-suffixed test modules (e.g. `test_foo_3_11.py` runs only on Python 3.11+). -- New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Version-suffix skips now show in the testset warning count. -- Missing optional dependencies (sympy, mpmath) in tests now emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. +- Test runner (`runtests.py`): version-suffixed test modules (e.g. `test_foo_3_11.py`) are automatically skipped on older Pythons. +- New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the testset warning count. +- Missing optional dependencies (sympy, mpmath) in tests emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. **Fixed**: diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 1c75ee88..98abfb25 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -4,3 +4,5 @@ - **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features: NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, IO/TextIO/BinaryIO, Pattern/Match, Generic, Type, Awaitable, Coroutine, AsyncIterable, AsyncIterator, ContextManager, AsyncContextManager, Generator, AsyncGenerator, NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef. Would improve `unpythonic.dispatch` (multiple dispatch). (Discovered during Phase 4 cleanup.) +- **D6**: Move the test runner (`runtests.py`) into an installable location (e.g. `unpythonic.test.runner` or similar) so other projects using `unpythonic.test.fixtures` can reuse the version-suffix gating, module discovery, and `emit_warning` integration. Currently it sits at the repo top level and is not installed as part of the package. (Discovered during D5 work.) + From a45c5d9c2a4bfc2a422f9cb71ae028b6533f6d3f Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 13:32:08 +0200 Subject: [PATCH 812/832] extract reusable test runner into unpythonic.test.runner New module `unpythonic.test.runner` provides `discover_testmodules()` and `run()` for projects using `unpythonic.test.fixtures`. Handles module discovery, version-suffix gating, and session/testset wrapping. The top-level `runtests.py` is now a thin wrapper that specifies unpythonic's test directories. Resolves D6. Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 2 +- README.md | 2 + TODO_DEFERRED.md | 1 - doc/macros.md | 15 ++++++- runtests.py | 75 ++++++--------------------------- unpythonic/test/runner.py | 89 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 120 insertions(+), 64 deletions(-) create mode 100644 unpythonic/test/runner.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d4e0a1ef..7dd346c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ - **Python 3.13 and 3.14 support**. - `autoreturn` macro now handles `match`/`case` statements. Each case branch has its own tail position. - New scope analyzer tests for `match`/`case` patterns and `try`/`except*`. -- Test runner (`runtests.py`): version-suffixed test modules (e.g. `test_foo_3_11.py`) are automatically skipped on older Pythons. +- New `unpythonic.test.runner` module: reusable test runner with module discovery, version-suffix gating (e.g. `test_foo_3_11.py` skipped on Python < 3.11), and integration with the test framework's warning system. Other projects using `unpythonic.test.fixtures` can import it directly. - New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the testset warning count. - Missing optional dependencies (sympy, mpmath) in tests emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. diff --git a/README.md b/README.md index ad755d9c..57609ac0 100644 --- a/README.md +++ b/README.md @@ -587,6 +587,8 @@ with session("simple framework demo"): test[2 * 2 == 4] # not reached ``` +For running tests, `unpythonic.test.runner` provides a reusable test runner with module discovery and version-suffix gating. See [`doc/macros.md`](doc/macros.md#unpythonictestfixtures-a-test-framework-for-macro-enabled-python) for details, and [`runtests.py`](runtests.py) for a usage example. + We provide the low-level syntactic constructs `test[]`, `test_raises[]` and `test_signals[]`, with the usual meanings. The last one is for testing code that uses conditions and restarts; see `unpythonic.conditions`. The test macros also come in block variants, `with test`, `with test_raises`, `with test_signals`. diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 98abfb25..58fe09d9 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -4,5 +4,4 @@ - **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features: NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, IO/TextIO/BinaryIO, Pattern/Match, Generic, Type, Awaitable, Coroutine, AsyncIterable, AsyncIterator, ContextManager, AsyncContextManager, Generator, AsyncGenerator, NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef. Would improve `unpythonic.dispatch` (multiple dispatch). (Discovered during Phase 4 cleanup.) -- **D6**: Move the test runner (`runtests.py`) into an installable location (e.g. `unpythonic.test.runner` or similar) so other projects using `unpythonic.test.fixtures` can reuse the version-suffix gating, module discovery, and `emit_warning` integration. Currently it sits at the repo top level and is not installed as part of the package. (Discovered during D5 work.) diff --git a/doc/macros.md b/doc/macros.md index ee4c31a0..961e01d5 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2038,7 +2038,20 @@ All the variants of the testing constructs catch any uncaught exceptions and sig Because `unpythonic.test.fixtures` is, by design, a minimalistic *no-framework* (cf. "NoSQL"), it is up to you to define - in your custom test runner - whether having any failures, errors or warnings should lead to the whole test suite failing. Whether the program's exit code is zero, is important e.g. for GitHub's CI workflows. -For example, in `unpythonic`'s own tests, warnings do not cause the test suite to fail, but errors and failures do. The very short [`runtests.py`](../runtests.py) (just under 60 SLOC) is a complete test runner using `unpythonic.test.fixtures`. +For example, in `unpythonic`'s own tests, warnings do not cause the test suite to fail, but errors and failures do. The top-level [`runtests.py`](../runtests.py) is a complete test runner using the reusable `unpythonic.test.runner` module: + +```python +import os +from unpythonic.test.runner import discover_testmodules, run + +import mcpyrate.activate # noqa: F401 + +testsets = [("my tests", discover_testmodules(os.path.join("mypackage", "tests")))] +if not run(testsets): + raise SystemExit(1) +``` + +`discover_testmodules` finds `test_*.py` files in a directory and returns dotted module names. `run` wraps the session/testset/import pattern, with automatic version-suffix gating (e.g. `test_foo_3_11.py` is skipped with a warning on Python < 3.11). #### Testing syntax quick reference diff --git a/runtests.py b/runtests.py index 9a6cc63f..759424d6 100644 --- a/runtests.py +++ b/runtests.py @@ -1,76 +1,29 @@ # -*- coding: utf-8 -*- -"""Run all tests for `unpythonic`. +"""Run all tests for ``unpythonic``. The test framework uses macros, but this top-level script does not. This can be -run under regular `python3` (i.e. does not need the `macropython` wrapper from -`mcpyrate`). +run under regular ``python3`` (i.e. does not need the ``macropython`` wrapper +from ``mcpyrate``). """ import os -import re import sys -from importlib import import_module -from unpythonic.test.fixtures import (session, testset, emit_warning, - tests_errored, tests_failed) -from unpythonic.collections import unbox +from unpythonic.test.runner import discover_testmodules, run import mcpyrate.activate # noqa: F401 -def listtestmodules(path): - testfiles = listtestfiles(path) - testmodules = [modname(path, fn) for fn in testfiles] - return list(sorted(testmodules)) - -def listtestfiles(path, prefix="test_", suffix=".py"): - return [fn for fn in os.listdir(path) if fn.startswith(prefix) and fn.endswith(suffix)] - -def modname(path, filename): # some/dir/mod.py --> some.dir.mod - modpath = re.sub(os.path.sep, r".", path) - themod = re.sub(r"\.py$", r"", filename) - return ".".join([modpath, themod]) - -def _version_suffix(modulename): - """Parse version suffix from module name. - - E.g. ``unpythonic.syntax.tests.test_scopeanalyzer_3_11`` → ``(3, 11)``, or ``None``. - """ - # Match the final component of a dotted module name. - m = re.search(r"_(\d+)_(\d+)$", modulename) - if m: - return (int(m.group(1)), int(m.group(2))) - return None - def main(): - with session(): - # All folders containing unit tests are named `tests` (plural). - # - # The testing framework is called `unpythonic.test.fixtures`, - # so it lives in the only subfolder in the project that is named - # `test` (singular). - testsets = (("regular code", (listtestmodules(os.path.join("unpythonic", "tests")) + - listtestmodules(os.path.join("unpythonic", "net", "tests")))), - ("macros", listtestmodules(os.path.join("unpythonic", "syntax", "tests"))), - ("dialects", listtestmodules(os.path.join("unpythonic", "dialects", "tests")))) - for tsname, modnames in testsets: - with testset(tsname): - for m in modnames: - # Wrap each module in its own testset to protect the umbrella testset - # against ImportError as well as any failures at macro expansion time. - with testset(m): - ver = _version_suffix(m) - if ver is not None and sys.version_info < ver: - msg = (f"Skipping '{m}' (requires Python {ver[0]}.{ver[1]}+, " - f"running {sys.version_info.major}.{sys.version_info.minor})") - emit_warning(msg) - continue - # TODO: We're not inside a package, so we currently can't use a relative import. - # TODO: So we just hope this resolves to the local `unpythonic` source code, - # TODO: not to an installed copy of the library. - mod = import_module(m) - mod.runtests() - all_passed = (unbox(tests_failed) + unbox(tests_errored)) == 0 - return all_passed + # All folders containing unit tests are named `tests` (plural). + # + # The testing framework is called `unpythonic.test.fixtures`, + # so it lives in the only subfolder in the project that is named + # `test` (singular). + testsets = [("regular code", (discover_testmodules(os.path.join("unpythonic", "tests")) + + discover_testmodules(os.path.join("unpythonic", "net", "tests")))), + ("macros", discover_testmodules(os.path.join("unpythonic", "syntax", "tests"))), + ("dialects", discover_testmodules(os.path.join("unpythonic", "dialects", "tests")))] + return run(testsets) if __name__ == '__main__': if not main(): diff --git a/unpythonic/test/runner.py b/unpythonic/test/runner.py new file mode 100644 index 00000000..afcac5df --- /dev/null +++ b/unpythonic/test/runner.py @@ -0,0 +1,89 @@ +# -*- coding: utf-8 -*- +"""Generic test runner for projects using ``unpythonic.test.fixtures``. + +Provides test module discovery, version-suffix gating, and a ``run`` +function that wraps the standard session/testset/import_module pattern. + +Usage from a project's top-level ``runtests.py``:: + + import os + from unpythonic.test.runner import discover_testmodules, run + + import mcpyrate.activate # noqa: F401 + + testsets = [("my tests", discover_testmodules(os.path.join("mypackage", "tests")))] + if not run(testsets): + raise SystemExit(1) + +Version-suffixed test modules (e.g. ``test_foo_3_11.py``) are automatically +skipped with a warning on older Pythons. +""" + +import os +import re +import sys +from importlib import import_module + +from .fixtures import session, testset, emit_warning, tests_errored, tests_failed +from ..collections import unbox + +__all__ = ["discover_testmodules", "run"] + +def discover_testmodules(path, prefix="test_", suffix=".py"): + """Discover test modules in a directory. + + Returns a sorted list of dotted module names (e.g. + ``["mypackage.tests.test_foo", "mypackage.tests.test_bar"]``). + + Modules are discovered by filename convention: files matching + ``{prefix}*{suffix}`` in the given directory. + """ + filenames = [fn for fn in os.listdir(path) if fn.startswith(prefix) and fn.endswith(suffix)] + modnames = [_filename_to_modulename(path, fn) for fn in filenames] + return list(sorted(modnames)) + +def _filename_to_modulename(path, filename): + """Convert a path and filename to a dotted module name. + + ``("some/dir", "mod.py")`` → ``"some.dir.mod"`` + """ + modpath = re.sub(os.path.sep, r".", path) + themod = re.sub(r"\.py$", r"", filename) + return ".".join([modpath, themod]) + +def _version_suffix(modulename): + """Parse version suffix from module name. + + E.g. ``"mypackage.tests.test_foo_3_11"`` → ``(3, 11)``, or ``None``. + """ + m = re.search(r"_(\d+)_(\d+)$", modulename) + if m: + return (int(m.group(1)), int(m.group(2))) + return None + +def run(testsets): + """Run test modules, reporting results through ``unpythonic.test.fixtures``. + + ``testsets``: iterable of ``(name, modulenames)`` pairs, where ``name`` + is a human-readable label and ``modulenames`` is a list of dotted module + names. Each module must export a ``runtests()`` function. + + Version-suffixed modules (e.g. ``test_foo_3_11``) are automatically + skipped with a warning on Pythons older than the indicated version. + + Returns ``True`` if all tests passed (no failures or errors). + """ + with session(): + for tsname, modnames in testsets: + with testset(tsname): + for m in modnames: + with testset(m): + ver = _version_suffix(m) + if ver is not None and sys.version_info < ver: + msg = (f"Skipping '{m}' (requires Python {ver[0]}.{ver[1]}+, " + f"running {sys.version_info.major}.{sys.version_info.minor})") + emit_warning(msg) + continue + mod = import_module(m) + mod.runtests() + return (unbox(tests_failed) + unbox(tests_errored)) == 0 From af4f91d3fbcf0c66608d5bf526712de30660d297 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 13:37:15 +0200 Subject: [PATCH 813/832] document pyc cache pitfall and test result reading in macros.md Added two new sections to the test framework documentation: - Bytecode cache pitfall: never py_compile macro-enabled code; symptoms, cause, and fix (macropython -c). - Reading test results: what Pass/Fail/Error/Warn mean, including the warn[] convention for optional dep skips. Resolves D1. Co-Authored-By: Claude Opus 4.6 --- TODO_DEFERRED.md | 2 -- doc/macros.md | 25 +++++++++++++++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 58fe09d9..c6ae12c0 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -1,7 +1,5 @@ # Deferred Issues -- **D1**: Document pyc cache pitfall and test result reading for other projects using mcpyrate/unpythonic.test.fixtures. The CLAUDE.md additions from the 2.0.0 modernization (never `py_compile` macro-enabled code; how to read test framework output with Pass/Fail/Error) are useful guidance for any project using these tools. Consider adding similar notes to mcpyrate's docs and/or unpythonic's user-facing documentation. (Discovered during Phase 3.) - - **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features: NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, IO/TextIO/BinaryIO, Pattern/Match, Generic, Type, Awaitable, Coroutine, AsyncIterable, AsyncIterator, ContextManager, AsyncContextManager, Generator, AsyncGenerator, NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef. Would improve `unpythonic.dispatch` (multiple dispatch). (Discovered during Phase 4 cleanup.) diff --git a/doc/macros.md b/doc/macros.md index 961e01d5..d1b8ca0f 100644 --- a/doc/macros.md +++ b/doc/macros.md @@ -2053,6 +2053,31 @@ if not run(testsets): `discover_testmodules` finds `test_*.py` files in a directory and returns dotted module names. `run` wraps the session/testset/import pattern, with automatic version-suffix gating (e.g. `test_foo_3_11.py` is skipped with a warning on Python < 3.11). +#### Important: bytecode cache pitfall + +**Never compile `.py` files in a macro-enabled project** using `py_compile`, `python -m compileall`, pip's `--compile` flag, or any other mechanism that bypasses the macro expander. These tools produce `.pyc` files that do not contain macro-expanded code, which will break macro imports at run time. + +The symptom is typically `ImportError: cannot import name 'macros' from 'mcpyrate.quotes'` (or similar). This happens because the stale `.pyc` is loaded instead of the `.py` source, so the macro expander never runs. + +To fix this, clean the bytecode caches: + +```bash +macropython -c mypackage +``` + +This removes all `__pycache__` directories under the given path. After cleaning, re-run your tests normally — the macro expander will recompile the source files correctly. + +#### Reading test results + +The framework reports **Pass**, **Fail**, **Error**, and **Total** per testset, with optional **Warn** counts. These categories mean: + +- **Pass**: test assertion succeeded. +- **Fail**: test ran to completion, but the assertion was not satisfied. +- **Error**: test did not run to completion (unexpected exception or signal inside a `test[]` expression). This also includes intentional `error[]` signals — so a few errors from skip patterns (e.g. optional dependency not installed) may be normal. Check the actual error messages, not just the count. (Since 2.0.0, optional dependency skips use `warn[]` instead.) +- **Warn**: a human-initiated warning (via `warn[]` or `emit_warning()`). Warnings are not counted in the total, and do not cause the test suite to fail. + +Nested testsets show hierarchy with indentation and asterisk depth (`**`, `****`, `******`, etc.). Counts propagate upward — the top-level summary reflects all tests across all testsets. + #### Testing syntax quick reference **Imports** - complete list: From 0edba2ded739d78da2291ac876c83e4c979ac312 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 13:42:00 +0200 Subject: [PATCH 814/832] emit_warning: improve docstring --- unpythonic/test/fixtures.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/unpythonic/test/fixtures.py b/unpythonic/test/fixtures.py index 29a200be..6bce7a67 100644 --- a/unpythonic/test/fixtures.py +++ b/unpythonic/test/fixtures.py @@ -187,14 +187,15 @@ def _reset(counter): def emit_warning(msg): """Emit a test warning from infrastructure code (outside a ``test[]`` expression). - Use this in test runners and other infrastructure that needs to signal - a warning through the test framework without being inside a ``test[]`` + If you are writing tests, use the `warn[]` macro instead. + + Use this function in test runners and other infrastructure that needs to + signal a warning through the test framework without being inside a ``test[]`` or ``warn[]`` macro. The warning will be displayed and counted by the nearest enclosing ``testset``. - - Unlike the ``warn[]`` macro, this does not adjust ``tests_run``, - because no test has been counted for this warning to "replace". """ + # Unlike the ``warn[]`` macro, this does not adjust ``tests_run``, + # because no test has been counted for this warning to "replace". _update(tests_warned, +1) cerror(TestWarning(msg)) From 665cc4b2f375c641e63d8ca57a08522593c34742 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 14:07:32 +0200 Subject: [PATCH 815/832] typecheck: support NoReturn, Never, Literal, Type, ClassVar, Final, DefaultDict, OrderedDict, Counter, ChainMap (D4 set 1) Add support for the easy-win typing features identified in D4. Also mark typing.Text and typing.ByteString as deprecated (remove at floor Python 3.12), remove stale Python 3.6 guard in tests, and add explanatory comment about why empty collections reject parametric type specs. Co-Authored-By: Claude Opus 4.6 --- TODO_DEFERRED.md | 23 +++++++- unpythonic/tests/test_typecheck.py | 82 ++++++++++++++++++++++++++- unpythonic/typecheck.py | 90 +++++++++++++++++++++++++----- 3 files changed, 177 insertions(+), 18 deletions(-) diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index c6ae12c0..28a09aa8 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -1,5 +1,26 @@ # Deferred Issues -- **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features: NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, IO/TextIO/BinaryIO, Pattern/Match, Generic, Type, Awaitable, Coroutine, AsyncIterable, AsyncIterator, ContextManager, AsyncContextManager, Generator, AsyncGenerator, NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef. Would improve `unpythonic.dispatch` (multiple dispatch). (Discovered during Phase 4 cleanup.) +- **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features. Split into three sets: + **Set 1 — Easy wins** (do first): + - `NoReturn` — always `False` + - `Type[X]` — check `isinstance(value, type) and issubclass(value, X)` + - `Literal[v1, v2, ...]` — check `value in args` + - `ClassVar[T]`, `Final[T]` — strip wrapper, check inner type + - `DefaultDict[K, V]`, `Counter[T]`, `OrderedDict[K, V]`, `ChainMap[K, V]` — slot into existing mapping/collection patterns + - Also: deprecation markers on `typing.Text` and `typing.ByteString` (remove when floor bumps to Python 3.12); clean up stale `Python 3.6+` guard in `test_typecheck.py:182` + + **Set 2 — Useful for dispatch** (follow-up): + - `IO`, `TextIO`, `BinaryIO` — simple `isinstance` checks + - `Pattern[T]`, `Match[T]` — `isinstance` against `re.Pattern`/`re.Match` + - `ContextManager`, `AsyncContextManager` — `isinstance` checks + - `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator` — `isinstance` checks + - `Generator`, `AsyncGenerator` — `isinstance` checks (no yield/send/return type checking) + - `NamedTuple` — tricky but doable + + **Set 3 — Hard / questionable value** (defer or discuss): + - `Protocol` — full structural subtyping, heavy + - `TypedDict` — required vs. optional keys, medium-hard + - `Generic` — abstract, unclear semantics for value checking + - `ForwardRef` — needs a namespace to resolve the string diff --git a/unpythonic/tests/test_typecheck.py b/unpythonic/tests/test_typecheck.py index 28d62668..bfb47383 100644 --- a/unpythonic/tests/test_typecheck.py +++ b/unpythonic/tests/test_typecheck.py @@ -4,6 +4,7 @@ from ..test.fixtures import session, testset import collections +import sys import typing from ..collections import frozendict @@ -32,6 +33,17 @@ def runtests(): test[isoftype("something", typing.Any)] test[isoftype(lambda: ..., typing.Any)] + # NoReturn / Never — the bottom type; no value can match. + with testset("typing.NoReturn"): + test[not isoftype(None, typing.NoReturn)] + test[not isoftype(42, typing.NoReturn)] + test[not isoftype("anything", typing.NoReturn)] + + if sys.version_info >= (3, 11): + with testset("typing.Never"): + test[not isoftype(None, typing.Never)] + test[not isoftype(42, typing.Never)] + # TypeVar, bare; a named type, but behaves like Any. with testset("typing.TypeVar (bare; like a named Any)"): X = typing.TypeVar("X") @@ -67,6 +79,45 @@ def runtests(): test[isoftype(1337, typing.Optional[int])] test[not isoftype(3.14, typing.Optional[int])] + with testset("typing.Literal"): + test[isoftype(1, typing.Literal[1, 2, 3])] + test[isoftype(3, typing.Literal[1, 2, 3])] + test[not isoftype(4, typing.Literal[1, 2, 3])] + test[isoftype("red", typing.Literal["red", "green", "blue"])] + test[not isoftype("yellow", typing.Literal["red", "green", "blue"])] + # Literal values are compared by equality, not identity + test[isoftype(True, typing.Literal[True, False])] + test[not isoftype(None, typing.Literal[True, False])] + + with testset("typing.Type"): + test[isoftype(int, typing.Type[int])] + test[isoftype(bool, typing.Type[int])] # bool is a subclass of int + test[not isoftype(str, typing.Type[int])] + test[not isoftype(42, typing.Type[int])] # an instance, not a class + # bare Type: any class matches + test[isoftype(int, typing.Type)] + test[isoftype(str, typing.Type)] + test[not isoftype(42, typing.Type)] + + with testset("typing.ClassVar"): + test[isoftype(42, typing.ClassVar[int])] + test[not isoftype("hello", typing.ClassVar[int])] + # Compound: ClassVar wrapping a Union + test[isoftype(42, typing.ClassVar[typing.Union[int, str]])] + test[isoftype("hello", typing.ClassVar[typing.Union[int, str]])] + test[not isoftype(3.14, typing.ClassVar[typing.Union[int, str]])] + + with testset("typing.Final"): + test[isoftype(42, typing.Final[int])] + test[not isoftype("hello", typing.Final[int])] + test[isoftype("hello", typing.Final[str])] + + # Empty collections reject parametric type specs (e.g. `Tuple[int, ...]`, + # `List[int]`, `Dict[str, int]`). An empty collection has no elements to + # infer the type from, so matching it against a specific element type would + # be guesswork — which would make multiple dispatch unpredictable. + # Bare (unparametrized) specs like `Tuple` or `Dict` still accept empties. + with testset("typing.Tuple"): test[isoftype((1, 2, 3), typing.Tuple)] test[isoftype((1, 2, 3), typing.Tuple[int, ...])] @@ -101,6 +152,34 @@ def runtests(): # no type arguments: any key/value types ok (consistent with Python 3.7+) test[isoftype({"cat": "animal", "pi": 3.14159, 2.71828: "e"}, typing.Dict)] + with testset("typing.DefaultDict"): + dd = collections.defaultdict(int, {"a": 1, "b": 2}) + test[isoftype(dd, typing.DefaultDict[str, int])] + test[not isoftype(dd, typing.DefaultDict[int, int])] + test[not isoftype({}, typing.DefaultDict[str, int])] # regular dict is not defaultdict + test[not isoftype(collections.defaultdict(int), typing.DefaultDict[str, int])] # empty + + with testset("typing.OrderedDict"): + od = collections.OrderedDict({"x": 1, "y": 2}) + test[isoftype(od, typing.OrderedDict[str, int])] + test[not isoftype(od, typing.OrderedDict[int, int])] + test[not isoftype({}, typing.OrderedDict[str, int])] # regular dict is not OrderedDict + test[not isoftype(collections.OrderedDict(), typing.OrderedDict[str, int])] # empty + + with testset("typing.Counter"): + c = collections.Counter("abracadabra") + test[isoftype(c, typing.Counter[str])] + test[not isoftype(c, typing.Counter[int])] + test[not isoftype({}, typing.Counter[str])] # regular dict is not Counter + test[not isoftype(collections.Counter(), typing.Counter[str])] # empty + + with testset("typing.ChainMap"): + cm = collections.ChainMap({"a": 1}, {"b": 2}) + test[isoftype(cm, typing.ChainMap[str, int])] + test[not isoftype(cm, typing.ChainMap[int, int])] + test[not isoftype({}, typing.ChainMap[str, int])] # regular dict is not ChainMap + test[not isoftype(collections.ChainMap(), typing.ChainMap[str, int])] # empty + # type alias (at run time, this is just an assignment) with testset("type alias"): U = typing.Union[int, str] @@ -179,8 +258,7 @@ def runtests(): test[isoftype([1, 2, 3], typing.Iterable)] test[isoftype([1, 2, 3], typing.Reversible)] test[isoftype([1, 2, 3], typing.Container)] - if hasattr(typing, "Collection"): # Python 3.6+ - test[isoftype([1, 2, 3], typing.Collection)] # Sized Iterable Container + test[isoftype([1, 2, 3], typing.Collection)] # Sized Iterable Container with testset("typing.KeysView, typing.ValuesView, typing.ItemsView"): d = {17: "cat", 23: "fox", 42: "python"} diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index de58c805..dbc1d16e 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -1,10 +1,9 @@ # -*- coding: utf-8; -*- -"""Simplistic run-time type checker. +"""Lightweight run-time type checker. -This implements just a minimal feature set needed for checking function -arguments in typical uses of multiple dispatch (see `unpythonic.dispatch`). -That said, this DOES support many (but not all) features of the `typing` stdlib -module. +Originally built for the minimal feature set needed by multiple dispatch +(see `unpythonic.dispatch`), but designed as a general-purpose utility. +Supports many (but not all) features of the `typing` stdlib module. We currently provide `isoftype` (cf. `isinstance`), but no `issubtype` (cf. `issubclass`). @@ -15,6 +14,7 @@ """ import collections +import sys import types import typing @@ -40,14 +40,21 @@ def isoftype(value, T): - `TypeVar` - `NewType` (any instance of the underlying actual type will match) - `Union[T1, T2, ..., TN]` + - `NoReturn`, `Never` (no value matches; `Never` requires Python 3.11+) + - `Literal[v1, v2, ...]` + - `Type[X]` (value must be a class that is `X` or a subclass of `X`) + - `ClassVar[T]`, `Final[T]` (wrapper stripped, inner type checked) - `Tuple`, `Tuple[T, ...]`, `Tuple[T1, T2, ..., TN]`, `Sequence[T]` - `List[T]`, `MutableSequence[T]` - `FrozenSet[T]`, `AbstractSet[T]` - `Set[T]`, `MutableSet[T]` - - `Dict[K, V]`, `MutableMapping[K, V]`, `Mapping[K, V]` + - `Dict[K, V]`, `DefaultDict[K, V]`, `OrderedDict[K, V]` + - `Counter[T]` (element type checked; value type is always `int`) + - `ChainMap[K, V]` + - `MutableMapping[K, V]`, `Mapping[K, V]` - `ItemsView[K, V]`, `KeysView[K]`, `ValuesView[V]` - `Callable` (argument and return value types currently NOT checked) - - `Text` + - `Text` (deprecated since Python 3.11; will be removed at floor Python 3.12) Any checks on the type arguments of the meta-utilities are performed recursively using `isoftype`, in order to allow compound specifications. @@ -66,15 +73,22 @@ def isoftype(value, T): # Python provides no official public API for run-time type introspection. # # Unsupported typing features: - # NamedTuple, DefaultDict, Counter, ChainMap, OrderedDict, - # IO, TextIO, BinaryIO, Pattern, Match, Generic, Type, + # NamedTuple, + # IO, TextIO, BinaryIO, Pattern, Match, # Awaitable, Coroutine, AsyncIterable, AsyncIterator, # ContextManager, AsyncContextManager, Generator, AsyncGenerator, - # NoReturn, ClassVar, Final, Protocol, TypedDict, Literal, ForwardRef + # Generic, Protocol, TypedDict, ForwardRef if T is typing.Any: return True + # NoReturn means a function never returns — no value has this type. + # Never (3.11+) is the bottom type; semantically the same for our purposes. + if T is typing.NoReturn: + return False + if sys.version_info >= (3, 11) and T is typing.Never: + return False + # AnyStr normalizes to TypeVar("AnyStr", str, bytes) if isinstance(T, typing.TypeVar): if not T.__constraints__: # just an abstract type name @@ -102,6 +116,28 @@ def isNewType(T): # print(type(i)) # int return isinstance(value, T.__supertype__) + # Literal[v1, v2, ...] — value must be one of the listed constants. + if typing.get_origin(T) is typing.Literal: + return value in T.__args__ + + # Type[X] — value must be a class that is X or a subclass of X. + if typing.get_origin(T) is type: + if not isinstance(value, type): + return False + args = getattr(T, "__args__", None) + if args is None: + return True # bare Type, any class matches + return issubclass(value, args[0]) + + # ClassVar[T] and Final[T] — these are declaration wrappers. At runtime, + # we just strip the wrapper and check the inner type. + for wrapper_origin in (typing.ClassVar, typing.Final): + if typing.get_origin(T) is wrapper_origin: + args = getattr(T, "__args__", None) + if args is None: + return True # bare ClassVar or Final, no inner type constraint + return isoftype(value, args[0]) + # Some one-trick ponies. for U in (typing.Iterator, # can't non-destructively check element type typing.Iterable, # can't non-destructively check element type @@ -128,8 +164,10 @@ def isNewType(T): # We don't have a match yet, so T might still be one of those meta-utilities # that hate `issubclass` with a passion. - if safeissubclass(T, typing.Text): # https://docs.python.org/3/library/typing.html#typing.Text - return isinstance(value, str) # alias for str + # DEPRECATED: typing.Text is deprecated since Python 3.11 (it's just an alias for str). + # TODO: Remove this branch when the floor bumps to Python 3.12. + if safeissubclass(T, typing.Text): + return isinstance(value, str) if typing.get_origin(T) is tuple: if not isinstance(value, tuple): @@ -162,7 +200,25 @@ def ismapping(runtimetype): return False K, V = args return all(isoftype(k, K) and isoftype(v, V) for k, v in value.items()) - for runtimetype in (dict, collections.abc.MutableMapping, collections.abc.Mapping): + # Counter[T] is a mapping (keys: T, values: int), but has only one type arg. + if typing.get_origin(T) is collections.Counter: + if not isinstance(value, collections.Counter): + return False + args = getattr(T, "__args__", None) + if args is None: + args = (typing.TypeVar("T"),) + assert len(args) == 1 + if not value: + return False + U = args[0] + return all(isoftype(k, U) and isinstance(v, int) for k, v in value.items()) + + for runtimetype in (dict, + collections.defaultdict, + collections.OrderedDict, + collections.ChainMap, + collections.abc.MutableMapping, + collections.abc.Mapping): if typing.get_origin(T) is runtimetype: return ismapping(runtimetype) @@ -190,8 +246,12 @@ def iscollection(statictype, runtimetype): if not isinstance(value, runtimetype): return False if typing.get_origin(statictype) is collections.abc.ByteString: + # DEPRECATED: typing.ByteString is deprecated since Python 3.12. + # TODO: Remove this branch and the ByteString entry in the loop below + # when the floor bumps to Python 3.12. + # # WTF? A ByteString is a Sequence[int], but only statically. - # At run time, the `__args__` are actually empty - it looks + # At run time, the `__args__` are actually empty — it looks # like a bare Sequence, which is invalid. HACK the special case. typeargs = (int,) else: @@ -209,7 +269,7 @@ def iscollection(statictype, runtimetype): (typing.FrozenSet, frozenset), (typing.Set, set), (typing.Deque, collections.deque), - (typing.ByteString, collections.abc.ByteString), # must check before Sequence + (typing.ByteString, collections.abc.ByteString), # DEPRECATED; must check before Sequence (typing.MutableSet, collections.abc.MutableSet), # must check mutable first # because a mutable value has *also* the interface of the immutable variant # (e.g. MutableSet is a subtype of AbstractSet) From f32048ca5efbd2a9f9aa71b9a7387bf0ff6ee4a6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 14:08:28 +0200 Subject: [PATCH 816/832] changelog: document D4 set 1 typecheck additions and deprecations Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7dd346c6..4f61d517 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ - New `unpythonic.test.runner` module: reusable test runner with module discovery, version-suffix gating (e.g. `test_foo_3_11.py` skipped on Python < 3.11), and integration with the test framework's warning system. Other projects using `unpythonic.test.fixtures` can import it directly. - New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the testset warning count. - Missing optional dependencies (sympy, mpmath) in tests emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. +- Runtime type checker (`unpythonic.typecheck`): new supported typing features — `NoReturn`, `Never` (3.11+), `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`. **Fixed**: @@ -33,6 +34,7 @@ **Deprecated**: - Parenthesis syntax for macro arguments (e.g. `let((x, 1), (y, 2))`). Use bracket syntax instead: `let[[x, 1], [y, 2]]`. The parenthesis syntax is kept for backward compatibility but may be removed in a future version. +- Runtime type checker: `typing.Text` (deprecated since Python 3.11) and `typing.ByteString` (deprecated since Python 3.12) support is now marked for removal when the floor bumps to Python 3.12. --- From 59fea4040422ca63a55fe3de755adc300772ddfb Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 16:30:09 +0200 Subject: [PATCH 817/832] typecheck: support IO, TextIO, BinaryIO, Pattern, Match, ContextManager, Generator, async types (D4 set 2) Add support for IO types (mapped to io module ABCs), Pattern/Match (with string type checking), ContextManager/AsyncContextManager, Awaitable/Coroutine, AsyncIterable/AsyncIterator, Generator/AsyncGenerator. Also add dispatch integration tests exercising Sets 1 and 2 features through @generic (Literal, Type, mapping variants, IO, Pattern, Generator, ContextManager). New deferred items: D5 (parametric one-trick ponies and dispatch-layer warnings), D7 (doc/features.md update for isoftype). Co-Authored-By: Claude Opus 4.6 --- TODO_DEFERRED.md | 28 +++++---- unpythonic/tests/test_dispatch.py | 89 +++++++++++++++++++++++++++++ unpythonic/tests/test_typecheck.py | 92 ++++++++++++++++++++++++++++++ unpythonic/typecheck.py | 64 +++++++++++++++++++-- 4 files changed, 254 insertions(+), 19 deletions(-) diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 28a09aa8..5cad6545 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -1,22 +1,13 @@ # Deferred Issues -- **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features. Split into three sets: +Next unused item code: D8 - **Set 1 — Easy wins** (do first): - - `NoReturn` — always `False` - - `Type[X]` — check `isinstance(value, type) and issubclass(value, X)` - - `Literal[v1, v2, ...]` — check `value in args` - - `ClassVar[T]`, `Final[T]` — strip wrapper, check inner type - - `DefaultDict[K, V]`, `Counter[T]`, `OrderedDict[K, V]`, `ChainMap[K, V]` — slot into existing mapping/collection patterns - - Also: deprecation markers on `typing.Text` and `typing.ByteString` (remove when floor bumps to Python 3.12); clean up stale `Python 3.6+` guard in `test_typecheck.py:182` +- **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features. - **Set 2 — Useful for dispatch** (follow-up): - - `IO`, `TextIO`, `BinaryIO` — simple `isinstance` checks - - `Pattern[T]`, `Match[T]` — `isinstance` against `re.Pattern`/`re.Match` - - `ContextManager`, `AsyncContextManager` — `isinstance` checks - - `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator` — `isinstance` checks - - `Generator`, `AsyncGenerator` — `isinstance` checks (no yield/send/return type checking) - - `NamedTuple` — tricky but doable + **Set 1 — Easy wins**: DONE (`665cc4b`) + **Set 2 — Useful for dispatch**: DONE (this commit) + - `NamedTuple`: specific subclasses already work via `isinstance` fallback; no special handling needed. + **Dispatch integration tests** for Sets 1 and 2: DONE (this commit) **Set 3 — Hard / questionable value** (defer or discuss): - `Protocol` — full structural subtyping, heavy @@ -24,3 +15,10 @@ - `Generic` — abstract, unclear semantics for value checking - `ForwardRef` — needs a namespace to resolve the string +- **D5**: `typecheck.py` / `dispatch.py` — parametric forms of existing one-trick ponies (e.g. `Iterator[int]`, `Iterable[str]`) raise `NotImplementedError`. The bare forms work. The `NotImplementedError` is arguably correct fail-fast behavior, since ignoring the type arg would silently accept wrong element types and make dispatching on e.g. `Iterable[int]` vs. `Iterable[float]` silently misroute. Same situation already exists for `Callable`, `Generator`, `ContextManager`, and async types. Possible improvements (dispatch layer, not typecheck): + - Emit a warning when a type arg is silently ignored during dispatch. + - Raise `TypeError` when registering indistinguishable multimethods (e.g. `Iterable[int]` then `Iterable[float]`). + (Discovered during D4 Set 2 work.) + +- **D7**: `doc/features.md` — the `isoftype` section needs updating: add examples for new typing features (D4 Sets 1+2), remove stale Python 3.6–3.9 CAUTION, add a note that this is a non-destructive runtime typechecker (which limits what it can check — e.g. element types of iterators, arg/return types of callables). Also consider noting this in the `@generic` docstring. (Discovered during D4 work.) + diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index e5efabbc..355bc92a 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -3,7 +3,12 @@ from ..syntax import macros, test, test_raises, fail, the # noqa: F401 from ..test.fixtures import session, testset, returns_normally +import collections +import contextlib +import io +import re import typing + from ..fun import curry from ..dispatch import generic, augment, typed, format_methods @@ -379,6 +384,90 @@ def flip(traitvalue: IsNotFlippable, x: typing.Any): # noqa: F811 test_raises[TypeError, flip(42), "int should not be flippable"] test_raises[NotImplementedError, flip(2.0), "float should not be registered for the flippable trait"] + # Exercise new typing features (D4 sets 1 and 2) through the dispatch machinery. + # Most-recently-registered multimethod is tried first, so register the + # general case first and the specific ones after (to override). + with testset("@generic with Literal dispatch"): + @generic + def handle_code(code: int): + return "other" + @generic + def handle_code(code: typing.Literal[200, 201]): # noqa: F811 + return "success" + @generic + def handle_code(code: typing.Literal[404]): # noqa: F811 + return "not found" + test[handle_code(200) == "success"] + test[handle_code(201) == "success"] + test[handle_code(404) == "not found"] + test[handle_code(500) == "other"] + + with testset("@generic with Type dispatch"): + @generic + def describe_type(cls: typing.Type[int]): + return "integer type" + @generic + def describe_type(cls: typing.Type[str]): # noqa: F811 + return "string type" + test[describe_type(int) == "integer type"] + test[describe_type(bool) == "integer type"] # bool is a subclass of int + test[describe_type(str) == "string type"] + test_raises[TypeError, describe_type(float)] + + with testset("@generic with mapping variants"): + @generic + def process_mapping(d: typing.Dict[str, int]): + return "dict" + @generic + def process_mapping(d: typing.DefaultDict[str, int]): # noqa: F811 + return "defaultdict" + @generic + def process_mapping(d: typing.Counter[str]): # noqa: F811 + return "counter" + @generic + def process_mapping(d: typing.OrderedDict[str, int]): # noqa: F811 + return "ordereddict" + test[process_mapping(collections.defaultdict(int, a=1)) == "defaultdict"] + test[process_mapping(collections.Counter("abc")) == "counter"] + test[process_mapping(collections.OrderedDict(a=1)) == "ordereddict"] + test[process_mapping({"a": 1}) == "dict"] + + with testset("@generic with IO dispatch"): + @generic + def read_stream(s: typing.TextIO): + return "text" + @generic + def read_stream(s: typing.BinaryIO): # noqa: F811 + return "binary" + test[read_stream(io.StringIO("hello")) == "text"] + test[read_stream(io.BytesIO(b"hello")) == "binary"] + + with testset("@generic with Pattern dispatch"): + @generic + def describe_pattern(p: typing.Pattern[str]): + return "str pattern" + @generic + def describe_pattern(p: typing.Pattern[bytes]): # noqa: F811 + return "bytes pattern" + test[describe_pattern(re.compile(r"\d+")) == "str pattern"] + test[describe_pattern(re.compile(rb"\d+")) == "bytes pattern"] + + with testset("@generic with Generator and ContextManager"): + @generic + def classify(x: typing.Generator): + return "generator" + @generic + def classify(x: typing.ContextManager): # noqa: F811 + return "context manager" + @generic + def classify(x: int): # noqa: F811 + return "int" + def mygen(): + yield 1 + test[classify(mygen()) == "generator"] + test[classify(contextlib.nullcontext()) == "context manager"] + test[classify(42) == "int"] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/tests/test_typecheck.py b/unpythonic/tests/test_typecheck.py index bfb47383..e1d2d933 100644 --- a/unpythonic/tests/test_typecheck.py +++ b/unpythonic/tests/test_typecheck.py @@ -3,7 +3,11 @@ from ..syntax import macros, test, test_raises, warn # noqa: F401 from ..test.fixtures import session, testset +import asyncio import collections +import contextlib +import io +import re import sys import typing @@ -281,6 +285,94 @@ def runtests(): # https://docs.python.org/3/glossary.html#term-dictionary-view # https://docs.python.org/3/library/stdtypes.html#dict-views + with testset("typing.IO, typing.TextIO, typing.BinaryIO"): + sio = io.StringIO("hello") + bio = io.BytesIO(b"hello") + test[isoftype(sio, typing.IO)] + test[isoftype(bio, typing.IO)] + test[isoftype(sio, typing.TextIO)] + test[not isoftype(bio, typing.TextIO)] + test[isoftype(bio, typing.BinaryIO)] + test[not isoftype(sio, typing.BinaryIO)] + test[not isoftype(42, typing.IO)] + # Parametric IO: IO[str] matches text, IO[bytes] matches binary + test[isoftype(sio, typing.IO[str])] + test[not isoftype(bio, typing.IO[str])] + test[isoftype(bio, typing.IO[bytes])] + test[not isoftype(sio, typing.IO[bytes])] + + with testset("typing.Pattern, typing.Match"): + pstr = re.compile(r"\d+") + pbytes = re.compile(rb"\d+") + mstr = pstr.match("123") + mbytes = pbytes.match(b"123") + # Bare Pattern/Match — any string type + test[isoftype(pstr, typing.Pattern)] + test[isoftype(pbytes, typing.Pattern)] + test[isoftype(mstr, typing.Match)] + test[isoftype(mbytes, typing.Match)] + test[not isoftype("not a pattern", typing.Pattern)] + test[not isoftype(42, typing.Match)] + # Parametric — string type checked + test[isoftype(pstr, typing.Pattern[str])] + test[not isoftype(pstr, typing.Pattern[bytes])] + test[isoftype(pbytes, typing.Pattern[bytes])] + test[not isoftype(pbytes, typing.Pattern[str])] + test[isoftype(mstr, typing.Match[str])] + test[not isoftype(mstr, typing.Match[bytes])] + test[isoftype(mbytes, typing.Match[bytes])] + test[not isoftype(mbytes, typing.Match[str])] + + with testset("typing.ContextManager"): + # contextlib.nullcontext is a context manager + cm = contextlib.nullcontext() + test[isoftype(cm, typing.ContextManager)] + test[isoftype(cm, typing.ContextManager[None])] # type arg ignored (can't check) + test[not isoftype(42, typing.ContextManager)] + + with testset("typing.Generator"): + def mygen(): + yield 1 + yield 2 + g = mygen() + test[isoftype(g, typing.Generator)] + test[isoftype(g, typing.Generator[int, None, None])] # type args ignored + test[not isoftype(42, typing.Generator)] + test[not isoftype([1, 2, 3], typing.Generator)] # iterable, but not a generator + + with testset("typing.Awaitable, typing.Coroutine"): + async def mycoro(): + return 42 + c = mycoro() + test[isoftype(c, typing.Awaitable)] + test[isoftype(c, typing.Coroutine)] + test[isoftype(c, typing.Awaitable[int])] # type arg ignored + test[not isoftype(42, typing.Awaitable)] + test[not isoftype(42, typing.Coroutine)] + c.close() # prevent RuntimeWarning about unawaited coroutine + + with testset("typing.AsyncIterable, typing.AsyncIterator"): + class MyAsyncIter: + def __aiter__(self): + return self + async def __anext__(self): + raise StopAsyncIteration + ai = MyAsyncIter() + test[isoftype(ai, typing.AsyncIterable)] + test[isoftype(ai, typing.AsyncIterator)] + test[isoftype(ai, typing.AsyncIterable[int])] # type arg ignored + test[not isoftype(42, typing.AsyncIterable)] + test[not isoftype([1, 2], typing.AsyncIterator)] # sync iterable, not async + + with testset("typing.AsyncGenerator"): + async def myasyncgen(): + yield 1 + ag = myasyncgen() + test[isoftype(ag, typing.AsyncGenerator)] + test[isoftype(ag, typing.AsyncGenerator[int, None])] # type args ignored + test[not isoftype(42, typing.AsyncGenerator)] + asyncio.run(ag.aclose()) # prevent RuntimeWarning + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index dbc1d16e..1145440d 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -14,6 +14,9 @@ """ import collections +import contextlib +import io +import re import sys import types import typing @@ -54,6 +57,12 @@ def isoftype(value, T): - `MutableMapping[K, V]`, `Mapping[K, V]` - `ItemsView[K, V]`, `KeysView[K]`, `ValuesView[V]` - `Callable` (argument and return value types currently NOT checked) + - `IO`, `TextIO`, `BinaryIO` (mapped to ``io`` module ABCs) + - `Pattern[T]`, `Match[T]` (string type checked when parametric) + - `ContextManager[T]`, `AsyncContextManager[T]` + - `Awaitable[T]`, `Coroutine[T1, T2, T3]` + - `AsyncIterable[T]`, `AsyncIterator[T]` + - `Generator[Y, S, R]`, `AsyncGenerator[Y, S]` - `Text` (deprecated since Python 3.11; will be removed at floor Python 3.12) Any checks on the type arguments of the meta-utilities are performed @@ -73,10 +82,7 @@ def isoftype(value, T): # Python provides no official public API for run-time type introspection. # # Unsupported typing features: - # NamedTuple, - # IO, TextIO, BinaryIO, Pattern, Match, - # Awaitable, Coroutine, AsyncIterable, AsyncIterator, - # ContextManager, AsyncContextManager, Generator, AsyncGenerator, + # NamedTuple (specific NamedTuple subclasses work via isinstance fallback), # Generic, Protocol, TypedDict, ForwardRef if T is typing.Any: @@ -169,6 +175,56 @@ def isNewType(T): if safeissubclass(T, typing.Text): return isinstance(value, str) + # IO, TextIO, BinaryIO — typing module stubs that don't participate in the + # MRO of real IO classes. Map to the io module ABCs instead. + # IO[str] → TextIO, IO[bytes] → BinaryIO when parametric. + if T is typing.IO or typing.get_origin(T) is typing.IO: + args = getattr(T, "__args__", None) + if args is not None: + if args[0] is str: + return isinstance(value, io.TextIOBase) + if args[0] is bytes: + return isinstance(value, (io.RawIOBase, io.BufferedIOBase)) + return isinstance(value, io.IOBase) + if T is typing.TextIO: + return isinstance(value, io.TextIOBase) + if T is typing.BinaryIO: + return isinstance(value, (io.RawIOBase, io.BufferedIOBase)) + + # Pattern[T] and Match[T] — the type arg (str or bytes) can be checked. + if typing.get_origin(T) is re.Pattern: + if not isinstance(value, re.Pattern): + return False + args = getattr(T, "__args__", None) + if args is not None: + return isinstance(value.pattern, args[0]) + return True + if typing.get_origin(T) is re.Match: + if not isinstance(value, re.Match): + return False + args = getattr(T, "__args__", None) + if args is not None: + return isinstance(value.string, args[0]) + return True + + # ContextManager and AsyncContextManager — can't check the return type + # of __enter__ non-destructively, so just check the ABC. + if typing.get_origin(T) is contextlib.AbstractContextManager: + return isinstance(value, contextlib.AbstractContextManager) + if typing.get_origin(T) is contextlib.AbstractAsyncContextManager: + return isinstance(value, contextlib.AbstractAsyncContextManager) + + # Async ABCs and generator types — type parameters (yield, send, return) + # can't be checked non-destructively, so just check the ABC. + for runtimetype in (collections.abc.Awaitable, + collections.abc.Coroutine, + collections.abc.AsyncIterable, + collections.abc.AsyncIterator, + collections.abc.Generator, + collections.abc.AsyncGenerator): + if typing.get_origin(T) is runtimetype: + return isinstance(value, runtimetype) + if typing.get_origin(T) is tuple: if not isinstance(value, tuple): return False From bec2470b6fa511eb21e697526d894ae358b03cf8 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 16:30:44 +0200 Subject: [PATCH 818/832] changelog: document D4 set 2 typecheck additions Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f61d517..f9baf739 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,7 @@ - New `unpythonic.test.runner` module: reusable test runner with module discovery, version-suffix gating (e.g. `test_foo_3_11.py` skipped on Python < 3.11), and integration with the test framework's warning system. Other projects using `unpythonic.test.fixtures` can import it directly. - New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the testset warning count. - Missing optional dependencies (sympy, mpmath) in tests emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. -- Runtime type checker (`unpythonic.typecheck`): new supported typing features — `NoReturn`, `Never` (3.11+), `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`. +- Runtime type checker (`unpythonic.typecheck`): new supported typing features — `NoReturn`, `Never` (3.11+), `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO` (mapped to `io` module ABCs), `Pattern[T]`/`Match[T]` (string type checked when parametric), `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator`, `Generator`, `AsyncGenerator`. **Fixed**: From 6aa72ea871954be79ef912c3dbe6ac35315e85f4 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 16:35:06 +0200 Subject: [PATCH 819/832] doc: update isoftype section with full supported types table and new examples (D7) Rewrite the isoftype section in doc/features.md: add supported types table, new 2.0.0 examples (Literal, Type, Counter, Pattern, IO), explain non-destructive checking design, document which type params are silently ignored, remove stale Python 3.6-3.9 CAUTION. Add v2.0.0 change note to the section header. Remove resolved D7 from deferred list. Co-Authored-By: Claude Opus 4.6 --- TODO_DEFERRED.md | 2 -- doc/features.md | 63 +++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 51 insertions(+), 14 deletions(-) diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 5cad6545..0c4e7999 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -20,5 +20,3 @@ Next unused item code: D8 - Raise `TypeError` when registering indistinguishable multimethods (e.g. `Iterable[int]` then `Iterable[float]`). (Discovered during D4 Set 2 work.) -- **D7**: `doc/features.md` — the `isoftype` section needs updating: add examples for new typing features (D4 Sets 1+2), remove stale Python 3.6–3.9 CAUTION, add a note that this is a non-destructive runtime typechecker (which limits what it can check — e.g. element types of iterators, arg/return types of callables). Also consider noting this in the `@generic` docstring. (Discovered during D4 work.) - diff --git a/doc/features.md b/doc/features.md index 681835e6..40766340 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3731,6 +3731,8 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of ### `generic`, `typed`, `isoftype`: multiple dispatch +**Changed in v2.0.0.** *`isoftype` now supports many more `typing` features: `NoReturn`, `Never`, `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO`, `Pattern`/`Match`, `ContextManager`/`AsyncContextManager`, `Awaitable`/`Coroutine`, `AsyncIterable`/`AsyncIterator`, `Generator`/`AsyncGenerator`. See the [`isoftype` section](#isoftype-the-big-sister-of-isinstance) for the full list.* + **Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Starting with this release, they receive the same semantic-versioning guarantees as the rest of `unpythonic`.* *Added the `@augment` parametric decorator that can register a new multimethod on an existing generic function originally defined in another lexical scope.* @@ -3749,7 +3751,7 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of **Added in v0.14.2**. -The `generic` decorator allows creating [multiple-dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) generic functions with type annotation syntax. We also provide some friendly utilities: `augment` adds a new multimethod to an existing generic function, `typed` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type-checking code), and `isoftype` (which powers the first three) is the big sister of `isinstance`, with support for many (but unfortunately not all) features of the `typing` standard library module. +The `generic` decorator allows creating [multiple-dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) generic functions with type annotation syntax. We also provide some friendly utilities: `augment` adds a new multimethod to an existing generic function, `typed` creates a single-method generic with the same syntax (i.e. provides a compact notation for writing dynamic type-checking code), and `isoftype` (which powers the first three) is the big sister of `isinstance`, with support for many (but not all) features of the `typing` standard library module. This is a purely run-time implementation, so it does **not** give performance benefits, but it can make code more readable, and makes it modular to add support for new input types (or different call signatures) to an existing function later. @@ -3844,7 +3846,7 @@ assert kittify(x=1, y=2) == "int" assert kittify(x=1.0, y=2.0) == "float" ``` -See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see `isoftype` below. +See [the unit tests](../unpythonic/tests/test_dispatch.py) for more. For which features of the `typing` stdlib module are supported, see [`isoftype`](#isoftype-the-big-sister-of-isinstance) below. ##### `@generic` and OOP @@ -4037,14 +4039,41 @@ assert jack("foo") == "foo" jack(3.14) # TypeError ``` -For which features of the `typing` stdlib module are supported, see `isoftype` below. +For which features of the `typing` stdlib module are supported, see [`isoftype`](#isoftype-the-big-sister-of-isinstance) below. #### `isoftype`: the big sister of `isinstance` -Type check object instances against type specifications at run time. This is the machinery that powers `generic` and `typed`. This goes beyond `isinstance` in that many (but unfortunately not all) features of the `typing` standard library module are supported. - -Any checks on the type arguments of the meta-utilities defined in the `typing` stdlib module are performed recursively using `isoftype` itself, in order to allow compound abstract specifications. +Type check object instances against type specifications at run time. This is the machinery that powers `generic` and `typed`. This goes beyond `isinstance` in that many (but not all) features of the `typing` standard library module are supported. + +`isoftype` is a **non-destructive** runtime type checker. It never consumes iterators, calls functions, or enters context managers to inspect their types. This limits what it can check — for example, element types of iterators and argument/return types of callables cannot be verified — but it means `isoftype` is always safe to call, even in hot loops or dispatch logic. + +Any checks on the type arguments of the meta-utilities defined in the `typing` stdlib module are performed recursively using `isoftype` itself, in order to allow compound specifications. + +**Supported `typing` features:** + +| Category | Supported types | +|----------|----------------| +| Basics | `Any`, `TypeVar`, `NewType`, `Union`, `Optional` | +| Bottom | `NoReturn`, `Never` (3.11+) | +| Values | `Literal[v1, v2, ...]` | +| Classes | `Type[X]` | +| Wrappers | `ClassVar[T]`, `Final[T]` (stripped, inner type checked) | +| Tuples | `Tuple`, `Tuple[T, ...]`, `Tuple[T1, T2, ..., TN]` | +| Sequences | `List[T]`, `Sequence[T]`, `MutableSequence[T]`, `Deque[T]` | +| Sets | `Set[T]`, `FrozenSet[T]`, `AbstractSet[T]`, `MutableSet[T]` | +| Mappings | `Dict[K, V]`, `DefaultDict[K, V]`, `OrderedDict[K, V]`, `Counter[T]`, `ChainMap[K, V]`, `Mapping[K, V]`, `MutableMapping[K, V]` | +| Views | `KeysView[K]`, `ValuesView[V]`, `ItemsView[K, V]` | +| IO | `IO`, `IO[str]`, `IO[bytes]`, `TextIO`, `BinaryIO` | +| Regex | `Pattern[T]`, `Match[T]` (string type checked) | +| Callables | `Callable` (arg/return types **not** checked) | +| Generators | `Generator`, `AsyncGenerator` (yield/send/return types **not** checked) | +| Async | `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator` | +| Context managers | `ContextManager`, `AsyncContextManager` | +| Protocols | `SupportsInt`, `SupportsFloat`, `SupportsComplex`, `SupportsBytes`, `SupportsIndex`, `SupportsAbs`, `SupportsRound` | +| ABCs | `Iterator`, `Iterable`, `Reversible`, `Container`, `Collection`, `Hashable`, `Sized` | + +**Not supported:** `Protocol` (structural subtyping), `TypedDict`, `Generic`, `ForwardRef`. Specific `NamedTuple` subclasses work via the `isinstance` fallback. Some examples: @@ -4052,11 +4081,11 @@ Some examples: import typing from unpythonic import isoftype -# concrete types - uninteresting, we just delegate to `isinstance` +# concrete types — just delegates to isinstance assert isoftype(17, int) assert isoftype(lambda: ..., typing.Callable) -# typing.newType +# typing.NewType UserId = typing.NewType("UserId", int) assert isoftype(UserId(42), UserId) # Note limitation: since NewType types discard their type information at @@ -4100,16 +4129,26 @@ assert isoftype({1: "foo", 2: "bar"}, typing.MutableMapping[int, str]) assert isoftype((1, 2, 3), typing.Sequence[int]) assert isoftype({1, 2, 3}, typing.AbstractSet[int]) +# new in 2.0.0 +assert isoftype(200, typing.Literal[200, 404, 500]) +assert isoftype(int, typing.Type[int]) +assert isoftype(bool, typing.Type[int]) # bool is a subclass of int +import collections +assert isoftype(collections.Counter("hello"), typing.Counter[str]) +import re +assert isoftype(re.compile(r"\d+"), typing.Pattern[str]) +import io +assert isoftype(io.StringIO("hi"), typing.TextIO) +assert isoftype(io.BytesIO(b"hi"), typing.BinaryIO) + # one-trick ponies assert isoftype(3.14, typing.SupportsRound) assert isoftype([1, 2, 3], typing.Sized) ``` -See [the unit tests](../unpythonic/tests/test_typecheck.py) for more. - -**CAUTION**: Callables are just checked for being callable; no further analysis is done. Type-checking callables properly requires a much more complex type checker. +See [the unit tests](../unpythonic/tests/test_typecheck.py) for the full set of supported features. -**CAUTION**: The `isoftype` function is one big hack. In Python 3.6 through 3.9, there is no consistent way to handle a type specification at run time. We must access some private attributes of the `typing` meta-utilities, because that seems to be the only way to get what we need to do this. +**CAUTION**: For types where the type parameters describe behavior rather than stored data — `Callable`, `Generator`, `AsyncGenerator`, `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator` — only the ABC is checked. The type parameters (argument types, yield/send/return types, etc.) are silently ignored, because checking them would require consuming or invoking the value. #### Notes From b221cef0d591147e32aedd15f273654dd940d55e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 16:39:49 +0200 Subject: [PATCH 820/832] move D4 set 3 to GitHub issue #98; remove resolved D7 Co-Authored-By: Claude Opus 4.6 --- TODO_DEFERRED.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 0c4e7999..3cf9bc4f 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -2,19 +2,6 @@ Next unused item code: D8 -- **D4**: `typecheck.py` — expand runtime type checker to support more `typing` features. - - **Set 1 — Easy wins**: DONE (`665cc4b`) - **Set 2 — Useful for dispatch**: DONE (this commit) - - `NamedTuple`: specific subclasses already work via `isinstance` fallback; no special handling needed. - **Dispatch integration tests** for Sets 1 and 2: DONE (this commit) - - **Set 3 — Hard / questionable value** (defer or discuss): - - `Protocol` — full structural subtyping, heavy - - `TypedDict` — required vs. optional keys, medium-hard - - `Generic` — abstract, unclear semantics for value checking - - `ForwardRef` — needs a namespace to resolve the string - - **D5**: `typecheck.py` / `dispatch.py` — parametric forms of existing one-trick ponies (e.g. `Iterator[int]`, `Iterable[str]`) raise `NotImplementedError`. The bare forms work. The `NotImplementedError` is arguably correct fail-fast behavior, since ignoring the type arg would silently accept wrong element types and make dispatching on e.g. `Iterable[int]` vs. `Iterable[float]` silently misroute. Same situation already exists for `Callable`, `Generator`, `ContextManager`, and async types. Possible improvements (dispatch layer, not typecheck): - Emit a warning when a type arg is silently ignored during dispatch. - Raise `TypeError` when registering indistinguishable multimethods (e.g. `Iterable[int]` then `Iterable[float]`). From dd30ba21e091c45f64f75e4042cb9e7369c992c7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Thu, 12 Mar 2026 17:27:54 +0200 Subject: [PATCH 821/832] typecheck: support TypedDict, Protocol, parametric abstract ABCs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TypedDict: structural checking of required/optional keys and value types via is_typeddict + __required_keys__/__optional_keys__ + get_type_hints. Supports total=False, inheritance, compound types. Protocol: @runtime_checkable Protocols delegate to isinstance; non-runtime-checkable raise TypeError with actionable message. Uses _is_protocol for detection (issubclass gives false positives on Python 3.10). Parametric abstract ABCs (D5 typecheck layer): Iterable[T], Collection[T], Reversible[T] do best-effort element checking — elements checked when value is Sized (concrete collection), ABC-only for opaque iterators. Iterator[T] and Container[T] accept parametric form with type arg silently ignored. Hashable and Sized remain non-generic. Unified get_origin approach for all. Dispatch-layer improvements (indistinguishable multimethod detection) deferred to #99. Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 3 + TODO_DEFERRED.md | 5 +- doc/features.md | 35 ++++++++-- unpythonic/tests/test_dispatch.py | 23 +++++++ unpythonic/tests/test_typecheck.py | 105 +++++++++++++++++++++++++++++ unpythonic/typecheck.py | 79 ++++++++++++++++++---- 6 files changed, 229 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f9baf739..4eb8cbf8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,9 @@ - New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the testset warning count. - Missing optional dependencies (sympy, mpmath) in tests emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. - Runtime type checker (`unpythonic.typecheck`): new supported typing features — `NoReturn`, `Never` (3.11+), `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO` (mapped to `io` module ABCs), `Pattern[T]`/`Match[T]` (string type checked when parametric), `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator`, `Generator`, `AsyncGenerator`. +- Runtime type checker: `TypedDict` support — structural checking of required/optional keys and value types. +- Runtime type checker: `Protocol` support — `@runtime_checkable` Protocols work via `isinstance`; non-runtime-checkable Protocols raise `TypeError` with an actionable message. +- Runtime type checker: parametric forms of abstract ABCs — `Iterable[T]`, `Collection[T]`, `Reversible[T]` perform best-effort element checking (elements checked when value is `Sized`; ABC-only for opaque iterators). `Iterator[T]` and `Container[T]` accept parametric form with type arg silently ignored. **Fixed**: diff --git a/TODO_DEFERRED.md b/TODO_DEFERRED.md index 3cf9bc4f..2f5d4681 100644 --- a/TODO_DEFERRED.md +++ b/TODO_DEFERRED.md @@ -2,8 +2,5 @@ Next unused item code: D8 -- **D5**: `typecheck.py` / `dispatch.py` — parametric forms of existing one-trick ponies (e.g. `Iterator[int]`, `Iterable[str]`) raise `NotImplementedError`. The bare forms work. The `NotImplementedError` is arguably correct fail-fast behavior, since ignoring the type arg would silently accept wrong element types and make dispatching on e.g. `Iterable[int]` vs. `Iterable[float]` silently misroute. Same situation already exists for `Callable`, `Generator`, `ContextManager`, and async types. Possible improvements (dispatch layer, not typecheck): - - Emit a warning when a type arg is silently ignored during dispatch. - - Raise `TypeError` when registering indistinguishable multimethods (e.g. `Iterable[int]` then `Iterable[float]`). - (Discovered during D4 Set 2 work.) +- **D5**: `dispatch.py` — moved to GitHub issue #99. Dispatch-layer improvements for parametric ABCs (warn/error on indistinguishable multimethods). Typecheck-layer part resolved. diff --git a/doc/features.md b/doc/features.md index 40766340..7eaf06e1 100644 --- a/doc/features.md +++ b/doc/features.md @@ -3731,7 +3731,7 @@ The core idea can be expressed in fewer than 100 lines of Python; ours is (as of ### `generic`, `typed`, `isoftype`: multiple dispatch -**Changed in v2.0.0.** *`isoftype` now supports many more `typing` features: `NoReturn`, `Never`, `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO`, `Pattern`/`Match`, `ContextManager`/`AsyncContextManager`, `Awaitable`/`Coroutine`, `AsyncIterable`/`AsyncIterator`, `Generator`/`AsyncGenerator`. See the [`isoftype` section](#isoftype-the-big-sister-of-isinstance) for the full list.* +**Changed in v2.0.0.** *`isoftype` now supports many more `typing` features: `NoReturn`, `Never`, `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO`, `Pattern`/`Match`, `ContextManager`/`AsyncContextManager`, `Awaitable`/`Coroutine`, `AsyncIterable`/`AsyncIterator`, `Generator`/`AsyncGenerator`, `TypedDict`, `@runtime_checkable` `Protocol`, and parametric forms of abstract ABCs (`Iterable[T]`, `Collection[T]`, `Reversible[T]` with best-effort element checking; `Iterator[T]`, `Container[T]`). See the [`isoftype` section](#isoftype-the-big-sister-of-isinstance) for the full list.* **Changed in v0.15.0**. *The `dispatch` and `typecheck` modules providing this functionality are now considered stable (no longer experimental). Starting with this release, they receive the same semantic-versioning guarantees as the rest of `unpythonic`.* @@ -4071,9 +4071,13 @@ Any checks on the type arguments of the meta-utilities defined in the `typing` s | Async | `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator` | | Context managers | `ContextManager`, `AsyncContextManager` | | Protocols | `SupportsInt`, `SupportsFloat`, `SupportsComplex`, `SupportsBytes`, `SupportsIndex`, `SupportsAbs`, `SupportsRound` | -| ABCs | `Iterator`, `Iterable`, `Reversible`, `Container`, `Collection`, `Hashable`, `Sized` | +| Protocol (user) | `@runtime_checkable` Protocol subclasses (structural subtyping via `isinstance`) | +| TypedDict | Structural check: required/optional keys, value types recursively checked | +| ABCs (best-effort) | `Iterable[T]`, `Collection[T]`, `Reversible[T]` (elements checked when value is `Sized`; ABC-only when not) | +| ABCs (type arg ignored) | `Iterator[T]`, `Container[T]` (parametric form accepted, type arg silently ignored) | +| ABCs (non-generic) | `Hashable`, `Sized` | -**Not supported:** `Protocol` (structural subtyping), `TypedDict`, `Generic`, `ForwardRef`. Specific `NamedTuple` subclasses work via the `isinstance` fallback. +**Not supported:** `Generic`, `ForwardRef`. Specific `NamedTuple` subclasses work via the `isinstance` fallback. Non-`@runtime_checkable` Protocols raise `TypeError` with an actionable message. Some examples: @@ -4144,11 +4148,34 @@ assert isoftype(io.BytesIO(b"hi"), typing.BinaryIO) # one-trick ponies assert isoftype(3.14, typing.SupportsRound) assert isoftype([1, 2, 3], typing.Sized) + +# best-effort element checking for abstract iterables +assert isoftype([1, 2, 3], typing.Iterable[int]) # concrete → elements checked +assert not isoftype([1, 2, 3], typing.Iterable[str]) # wrong element type +assert isoftype(iter([1, 2, 3]), typing.Iterable[int]) # opaque iterator → ABC only + +# TypedDict — structural checking of keys and value types +class Point(typing.TypedDict): + x: float + y: float +assert isoftype({"x": 1.0, "y": 2.0}, Point) +assert not isoftype({"x": 1.0}, Point) # missing required key + +# Protocol (must be @runtime_checkable) +@typing.runtime_checkable +class Drawable(typing.Protocol): + def draw(self) -> None: ... +class Circle: + def draw(self): + pass +assert isoftype(Circle(), Drawable) ``` See [the unit tests](../unpythonic/tests/test_typecheck.py) for the full set of supported features. -**CAUTION**: For types where the type parameters describe behavior rather than stored data — `Callable`, `Generator`, `AsyncGenerator`, `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator` — only the ABC is checked. The type parameters (argument types, yield/send/return types, etc.) are silently ignored, because checking them would require consuming or invoking the value. +**CAUTION**: For types where the type parameters describe behavior rather than stored data — `Callable`, `Generator`, `AsyncGenerator`, `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator`, `Iterator`, `Container` — only the ABC is checked. The type parameters are silently ignored, because checking them would require consuming or invoking the value. + +For `Iterable[T]`, `Collection[T]`, and `Reversible[T]`, element types are checked **best-effort**: if the value is `Sized` (a concrete collection like `list`, `set`, etc.), elements are checked; if it's an opaque iterator, only the ABC is checked. Empty concrete collections reject parametric specs (consistent with `List[T]`, `Sequence[T]`, etc.). #### Notes diff --git a/unpythonic/tests/test_dispatch.py b/unpythonic/tests/test_dispatch.py index 355bc92a..e75e513b 100644 --- a/unpythonic/tests/test_dispatch.py +++ b/unpythonic/tests/test_dispatch.py @@ -468,6 +468,29 @@ def mygen(): test[classify(contextlib.nullcontext()) == "context manager"] test[classify(42) == "int"] + with testset("@generic with Iterable dispatch"): + # Best-effort element checking: concrete collections dispatch correctly. + @generic + def process_items(x: typing.Iterable[int]): + return "ints" + @generic + def process_items(x: typing.Iterable[str]): # noqa: F811 + return "strs" + test[process_items([1, 2, 3]) == "ints"] + test[process_items(["a", "b"]) == "strs"] + test[process_items((1, 2)) == "ints"] + test[process_items({"hello", "world"}) == "strs"] + + with testset("@generic with Collection dispatch"): + @generic + def summarize(x: typing.Collection[int]): + return f"collection of {len(list(x))} ints" + @generic + def summarize(x: typing.Collection[str]): # noqa: F811 + return f"collection of {len(list(x))} strs" + test[summarize([1, 2, 3]) == "collection of 3 ints"] + test[summarize(["a", "b"]) == "collection of 2 strs"] + if __name__ == '__main__': # pragma: no cover with session(__file__): runtests() diff --git a/unpythonic/tests/test_typecheck.py b/unpythonic/tests/test_typecheck.py index e1d2d933..3b8af53f 100644 --- a/unpythonic/tests/test_typecheck.py +++ b/unpythonic/tests/test_typecheck.py @@ -264,6 +264,111 @@ def runtests(): test[isoftype([1, 2, 3], typing.Container)] test[isoftype([1, 2, 3], typing.Collection)] # Sized Iterable Container + with testset("parametric ABCs — uncheckable (type arg ignored)"): + # Iterator: consumed by iteration, can't check elements. + test[isoftype(iter([1, 2, 3]), typing.Iterator[int])] + test[isoftype(iter([1, 2, 3]), typing.Iterator[str])] # type arg ignored + test[not isoftype(42, typing.Iterator[int])] + + # Container: only has __contains__, can't enumerate elements. + test[isoftype([1, 2, 3], typing.Container[int])] + test[isoftype([1, 2, 3], typing.Container[str])] # type arg ignored + test[not isoftype(42, typing.Container[int])] + + with testset("parametric ABCs — best-effort element checking"): + # Iterable[T]: elements checked when value is Sized (concrete collection). + test[isoftype([1, 2, 3], typing.Iterable[int])] + test[not isoftype([1, 2, 3], typing.Iterable[str])] + test[not isoftype([], typing.Iterable[int])] # empty rejects parametric + test[isoftype([], typing.Iterable)] # bare form still accepts empty + test[not isoftype(42, typing.Iterable[int])] + # Opaque iterator (not Sized) — accepts on ABC alone, can't check elements. + test[isoftype(iter([1, 2, 3]), typing.Iterable[int])] + test[isoftype(iter([1, 2, 3]), typing.Iterable[str])] # can't check, accepts + + # Collection[T]: Sized + Iterable + Container. + test[isoftype([1, 2, 3], typing.Collection[int])] + test[not isoftype([1, 2, 3], typing.Collection[str])] + test[not isoftype([], typing.Collection[int])] # empty rejects parametric + test[isoftype([], typing.Collection)] # bare form accepts empty + test[not isoftype(42, typing.Collection[int])] + + # Reversible[T] + test[isoftype([1, 2, 3], typing.Reversible[int])] + test[not isoftype([1, 2, 3], typing.Reversible[str])] + test[not isoftype([], typing.Reversible[int])] # empty rejects parametric + test[isoftype([], typing.Reversible)] # bare form accepts empty + test[not isoftype(42, typing.Reversible[int])] + + # Compound type in element spec + test[isoftype([1, "two", 3], typing.Iterable[typing.Union[int, str]])] + test[not isoftype([1, "two", 3.0], typing.Iterable[typing.Union[int, str]])] + + with testset("typing.TypedDict"): + class Point(typing.TypedDict): + x: float + y: float + + test[isoftype({"x": 1.0, "y": 2.0}, Point)] + test[not isoftype({"x": 1.0}, Point)] # missing required key + test[not isoftype({"x": 1.0, "y": 2.0, "z": 3.0}, Point)] # extra key + test[not isoftype({"x": "hello", "y": 2.0}, Point)] # wrong value type + test[not isoftype(42, Point)] # not a dict + test[not isoftype([], Point)] # not a dict + + # total=False: all keys optional + class Config(typing.TypedDict, total=False): + debug: bool + verbose: bool + + test[isoftype({}, Config)] # all optional, empty is ok + test[isoftype({"debug": True}, Config)] + test[isoftype({"debug": True, "verbose": False}, Config)] + test[not isoftype({"debug": "yes"}, Config)] # wrong type + test[not isoftype({"unknown": True}, Config)] # extra key + + # Inheritance + class Base(typing.TypedDict): + name: str + + class Derived(Base): + age: int + + test[isoftype({"name": "alice", "age": 30}, Derived)] + test[not isoftype({"name": "alice"}, Derived)] # missing age + test[not isoftype({"age": 30}, Derived)] # missing name + + # Compound value types + class Nested(typing.TypedDict): + tags: typing.List[str] + count: typing.Optional[int] + + test[isoftype({"tags": ["a", "b"], "count": 42}, Nested)] + test[isoftype({"tags": ["a"], "count": None}, Nested)] + test[not isoftype({"tags": [1, 2], "count": 42}, Nested)] # wrong list element type + + with testset("typing.Protocol"): + @typing.runtime_checkable + class Drawable(typing.Protocol): + def draw(self) -> None: ... + + class Circle: + def draw(self): + pass + + class Square: + pass + + test[isoftype(Circle(), Drawable)] + test[not isoftype(Square(), Drawable)] + test[not isoftype(42, Drawable)] + + # Non-runtime-checkable Protocol raises TypeError + class NonCheckable(typing.Protocol): + def frobnicate(self) -> int: ... + + test_raises[TypeError, isoftype(Circle(), NonCheckable)] + with testset("typing.KeysView, typing.ValuesView, typing.ItemsView"): d = {17: "cat", 23: "fox", 42: "python"} test[isoftype(d.keys(), typing.KeysView[int])] diff --git a/unpythonic/typecheck.py b/unpythonic/typecheck.py index 1145440d..09ec0014 100644 --- a/unpythonic/typecheck.py +++ b/unpythonic/typecheck.py @@ -63,6 +63,12 @@ def isoftype(value, T): - `Awaitable[T]`, `Coroutine[T1, T2, T3]` - `AsyncIterable[T]`, `AsyncIterator[T]` - `Generator[Y, S, R]`, `AsyncGenerator[Y, S]` + - `Iterable[T]`, `Collection[T]`, `Reversible[T]` (best-effort element + checking: elements checked when value is ``Sized``; ABC-only when not) + - `Iterator[T]`, `Container[T]` (parametric form accepted; type arg ignored) + - `Hashable`, `Sized` (non-generic; bare form only) + - `TypedDict` (structural check: required/optional keys, value types) + - ``@runtime_checkable`` ``Protocol`` subclasses - `Text` (deprecated since Python 3.11; will be removed at floor Python 3.12) Any checks on the type arguments of the meta-utilities are performed @@ -83,7 +89,7 @@ def isoftype(value, T): # # Unsupported typing features: # NamedTuple (specific NamedTuple subclasses work via isinstance fallback), - # Generic, Protocol, TypedDict, ForwardRef + # Generic, ForwardRef if T is typing.Any: return True @@ -144,18 +150,35 @@ def isNewType(T): return True # bare ClassVar or Final, no inner type constraint return isoftype(value, args[0]) - # Some one-trick ponies. - for U in (typing.Iterator, # can't non-destructively check element type - typing.Iterable, # can't non-destructively check element type - typing.Container, # can't check element type - typing.Collection, # Sized Iterable Container; can't check element type - typing.Hashable, - typing.Sized): - if U is T: - return isinstance(value, U) - - if T is typing.Reversible: # can't non-destructively check element type - return isinstance(value, typing.Reversible) + # Non-generic ABCs, and parametric ABCs where element type can't be checked. + # Iterator: consumed by iteration. Container: only has __contains__, can't enumerate. + # Hashable, Sized: not generic (can't be parameterized). + for abc in (collections.abc.Hashable, + collections.abc.Sized, + collections.abc.Iterator, + collections.abc.Container): + if typing.get_origin(T) is abc: + return isinstance(value, abc) + + # Parametric ABCs with best-effort element checking. + # If the value is Sized (a concrete collection), we can safely iterate + # and check elements. Otherwise (opaque iterator), accept on ABC alone. + for abc in (collections.abc.Iterable, + collections.abc.Collection, + collections.abc.Reversible): + if typing.get_origin(T) is abc: + if not isinstance(value, abc): + return False + args = getattr(T, "__args__", None) + if args is None: + return True # bare form, no element type constraint + assert len(args) == 1 + if not isinstance(value, collections.abc.Sized): + return True # opaque iterator — can't check elements non-destructively + if not value: # empty sized collection has no element type + return False + U = args[0] + return all(isoftype(elt, U) for elt in value) # "Protocols cannot be used with isinstance()", so: for U in (typing.SupportsInt, @@ -168,6 +191,24 @@ def isNewType(T): if U is T: return safeissubclass(type(value), U) + # TypedDict — structural check on dict contents. + # isinstance doesn't work with TypedDict, so we check keys and value types. + if typing.is_typeddict(T): + if not isinstance(value, dict): + return False + hints = typing.get_type_hints(T) + required = T.__required_keys__ + optional = T.__optional_keys__ + allowed = required | optional + if not required.issubset(value.keys()): + return False + if not set(value.keys()).issubset(allowed): + return False + for k, v in value.items(): + if not isoftype(v, hints[k]): + return False + return True + # We don't have a match yet, so T might still be one of those meta-utilities # that hate `issubclass` with a passion. # DEPRECATED: typing.Text is deprecated since Python 3.11 (it's just an alias for str). @@ -369,6 +410,18 @@ def iscollection(statictype, runtimetype): # return False # return True + # Protocol — support @runtime_checkable Protocols; clear error for others. + # Specific Protocols (Supports* ABCs) are already handled above by identity check. + # We use _is_protocol (not issubclass) because issubclass(X, Protocol) returns + # True for some non-Protocol types (e.g. int) on Python 3.10. + if isinstance(T, type) and T is not typing.Protocol and getattr(T, '_is_protocol', False): + if getattr(T, '_is_runtime_protocol', False): + return isinstance(value, T) + raise TypeError( + f"isoftype: {T.__qualname__} is a Protocol but not @typing.runtime_checkable, " + f"so runtime structural checks are not possible. " + f"Add @typing.runtime_checkable to enable isinstance checks.") + # Catch any `typing` meta-utilities we don't currently support. if hasattr(T, "__module__") and T.__module__ == "typing": # pragma: no cover, only happens when something goes wrong. fullname = repr(T.__class__) From 60034a7e4c3c058eb8b6db0dae8e6165be9d7f48 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Fri, 13 Mar 2026 13:41:34 +0200 Subject: [PATCH 822/832] Document PDM venv management in CLAUDE.md Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 16ccda0a..6db9bfe2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,16 @@ pdm use --venv in-project source .venv/bin/activate ``` +The project venv is managed by PDM (`pdm venv create`, `pdm use --venv in-project`). To switch Python versions, remove the old venv and create a new one: + +```bash +pdm venv remove in-project +pdm config venv.in_project true +pdm venv create 3.14 # or whichever version +pdm use --venv in-project +pdm install +``` + **Critical**: Never compile `.py` files in this project using `py_compile`, `python -m compileall`, `--compile`, or any other mechanism that bypasses the macro expander. Stale `.pyc` files compiled without macro support will break macro imports (symptom: `ImportError: cannot import name 'macros' from 'mcpyrate.quotes'`). If this happens, clean the caches with `macropython -c unpythonic` and re-run. ## Running tests From 746ad51a05e702a4874e5d247e93301f82304ff2 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:13:04 +0200 Subject: [PATCH 823/832] mcpyrate 4.0.0 upgrade consequences: add actionable advice --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4eb8cbf8..28075244 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ - **Python version support**: 3.10–3.14 (dropped 3.8, 3.9; added 3.13, 3.14). PyPy 3.11. - If you need `unpythonic` for Python 3.8 or 3.9, use version 1.0.0. - **Requires mcpyrate >= 4.0.0**. - - mcpyrate 4.0.0 dropped the `Str`, `Num`, `NameConstant` AST compatibility shims and the `getconstant` helper. + - mcpyrate 4.0.0 dropped the `Str`, `Num`, `NameConstant` AST compatibility shims and the `getconstant` helper. Use `ast.Constant` directly, and `.value` to get the constant's value. **New**: From f08c2be2cf4ba9b0fdff046bfcb050cb706a7890 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:14:06 +0200 Subject: [PATCH 824/832] changelog emit_warning: more accurate wording --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 28075244..67869aba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ - `autoreturn` macro now handles `match`/`case` statements. Each case branch has its own tail position. - New scope analyzer tests for `match`/`case` patterns and `try`/`except*`. - New `unpythonic.test.runner` module: reusable test runner with module discovery, version-suffix gating (e.g. `test_foo_3_11.py` skipped on Python < 3.11), and integration with the test framework's warning system. Other projects using `unpythonic.test.fixtures` can import it directly. -- New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the testset warning count. +- New `emit_warning()` function in `unpythonic.test.fixtures` for signaling test warnings from infrastructure code (outside `test[]`/`warn[]` macros). Used by the test runner for version-suffix skips, which show in the warning count for the innermost enclosing testset. - Missing optional dependencies (sympy, mpmath) in tests emit `warn[]` instead of `error[]`, correctly reflecting that these are expected skips, not failures. - Runtime type checker (`unpythonic.typecheck`): new supported typing features — `NoReturn`, `Never` (3.11+), `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO` (mapped to `io` module ABCs), `Pattern[T]`/`Match[T]` (string type checked when parametric), `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator`, `Generator`, `AsyncGenerator`. - Runtime type checker: `TypedDict` support — structural checking of required/optional keys and value types. From 8ecc41f818904e5e9104771975744a07106188ef Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:16:49 +0200 Subject: [PATCH 825/832] changelog wording --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67869aba..06978b8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ **Fixed**: -- Runtime type checker (`unpythonic.typecheck`): fixed compatibility with Python 3.14, where `typing.Union` is no longer a `_GenericAlias`. Now uses `typing.get_origin` (available since 3.8) instead of a local copy. +- Runtime type checker (`unpythonic.typecheck`): fixed compatibility with Python 3.14, where `typing.Union` is no longer a `_GenericAlias`. Now uses `typing.get_origin` (available since 3.8). - Runtime type checker: fixed `TypeVar` detection to use `isinstance(T, typing.TypeVar)` instead of a fragile `repr`-based heuristic. - Runtime type checker: `typing.Reversible` check now uses `isinstance` instead of a `hasattr("__reversed__")` workaround from the Python 3.5 era. - Runtime type checker: removed redundant `safeissubclass` fallbacks for generic types — `typing.get_origin` handles both bare and parameterized generics on 3.10+. From bf6c1c270efa2ec5e6f413611250bef33b9d2664 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:23:09 +0200 Subject: [PATCH 826/832] changelog wording --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 06978b8f..001a83cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,15 +28,16 @@ - Runtime type checker: fixed `TypeVar` detection to use `isinstance(T, typing.TypeVar)` instead of a fragile `repr`-based heuristic. - Runtime type checker: `typing.Reversible` check now uses `isinstance` instead of a `hasattr("__reversed__")` workaround from the Python 3.5 era. - Runtime type checker: removed redundant `safeissubclass` fallbacks for generic types — `typing.get_origin` handles both bare and parameterized generics on 3.10+. -- Scope analyzer: fixed `MatchCapturesCollector` bug where class references (e.g. `Point` in `case Point(x, y):`) were incorrectly collected as captured variable names. Match captures are `MatchAs`/`MatchStar` nodes with bare strings, not `Name` nodes. +- Scope analyzer: fixed `MatchCapturesCollector` bug where class references (e.g. `Point` in `case Point(x, y):`) were incorrectly collected as captured variable names. - Macro layer: updated all `hasattr(tree, "ctx")` checks to use `getattr` with defaults, for correct behavior on Python 3.13+ where AST fields always exist with default values. + - **Important**: Since Python 3.13, the default of `ctx` is `Load()`, hence no AST node has its `ctx` in a "not set yet" state anymore. Hence, any macro-created `Name` nodes that appear in a `Store` or `Del` position **MUST** have their `ctx` set appropriately by the macro author. Failing to do so **will** cause mysterious errors during macro expansion. - Macro layer: updated `arguments()` constructor calls to always include `posonlyargs=[]`, avoiding a `DeprecationWarning` on Python 3.13 (will become an error in 3.15). - MS Windows: `unpythonic.net.util` failed to load, due to missing `termios` module (which is *nix only) being loaded by `unpythonic.net.__init__` when it imports `unpythonic.net.ptyproxy`. - - Fixed by catching `ModuleNotFoundError`, disabling `ptyproxy` on MS Windows systems. + - Fixed by catching `ModuleNotFoundError`, disabling `ptyproxy` on MS Windows systems. Thus the remote REPL functionality `unpythonic.net.client/server` is not available on MS Windows, but the rest of `unpythonic` works fine. **Deprecated**: -- Parenthesis syntax for macro arguments (e.g. `let((x, 1), (y, 2))`). Use bracket syntax instead: `let[[x, 1], [y, 2]]`. The parenthesis syntax is kept for backward compatibility but may be removed in a future version. +- Parenthesis syntax for macro arguments (e.g. `let((x, 1), (y, 2))`). Use bracket syntax instead: `let[[x, 1], [y, 2]]`. The parenthesis syntax is kept for backward compatibility for now. - Runtime type checker: `typing.Text` (deprecated since Python 3.11) and `typing.ByteString` (deprecated since Python 3.12) support is now marked for removal when the floor bumps to Python 3.12. From fcc7b208fabe4afda3ff67ade17749d9505dbd6e Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:23:41 +0200 Subject: [PATCH 827/832] Update changelog: clarify Iterator/Container rationale, polish wording Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 001a83cf..34196372 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ - Runtime type checker (`unpythonic.typecheck`): new supported typing features — `NoReturn`, `Never` (3.11+), `Literal`, `Type`, `ClassVar`, `Final`, `DefaultDict`, `OrderedDict`, `Counter`, `ChainMap`, `IO`/`TextIO`/`BinaryIO` (mapped to `io` module ABCs), `Pattern[T]`/`Match[T]` (string type checked when parametric), `ContextManager`, `AsyncContextManager`, `Awaitable`, `Coroutine`, `AsyncIterable`, `AsyncIterator`, `Generator`, `AsyncGenerator`. - Runtime type checker: `TypedDict` support — structural checking of required/optional keys and value types. - Runtime type checker: `Protocol` support — `@runtime_checkable` Protocols work via `isinstance`; non-runtime-checkable Protocols raise `TypeError` with an actionable message. -- Runtime type checker: parametric forms of abstract ABCs — `Iterable[T]`, `Collection[T]`, `Reversible[T]` perform best-effort element checking (elements checked when value is `Sized`; ABC-only for opaque iterators). `Iterator[T]` and `Container[T]` accept parametric form with type arg silently ignored. +- Runtime type checker: parametric forms of abstract ABCs — `Iterable[T]`, `Collection[T]`, `Reversible[T]` perform best-effort element checking (elements checked when value is `Sized`; ABC-only for opaque iterators). `Iterator[T]` and `Container[T]` accept parametric form with type arg silently ignored (iterating an `Iterator` would consume it; `Container` only has `__contains__`, so elements can't be enumerated). **Fixed**: From cb5c6cf319b667dd88277f40f3631675a66787c6 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:24:34 +0200 Subject: [PATCH 828/832] Set changelog date for 2.0.0 release Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34196372..350f46e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -**2.0.0** (March 2026, in progress) — *"Six impossible things before breakfast"* edition: +**2.0.0** (16 March 2026) — *"Six impossible things before breakfast"* edition: **IMPORTANT**: From 017fd48fa669dcbbf5fb2c1ea6431a662980d1e7 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:26:34 +0200 Subject: [PATCH 829/832] Fix mcpyrate dependency: use PyPI package, not local path The local file path dependency was a development convenience that PyPI rightly rejects in sdists. Co-Authored-By: Claude Opus 4.6 (1M context) --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index e6808283..6c1b435f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,7 @@ license = { text = "BSD" } dynamic = ["version"] dependencies = [ - "mcpyrate @ file:///home/jje/Documents/koodit/mcpyrate", + "mcpyrate>=4.0.0", "sympy>=1.13" ] keywords=["functional-programming", "language-extension", "syntactic-macros", From 8c7d004f42c70409327d23d7eee9d572833c3d24 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:27:58 +0200 Subject: [PATCH 830/832] Bump version to 2.0.1-dev for post-release development Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 7 +++++++ unpythonic/__init__.py | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 350f46e5..a8abc0b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +**2.0.1** (in progress): + +*No user-visible changes yet.* + + +--- + **2.0.0** (16 March 2026) — *"Six impossible things before breakfast"* edition: **IMPORTANT**: diff --git a/unpythonic/__init__.py b/unpythonic/__init__.py index 48e202e4..fc4b8af6 100644 --- a/unpythonic/__init__.py +++ b/unpythonic/__init__.py @@ -7,7 +7,7 @@ for a trip down the rabbit hole. """ -__version__ = '2.0.0' +__version__ = '2.0.1-dev' from .amb import * # noqa: F401, F403 from .arity import * # noqa: F401, F403 From 8a85a61f0af9a240f5432eb7bcd055f42da646e0 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:40:14 +0200 Subject: [PATCH 831/832] Split lint into its own job on Python 3.14 Flake8 must run on a Python that supports all syntax in the codebase (e.g. except* requires 3.11+). The test matrix still covers 3.10+. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/python-package.yml | 29 ++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index f6f59be3..b43fefbd 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -12,8 +12,28 @@ on: branches: [ master ] jobs: - build: + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + - name: Set up Python + uses: actions/setup-python@v6 + with: + # Use latest Python so flake8 can parse all syntax (e.g. except* requires 3.11+) + python-version: "3.14" + - name: Install flake8 + run: | + python -m pip install --upgrade pip + pip install flake8 + - name: Lint with flake8 + run: | + # stop the build if there are Python syntax errors or undefined names + flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics + test: + needs: lint runs-on: ubuntu-latest strategy: matrix: @@ -28,14 +48,7 @@ jobs: - name: Install tools in CI venv run: | python -m pip install --upgrade pip - pip install flake8 pip install pdm - - name: Lint with flake8 - run: | - # stop the build if there are Python syntax errors or undefined names - flake8 . --config=flake8rc --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --config=flake8rc --count --exit-zero --max-complexity=100 --max-line-length=127 --statistics - name: Determine Python version string for PDM run: | echo "TARGET_PYTHON_VERSION_FOR_PDM=${{ matrix.python-version }}" | tr - @ >> "$GITHUB_ENV" From 64ab1fda57213c52afae25201a83f0b7f4d60a33 Mon Sep 17 00:00:00 2001 From: Juha Jeronen Date: Mon, 16 Mar 2026 12:47:35 +0200 Subject: [PATCH 832/832] Add workflow_dispatch trigger to CI workflows Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/coverage.yml | 1 + .github/workflows/python-package.yml | 1 + 2 files changed, 2 insertions(+) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index e54e686d..806df340 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -8,6 +8,7 @@ name: Coverage on: push: branches: [ master ] + workflow_dispatch: jobs: codecov: diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index b43fefbd..cd906fad 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -10,6 +10,7 @@ on: branches: [ master ] pull_request: branches: [ master ] + workflow_dispatch: jobs: lint: