Extended-Raw REPL Mode Proposal #18520

alonbl · 2025-12-06T20:16:55Z

alonbl
Dec 6, 2025

Extended-Raw REPL Mode Proposal

Summary

MicroPython currently provides two special REPL protocols: paste mode and raw mode. Paste mode is optimized for human input, while raw mode is optimized for machine-driven execution.

However, when the REPL runs over unreliable transports—particularly serial links with possible transmission or receive errors—neither mode provides any mechanism for framing, error detection, or recovery. Higher-level tools built on top of REPL (e.g., file transfer utilities such as #18420) are therefore vulnerable to corruption and desynchronization.

This proposal introduces a new, optional extended-raw mode that adds structured framing, integrity checks, and support for reliable multi-channel communication.

Motivation

Machine communication with MicroPython devices commonly occurs over serial interfaces that may experience:

Transmission corruption
Dropped bytes
Repeated or reordered data
Loss of framing
Noise during boot or reset

In the current REPL design:

No CRC or checksum exists
No packet boundaries exist
No sequence numbering exists
No resynchronization mechanism exists

As a result:

Corrupted data may execute silently
REPL can desynchronize and become unrecoverable
Tools such as file transfer, remote execution, and debugging cannot achieve reliability
Multi-stream communication (stdin/stdout/stderr/control) cannot be multiplexed safely

A modern, structured transport mode would enable tooling and automation to be significantly more robust.

Proposed Solution: Extended-Raw Mode

Add a new REPL mode that encapsulates messages in well-defined packets with framing, length, sequence numbering, and CRC.

Packet Structure

uint32_t signature;   // fixed sync signature for resynchronization
uint32_t length;      // length of payload
uint8_t  payload[];
uint32_t crc32;       // checksum of payload

Payload Structure

uint32_t sequence;    // detects missing/out-of-order packets
uint32_t command;     // message type identifier
uint8_t  payload[];

Example command types

Exit extended-raw mode
ACK (for confirming receipt of a specific sequence number)
Code block
stdin block
stdout block
stderr block
Control commands (e.g., simulated )

Key Benefits

1. Reliable Communication

CRC and sequence numbers allow detection and recovery from errors, missing packets, and corruption.

2. Tooling Support

File transfer, remote execution, and debugging tools can operate reliably even on noisy links.

3. Multi-channel Communication

Enables clean separation of stdout, stderr, stdin, control signals, and metadata.

4. Automatic Resynchronization

The fixed signature allows clients to re-align with the stream after corruption.

Backward Compatibility

Extended-raw mode is optional.
Existing paste mode and raw mode remain unchanged.
Entering/exiting extended-raw mode mirrors raw mode behavior.

Conclusion

Adding extended-raw mode provides a robust, fault-tolerant transport layer for REPL-based communication. This significantly improves reliability for embedded development and automation workflows.

andrewleech · 2025-12-06T21:46:26Z

andrewleech
Dec 6, 2025
Collaborator Sponsor

I do like the idea of a more dependable transport! As flexible as raw repl is, yeah it's not always reliable enough for all applications.

It'll need a few more layers than just the packet structure though; the actual logic of what to do when a corrupted packet is detected is where the real complication lies.

Personally I've been starting to lean into using a network connection more and more for this kind of purpose, though I've been sticking to hardware with a real network port and working on a USB-Net impl.

A growing number of ports support having PPP enabled; this is the traditional gold standard of reliable serial protocols that has this stuff figured out. I've thought about maybe trying to add PPP support directly into mpremote to avoid needing to set up modem handling in the OS. I haven't system this though, and for seamless operation it'd need a RAW-PPP mode switch sequence to switch the repl transport into PPP mode and maybe provide a native REPL-over-socket handler to keep mpremote working. Other types of data could be sent over separate sockets then.
This has the added benefit of full networking support so you can run other services at the same time then.

If that still feels too much, I'd strongly prefer to find a library / protocol that's already proven trustworthy to provides handshaking, sync, corruption detection, retransmission etc. if that framing and reliable transport layer is proven and efficient, then your payload structure to provide a more structured way of sending different kinds of data could certainly prove useful!

0 replies

alonbl · 2025-12-06T22:19:55Z

alonbl
Dec 6, 2025
Author

Hi,

PPP is a good option if we enable IP stack on all devices and have REPL over TCP.
This will also allow multiplexing different stream, as any stream may be independent TCP session with no framing.
For example, opening a session to port 8000 will interact with the interpreter while port 8001 is stdin, port 8002 is stdout and 8003 is stderr.
It also simplify file transfer as filesystem may open its own service.

I am bit worried about security, but we can bind these services to the dedicated interface. Not sure how to consider the additional interface address, maybe we can have a dedicated loopback interface, aka 127.0.0.x.

This is larger change than I had in mind... it will provide a great flexibility.

Is lwip dependency acceptable for core? I guess some changes are required to support parallel ppp instances, one will be allocated to the REPL.

Thanks,

1 reply

andrewleech Dec 7, 2025
Collaborator Sponsor

Lwip won't be possible to enable on all devices, it does require at least 20KB of ram for itself iirc, and a decent amount of flash too.
But it's also reasonable (in my mind) to have these higher level features only enabled on higher spec parts.

karfas · 2025-12-07T09:04:00Z

karfas
Dec 7, 2025

I might be wrong, but I see the REPL as a tool for development, not for regularly transferring files from/to the device. In development mode, you have the device on the desk with some short (USB-?)cables.
The use cases seem to be quite limited. You will have to terminate the running application to use the super-reliable REPL, while the application might still need to provide it's own (secure/reliable) file transfer mechanics.

Besides that, inventing some new framing/protocol comes with drawbacks (backward compatibility, unreliable until development is real finished, tooling support).

OK, using ftp (or another TCP-based protocol) over SLIP or PPP to make the link reliable seems to be ruled out for devices with limited memory.
However, ZModem exists since ~40 years and is proven to work fast over unreliable links; I think most terminal programs designed for serial lines still have support for it.

0 replies

alonbl · 2025-12-07T11:39:29Z

alonbl
Dec 7, 2025
Author

Hello @karfas,

The REPL is not only used during application development, it also plays an important role in manufacturing, software updates, system bring-up, diagnostics, and customer support.

In many devices, USB is not available (either due to hardware constraints or because it is already allocated to the functional firmware). In such cases, the debug UART is the only available communication channel, and this is exactly how we operate today. Because of that, the robustness of REPL transport becomes critical.

While ZModem is indeed a strong option for file transfer, it doesn’t fully address the broader requirements of the REPL environment. The REPL provides:

code execution
stdin/stdout/stderr channels
control signals (e.g., interrupts)
interactive bidirectional communication

ZModem only solves the file transfer portion, not the multiplexed program-and-control flow the REPL already supports. This is why a ZModem-only solution is not a complete match.

The transport we really need is conceptually similar to what SSH provides:
a multiplexed, error-checked channel that carries both program data and control information. In our case, the design can be significantly simpler than SSH, but the underlying idea is similar, integrity + framing + multiple logical streams.

As noted in my earlier proposal, a lightweight framed protocol would provide:

error detection (CRC)
resynchronization after corruption
multi-channel support
sequence tracking and retransmission
deterministic behavior over unreliable serial links

Regarding backward compatibility:
I don’t foresee issues, since the existing modes (paste and raw) remain untouched. The extended mode would be opt-in and fully compatible with all current behavior.

As for adding a full TCP/IP stack just to achieve reliable console behavior, this feels quite heavy for the problem at hand. If that ends up being the preferred direction, I’m happy to help implement it.

Thanks,

3 replies

karfas Dec 7, 2025

I am still not convinced regarding the use cases.

Most points of your feature list is also provided by another lightweight protocol from the 1980s: See Kermit, even when the file transfer times with Kermit are cruel.

alonbl Dec 7, 2025
Author

Hi @karfas ,

I worked with Kermit as well... had to use the uuencode basic program to transfer binary over kermit... as the serial was 7bit at that time.

As much as I love history, I still do not understand how file transfer solves the usability of a control channel, which one of its function may be file transfer.

Thanks,

karfas Dec 7, 2025

This has nothing to do with "loving history" (I hated Kermit in these days). ZModem, Kermit etc. are designed to work on hardware similar to the medium sized embedded systems where we use Micropython today.
Using an established standard simply looks better for me than inventing a protocol completely unknown outside the Micropython universe.

I am pretty sure the control/command channel between two kermits is reliable (with CRC, Retransmits etc.), but I have no idea how to marry this with the REPL.

MicroPython

Extended-Raw REPL Mode Proposal #18520

Uh oh!

alonbl Dec 6, 2025

Extended-Raw REPL Mode Proposal

Summary

Motivation

Proposed Solution: Extended-Raw Mode

Packet Structure

Payload Structure

Example command types

Key Benefits

1. Reliable Communication

2. Tooling Support

3. Multi-channel Communication

4. Automatic Resynchronization

Backward Compatibility

Conclusion

Replies: 4 comments · 4 replies

Uh oh!

Uh oh!

andrewleech Dec 6, 2025 Collaborator Sponsor

Uh oh!

alonbl Dec 6, 2025 Author

Uh oh!

andrewleech Dec 7, 2025 Collaborator Sponsor

Uh oh!

Uh oh!

karfas Dec 7, 2025

Uh oh!

alonbl Dec 7, 2025 Author

Uh oh!

karfas Dec 7, 2025

Uh oh!

alonbl Dec 7, 2025 Author

Uh oh!

karfas Dec 7, 2025

alonbl
Dec 6, 2025

Replies: 4 comments 4 replies

andrewleech
Dec 6, 2025
Collaborator Sponsor

alonbl
Dec 6, 2025
Author

andrewleech Dec 7, 2025
Collaborator Sponsor

karfas
Dec 7, 2025

alonbl
Dec 7, 2025
Author

alonbl Dec 7, 2025
Author