net.imap: add an IMAP4rev1 client by Chocapikk · Pull Request #28269 · vlang/v · GitHub
Skip to content

net.imap: add an IMAP4rev1 client - #28269

Open
Chocapikk wants to merge 3 commits into
vlang:masterfrom
Chocapikk:net-imap
Open

net.imap: add an IMAP4rev1 client#28269
Chocapikk wants to merge 3 commits into
vlang:masterfrom
Chocapikk:net-imap

Conversation

@Chocapikk

Copy link
Copy Markdown

net.smtp sends mail but nothing in vlib reads it, so a V program that wants to look at a mailbox has to shell out or reimplement IMAP. This adds net.imap, a client for IMAP4rev1 (RFC 3501).

I checked vpm and awesome-v first: there is no IMAP module in either.

mut c := imap.new_client(
    server:   'imap.example.com'
    username: 'someone@example.com'
    password: 'hunter2'
    ssl:      true
)!
defer { c.close() or {} }

inbox := c.select_mailbox('INBOX')!
println('${inbox.exists} messages, ${inbox.recent} recent')

unread := c.search('UNSEEN')!
for msg in c.fetch(unread, '(UID ENVELOPE)')! {
    if envelope := msg.envelope {
        println('${envelope.from[0].addr()}: ${envelope.subject}')
    }
}

Responses are tokenised, not searched

The module reads the grammar of RFC 3501 section 9 rather than looking for keywords in a response line. That is not tidiness, it is correctness: a message whose subject is UID 999 answers a search for UID exactly as well as the real field does, and a body containing ) ends a response that a bracket-counting parser would still be reading.

wire.v is a decoder over the four shapes the protocol is built from: atoms, quoted strings, literals and parenthesised lists. It reads the connection directly with one octet of lookahead, so a literal is taken by count and its content is never offered back to the parser. An item this module does not model is stepped over whole by skip_value, so a server extension cannot desynchronise the rest of the response.

Two grammar details worth naming, because both are easy to get wrong and both are covered by tests:

  • env-cc = "(" 1*address ")". The addresses in an envelope are concatenated with nothing between them, not separated by spaces like every other list in the protocol.
  • [ is a valid ATOM-CHAR; only ] is excluded. Reading a fetch item name with a plain atom reader therefore swallows the opening bracket of BODY[] and leaves the rest unreadable.

Mailbox names

Names are ordinary UTF-8 in the API and travel as the modified UTF-7 of section 5.1.3, encoded on the way out and decoded on the way in. Without it, list_mailboxes hands back &AMk-l&AOk-ments where the user's mailbox is called Éléments. The encoding is checked against the two examples printed in the RFC and cross-checked against an independent implementation.

Sequence sets

SeqSet keeps message numbers as merged ranges, so a search matching fifty thousand messages is sent as FETCH 1:50000 .... A comma separated list of every number would be a command line no server is obliged to accept.

What is implemented

CAPABILITY, NOOP, LOGOUT, STARTTLS, AUTHENTICATE PLAIN, LOGIN, SELECT, EXAMINE, CREATE, DELETE, RENAME, SUBSCRIBE, UNSUBSCRIBE, LIST, LSUB, STATUS, APPEND, CHECK, CLOSE, UNSELECT, EXPUNGE, SEARCH, FETCH, STORE, COPY, MOVE, and the UID forms of the last five.

FETCH parses UID, FLAGS, RFC822.SIZE, INTERNALDATE, ENVELOPE, BODYSTRUCTURE and any number of BODY[...] sections, keyed by the specification the server echoed back.

Some things that took deliberate handling:

  • An argument a quoted string cannot hold, an eight bit password or a message body, is sent as a literal: the introducer goes out, the client waits for the + continuation, then the octets follow. Section 2.2.1 says the server may send untagged data or reject the command instead of continuing, so the wait handles both rather than assuming the next line is the +.
  • A mailbox changes under a client at any moment, and the server says so by slipping EXISTS, EXPUNGE and FETCH responses into whatever command happens to be in flight. Those are collected rather than discarded, and Client.exists and Client.recent follow them.
  • A completion carrying a tag other than the outstanding one is an error. It means the client and the server disagree about which command is running, and nothing later recovers from that.
  • A literal announces its own length, so a server can ask a client to allocate as much memory as it likes. There is a cap, checked before the allocation.
  • An untagged BYE closes the session, and a greeting that turns the connection away closes the socket rather than leaving the server to time it out.

ESEARCH (RFC 4731) and MOVE (RFC 6851) are supported; supports() is there to guard the optional ones.

Tests

Four files, 513 assertions, no external server needed.

  • utf7_test.v covers the RFC's own examples, a round trip over Japanese, Russian, Chinese, French and an emoji, the astral plane surrogate pair, and six malformed inputs that must be refused.
  • seqset_test.v covers merging, ordering, *, parsing and the fifty thousand message case.
  • response_test.v runs the response reader over transcripts taken from the RFC, including its envelope and body structure examples, plus the two traps above: a body and an envelope subject that both read like protocol.
  • imap_test.v drives the client over a real socket against a scripted server, checking the exact bytes each command puts on the wire, the literal handshake for APPEND and for an eight bit password, and that an unsolicited EXISTS reaches the client.

v fmt -verify and v vet -W are clean on all nine files, and none of them show up in v test-cleancode.

…EXISTS

Dovecot answers EXPUNGE and CLOSE with the removals alone and no new
EXISTS behind them, so a client that waited for one went on reporting a
count that included messages it had just watched being taken out.

Each EXPUNGE now decrements the count, and an EXISTS the server did send
still wins over that arithmetic. Found by running a session against a
real Dovecot; the scripted server in the tests had been sending an EXISTS
and hiding it.
@Chocapikk

Copy link
Copy Markdown
Author

Building a set of scattered numbers rescanned and re-sorted every range
on each insertion, which is quadratic: twenty thousand numbers took ten
seconds. The ranges are kept sorted and disjoint, so insertion now
bisects for its place and merges in one pass.

The decoder reached into the reader for a single octet at a time, with a
one-element array allocated per byte. It reads a chunk at a time now, and
a literal larger than a chunk goes straight into the caller's buffer.

Four near-identical token loops become one, and a mailbox name that is
already plain ASCII skips the UTF-7 encoder rather than walking it.

  seq_set, 20000 scattered   10074 ms -> 1.2 ms
  parse 200 messages, 462 KB   4.98 ms -> 2.50 ms
  utf7_encode, ascii name       309 ns -> 27 ns
  utf7_decode, ascii name       155 ns -> 19 ns

@medvednikov medvednikov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the very thorough parser work and test coverage. I found several correctness issues that should be addressed before merge:

  1. Greeting/authentication state is lost (imap.v). read_greeting() accepts PREAUTH, but new_client() still calls LOGIN whenever username is set. A PREAUTH greeting already puts the connection in authenticated state, so both LOGIN and STARTTLS are invalid there. The greeting's CAPABILITY response code is also parsed and discarded, so a server can advertise LOGINDISABLED and the client will still send LOGIN (including the password). Please retain the greeting state/capabilities and gate STARTTLS/authentication accordingly. Regression tests for PREAUTH with configured credentials and for greeting-time LOGINDISABLED would catch both cases.

  2. A failed SELECT/EXAMINE leaves stale selected-mailbox state. IMAP returns to authenticated state after a failed selection. open_mailbox() only assigns c.selected on success, so selecting a missing mailbox while another mailbox is selected leaves c.selected pointing at the old mailbox. Clear the selected state (and any selected-mailbox counters) when a new selection is attempted/fails, and add a selected -> failed-select test.

  3. Several post-dial error paths can leak the transport. Failures during implicit TLS, STARTTLS, or the automatic login escape without shutdown(). Separately, an untagged BYE sets is_open = false, so a later close() returns before closing the local socket. Please make every error after a successful dial, plus BYE/logout handling, tear the transport down exactly once.

  4. Malformed server INTERNALDATE data can panic the process. parse_internal_date() feeds unchecked .int() results into time.new(), whose range validation panics. A response containing an out-of-range day/hour/etc. should return an IMAP parse error rather than terminate the client. The zone digits/ranges should be validated too.

  5. Unknown untagged responses containing literals still desynchronise the decoder. The fallback in read_untagged() calls d.text(), which stops at the CRLF after a literal marker such as {3} and leaves the literal payload to be parsed as the next response. A transcript like * XDATA {3}\r\nabc\r\na1 OK done\r\n demonstrates it. The unknown-response skip needs to be literal-aware to uphold the stated extension-safety guarantee.

  6. APPEND can mislabel local time as UTC. format_internal_date() always emits +0000 using the supplied struct's raw fields, while time.now() is local time. Convert with local_to_utc() before formatting (or emit the actual offset), otherwise the stored internal date is shifted by the caller's timezone.

Smaller edge case: seq_max = u32(0xffffffff) collides with the largest valid IMAP number. For example, parse_seq_set('4294967295')!.str() becomes *; using a wider internal bound would keep the numeric value distinct from the star sentinel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants