By: David Lang - davidlang proposed IMAP roadmap (affects non IMAP stuff 2004-05-26 13:22 I'm proposing this as a roadmap becouse I think it is a reasonable path and if we agree on the path I may be able to help implement some of these features

Among the stated goals for the IMAP interface and popfile are support for multiple IMAP servers support for forking the IMAP processes support for multiple users of popfile

proposed steps (not nessasarily in order, but I think roughly so)

1. re-factor the check_for_new messages and reclassiy_on_more (in progress)
2. restructure the folder lists per my thoughts in IMAP 1.6 (list of known folders with flags rather then seperate lists of watched and output folders)
3. factor out the move message to new folder to be a seperate routine
4. factor out the folder selection from the message analysis to the service routine, passing $folder,and $bucket along with $imap
5. change the initialization, start, stop, and service routines to have a seperate imap connection per folder (change self→imap__ to self→imap__($folder)
6. change the server config data to be a table in the database.(login info is part of the server config)
7. change the folder config/status data to be in the database instead of the config file, refrencing the server config data in each folder entry (includes moving the uid/uidvalidity data out of the config file to the database and updating it more frequently)
8. change the move_message function to handle multiple servers (detect that the source and destination are not on the same server and download/upload the message instead of doing an IMAP copy). this includes having move_messages maintain it's own set of connections to the servers so that it can send commands without having to worry about what commands are outstanding from the IMAP scanner routines have.
9. alter the UI to allow multiple servers to be entered in the IMAP config menu and change the folder add option to designate which server the folder is on
10. at this point it should be simple to allow popfile to folk off the IMAP scanning processes with each one having it's own move_messages routine. (how will database access work at this point??)
11. seperate the move_messages to it's own process, with the IMAP scanning processes sending it messages to tell it what messages to move where.
12. change the move_message function to support moving a message from a local file to an IMAP server
13. change the move_message function to honor the per-bucket modification flags (header alterations)
14. remove the per-bucket options from the IMAP config menu by adding them to the bucket menu (you would still need to go to the IMAP menu to tell popfile that a particular folder exists, but you would tell popfile that a particular bucket maps to a particular IMAP folder in the bucket menu)
15. alter the pop3/NNTP proxies to honor the per-bucket IMAP assignment by sending the real message via the move_message function (possibly moved to a seperate module??) and then pass a pre-generated stub message to the client
16. add additional delivery options to move_message (SMTP, LMTP, NNTP post, local files in various formats, etc)
17. add additional non-IMAP message source options (POP3 'fetchmail' option as well as the proxy option, SMTP listener, etc) these sources would retreive the messages and then classify them and deliver them via move_message

thoughts? as long as this is I know I've left something out. (I have remembered it a half a dozen times as I was typeing something else, only to forget it before I finished what I was doing) not to mention things that I have never considered.

this roadmap ends up morphing popfile into a swis-army-knife classifier

I start off with tiny concrete steps that are valuble in themselves, but they also build towards the possibility of implementing the later items fairly easily, some of the later options are much more general, and I don't know how nessasary they really will be, some of them are as much for overall consistancy as anythng else (if you say that bucket spam should go to this place, it really should go there, no matter if the source of that spam is via pop, IMAP, or SMTP)

at the moment popfile is a POP3 proxy that has IMAP bolted onto the side, I haven't seen the info on how the SMTP and NNTP stuff will work so I don't know how well they are really integrated, I think these steps will integrate the IMAP support in nicely and end up opening up a lot of new possibilities

as the migration towards multi-user support continues limiting the list of allowed folders (and the associated server definitions, which do include login credentials) for a particular user should allow this to co-exist without much conflict between the two projects

By: David Lang - davidlang
  RE: proposed IMAP roadmap (affects non IMAP s   
2004-05-26 13:58
I remembered one item I had forgotten.

sometime after item 5, call it 5.5 there are several optimizations that 
can be done to reduce the impact that popfile has on the server.
today popfile IMAP does a select for each folder on each pass, this 
is not quite as bad as a full login, but it's in the same ballpark (in 
fact the overhead of doing a select on a server is actually significantly 
higher then the act of logging in, until you account for the fact that after 
you login you then have to do a select) after you have a seperate 
connection per folder you can do the select when you login and not need
to after that.
along the same lines there are optimizations that could let us avoid the 
need to do a search for each pass, again avoiding a lot of work on the part 
of the server

  By: Manni Heumann - mannih2001
  RE: proposed IMAP roadmap (affects non IMAP stuff   
  2004-05-26 14:07
  That's an amazing roadmap, David.

  I will have to digest this before I can really comment on it.

  But here are my first thoughts:

  1. I hope that I after the refactoring in (1) and a period of bug-fixing and 
  (yuck) test writing, we will have a version of IMAP that is ready to ship with 
  the next release of POPFile. Once this is out, there will be a lot more bug-
  fixing, I guess, and I also guess that we will get some feedback that might 
  somehow change the roadmap (I'm not hoping for a changed roadmap, but 
  for the feedback, of course). So I am currently concentrating on (1) and I 
  hope it will be finished some day.

  2. Reading your roadmap, one might get the impression that IMAP is the one 
  and only tool for you. I don't know whether I understood the last points on 
  the roadmap correctly, but why would anybody want to have POP3 or NNTP 
  messages delivered to an IMAP server?

  3. Any idea about the time scale we are talking about?

  Manni

     By: David Lang - davidlang
     RE: proposed IMAP roadmap (affects non IMAP s   
     2004-05-26 14:32
     
     first off I'll answer #3 first, I have no time scale in mind for this, 
     just a list of baby steps that take us towards the goal
     (I spend so much time at work fighting people who have 'great' ideas 
     that work well in the short term and have to be scrapped in the long term 
     that I am constantly in the mindset of identifying a long term goal and 
     trying to make all changes in the short term support that goal)
     along the same lines to address #1, getting IMAP to the point where it can 
     ship with the next version is definantly the highest priority goal
     as for #2, I do use IMAP almost exclusinvly, but I am thinking that there 
     is an oppurtunity for some overall simplification of popfile that will give 
     it additional capabilities if we shuffle the pieces correctly (funnel 
     multiple things through a move_message function so that mixing protocol 
     types is just a matter of figuring out a sane way to configure it)
     as an example of why you would want to put pop3 messages in an IMAP folder
     Jim has his main e-mail address with his primary ISP, but he gets a lot of 
     mail (the pesky Lang gene again :-) and can't leave it there. what he 
     currently does is to use popfile to classify the mail, download it to Pegasus, 
     which then sorts the mail based on how popfile has classified it and then 
     uploads the mail to IMAP folders on another server into his 44 different 
     folders (one per bucket)
     while this process works it is a little more involved the it should need to be.
     Ideally (and if/when this roadmap is fully implemented) he could tell popfile 
     that it has a data source (watched folder) of type POP3 on server A and popfile 
     would go out, retreive the messages, classify them, and stuff them into the 
     correct IMAP folders.
     another reason for this sort of thing would be to aggragate multiple e-mail 
     accouunts on different servers into one set of buckets.
     a reason to go POP3/IMAP source to SMTP destination would be to have popfile 
     have a bucket called 'critical' that you train to recognise critical messages 
     and forward them to your phone/pager. this is far better then giving out your 
     phone/pager e-mail address as popfile will filter the messages so you only get 
     the ones that are really critical
     think of all the things that people use fetchmail to do, but instead of the 
     complicated fetchmail config language you would have the popfile classification 
     engine to do the decision making for you

        By: Manni Heumann - mannih2001
        RE: proposed IMAP roadmap (affects non IMAP s   
        2004-05-26 14:48

        I have no idea what people are doing with fetchmail, but what you are saying 
        makes a lot of sense. In fact, I would welcome the possibility of merging several 
        of my mail accounts on one good, reliable IMAP server.

        Manni

on 6-16 Josh suggested that an additional option for a destination bucket would be to set flags on a message. David Lang agrees that this would be a useful capability