Compare commits

..

137 Commits

Author SHA1 Message Date
Gilles LAMIRAL
8c82976c3f
Merge pull request #526 from elbaby/linux-france-gone
change references to the gone linux-france.org domain
2026-05-09 14:39:15 +02:00
Mariano Absatz (git)
fe237e3308 change references to the gone linux-france.org domain to either archive.org or pop2imap.lamiral.info 2026-05-08 19:14:25 -03:00
Gilles LAMIRAL
c558ffcb11
Merge pull request #504 from 0xdaedalus5/0xd/proofreading
Documentation Cleanup
2025-09-09 16:16:52 +02:00
0xdaedalus
cd17a623e5 Just a little proofreading. 2025-09-09 06:34:54 +00:00
Gilles LAMIRAL
023535075c
Merge pull request #500 from pradeepkatiyar007/patch-1
Update README
2025-07-17 13:35:07 +02:00
Pradeep Katiyar
0da5cfa4fb
Update README
I verified and listed another similar software for IMAP sync and backup.
2025-07-17 14:52:14 +05:30
Gilles LAMIRAL
0dd79e7e52
Merge pull request #491 from myvesta/patch-1
Update INSTALL.Debian.txt: Debian 12 Bookworm added
2025-01-21 16:34:01 +01:00
myvesta
5b92e1d908
Update INSTALL.Debian.txt: Debian 12 Bookworm added 2025-01-20 13:09:21 +01:00
Gilles LAMIRAL
a5ea7625a7
Merge pull request #445 from nextsux/patch-1
Add libcrypt-openssl-pkcs12-perl to Dockerfile
2024-04-23 14:13:13 +02:00
Martin Mlynář
9fc8afcae5
Add libcrypt-openssl-pkcs12-perl to Dockerfile
After merging #381 docker image started to crash when using google xoauth with:

Can't locate Crypt/OpenSSL/PKCS12.pm in @INC (you may need to install the Crypt::OpenSSL::PKCS12 module)
2024-03-16 15:58:06 +01:00
Gilles LAMIRAL
b0619eb2c8
Merge pull request #381 from FreddleSpl0it/improve-xoauth2
Improve XOAUTH2 authmech
2024-02-27 16:50:45 +01:00
Gilles LAMIRAL
c6d3925c68
Merge pull request #415 from polluks2/master
Fixed typos
2023-10-10 15:26:35 +02:00
polluks
7d7110d331 Fixed another typo 2023-10-10 15:14:50 +02:00
polluks
ed4816f3b1 Fixed typos 2023-10-10 15:12:19 +02:00
FreddleSpl0it
a877e6f162
Use pure perl code for xoauth2 authmech 2023-03-03 10:34:53 +01:00
Gilles LAMIRAL
5d6bc8149f
Merge pull request #363 from tapsiturtle/patch-1
Update FAQ.Passwords_on_Windows.txt
2022-10-26 12:57:10 +02:00
tapsiturtle
abe8fd2b0b
Update FAQ.Passwords_on_Windows.txt 2022-10-25 14:38:54 +02:00
Nick Bebout
4ec979525b Disable releasecheck in 2.229 2022-10-24 21:21:25 -05:00
Nick Bebout
399651d088 2.229 2022-10-24 21:20:30 -05:00
Nick Bebout
e2c23d57d6 Disable releasecheck in 2.200 2022-06-01 10:48:05 -05:00
Nick Bebout
42dd32ab62 2.200 2022-06-01 10:47:18 -05:00
Nick Bebout
8e4211e86d Disable releasecheck in 2.178 2022-02-15 12:30:07 -06:00
Nick Bebout
2f815205a9 2.178 2022-02-15 12:29:45 -06:00
Nick Bebout
f98c9946e5 Disable releasecheck in 2.140 2021-08-04 14:15:22 -05:00
Nick Bebout
f26b2573c3 2.140 2021-08-04 14:14:36 -05:00
Gilles LAMIRAL
4ce248c075
Merge pull request #299 from rkarlsba/patch-1
Update INSTALL.Debian.txt
2021-07-29 11:47:13 +02:00
Roy Sigurd Karlsbakk
8a04276819
Update INSTALL.Debian.txt
/usr/bin/time isn't in buster or later by default
2021-07-29 01:23:13 +02:00
Gilles LAMIRAL
452ebe34ac
Merge pull request #294 from mendel5/master
Explain encryption related to SSL/TLS
2021-07-07 10:30:39 +02:00
mendel5
60b09370bb
add explanation regarding encryption (ssl / tls)
related to https://github.com/imapsync/imapsync/issues/292
2021-07-06 12:53:34 +02:00
Gilles LAMIRAL
88f731e0d9
Merge pull request #289 from mendel5/patch-1
fix spelling
2021-06-22 17:34:37 +02:00
mendel5
c9ee7a9674
fix spelling
admint to admit
2021-06-17 22:30:07 +02:00
Gilles LAMIRAL
2c1124552b
Merge pull request #284 from florenthemmi/update-faq.d
Update FAQ.d
2021-05-03 11:01:27 +02:00
Florent Hemmi
e7fecaa480
Update FAQ.d 2021-05-01 17:46:04 +02:00
Gilles LAMIRAL
8d2730ac65
Merge pull request #281 from fhe-odoo/patch-1
typos
2021-04-25 17:52:13 +02:00
Fabrice Henrion
1b335c40d1
typos 2021-04-25 07:21:58 -04:00
Gilles LAMIRAL
0e46857ca3
Merge pull request #278 from 3nprob/docker-ps
Add procps to Dockerfile
2021-03-22 12:50:29 +01:00
3nprob
eb62ea989d Add procps to Dockerfile 2021-03-22 20:09:32 +09:00
Gilles LAMIRAL
7fda0fe570
Merge pull request #258 from danielwagn3r/master
Refactored Dockerfile
2020-12-28 13:33:37 +01:00
Daniel Wagner
89a2eee569
Create docker-image.yml 2020-12-28 12:26:21 +01:00
Daniel Wagner
a003ac4399
Update Dockerfile
Reverted back to Debian Buster
2020-12-28 09:50:43 +01:00
Daniel Wagner
542d5057ea
Update Dockerfile
Readded `set -xe`
2020-12-28 00:44:07 +01:00
Daniel Wagner
53b354d592
Merge pull request #1 from danielwagn3r/danielwagn3r-patch-dockerfile
Update Dockerfile
2020-12-28 00:41:18 +01:00
Daniel Wagner
ada6a26888
Update Dockerfile
Refactored Dockerfile
2020-12-28 00:41:03 +01:00
Gilles LAMIRAL
1709fee78f
Merge pull request #223 from mmoya/exchange-faq-updates
faq: Update Exchange FAQ urls
2020-06-17 21:21:37 +02:00
Maykel Moya
c88cbdd0ca faq: Update Exchange FAQ urls
Use the current location for those URLs that are actually redirecting.
2020-06-17 08:44:02 +02:00
Gilles LAMIRAL
4e8e3cb0f5
Merge pull request #220 from arjenz/master
Typofix connexion -> connection
2020-06-05 13:11:59 +02:00
Arjen Zonneveld
75bbd9d87e Typofix connexion -> connection 2020-06-05 11:44:17 +02:00
Gilles LAMIRAL
63a409b090
Merge pull request #218 from exander77/master
Fixed IMAPClient apped_string returning 0
2020-04-20 14:51:33 +02:00
Radomír Polách
1df80fa321 Fixed IMAPClient apped_string returning 0 2020-04-20 12:05:11 +02:00
Gilles LAMIRAL
c300a8e19f
Merge pull request #216 from exander77/master
Fixed empty subject
2020-04-14 14:40:58 +02:00
Radomír Polách
7b9063a168 Fixed empty subject 2020-04-13 22:32:58 +02:00
Nick Bebout
b3ff66078b Disable releasecheck in 1.977 2020-04-10 18:17:40 -05:00
Nick Bebout
1d08afaba6 1.977 2020-04-10 18:15:57 -05:00
Gilles LAMIRAL
93b917b12b
Update README.md
Reread intro
2019-12-02 11:31:19 +01:00
Gilles LAMIRAL
6e5e3905a0
Changed the pitch 2019-11-22 20:54:03 +01:00
Gilles LAMIRAL
87ec5db010
typo 2019-11-11 11:10:53 +01:00
Gilles LAMIRAL
4881f89d6f
Github stats per year 2019-11-11 11:09:23 +01:00
Gilles LAMIRAL
0a7b2056eb
Update README.md 2019-11-09 21:16:37 +01:00
Gilles LAMIRAL
c82103dd97
Added new github sponsoring link 2019-11-09 13:40:18 +01:00
Gilles LAMIRAL
de92124ef0
Create FUNDING.yml
I've been accepted in the GitHub Sponsorship.
It's related to 
https://github.com/imapsync/imapsync/issues/166
Feel free to argument and participate!
2019-10-23 16:55:36 +02:00
Gilles LAMIRAL
8fe1c099ef
Merge pull request #146 from dszymczuk/patch-1
Update INSTALL.Docker_build.txt
2019-07-14 16:36:06 +02:00
Gilles LAMIRAL
e32442e331
Merge pull request #182 from pchaitat/patch-1
libfile-tail-perl
2019-07-14 16:14:51 +02:00
chaitat
8a94761095
libfile-tail-perl
On my freshly installed Debian 9 (Linux 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux), when I followed the instruction (without the libfile-tail-perl), I got the below error message when I ran imapsync (git cloned from the commit id f9a77ee223 which was commited on Tue Jul 2 18:34:23 2019 -0500) for the first time.  But, after I had installed libfile-tail-perl package, I could successfully run imapsync.

"""
Can't locate File/Tail.pm in @INC (you may need to install the File::Tail module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at ./imapsync line 840.
BEGIN failed--compilation aborted at ./imapsync line 840.
"""
2019-07-04 01:12:08 +07:00
Nick Bebout
f9a77ee223 Update README.md to be based on 1.945 README 2019-07-02 18:34:23 -05:00
Nick Bebout
e90a8ffc3b Re-add README.md 2019-07-02 18:32:01 -05:00
Nick Bebout
cbd4bfb488 Disable releasecheck in 1.945 2019-07-02 18:26:15 -05:00
Nick Bebout
275436c5a0 1.945 2019-07-02 18:25:47 -05:00
Nick Bebout
0797e48248 Disable releasecheck in 1.921 2019-07-02 18:24:43 -05:00
Nick Bebout
f35c002793 1.921 2019-07-02 18:24:23 -05:00
Nick Bebout
b1bd66b23b Disable releasecheck in 1.920 2019-07-02 18:23:29 -05:00
Nick Bebout
852d9695d6 1.920 2019-07-02 18:20:46 -05:00
Nick Bebout
62531f58cd 1.920 2019-07-02 18:17:46 -05:00
Gilles LAMIRAL
eafd200ec0
Merge pull request #176 from SSteve/uft_typos
Fix typos "uft" -> "utf"
2019-04-23 00:37:07 +02:00
Steve Nicholson
cd165c9342 Fix typos "uft" -> "utf" 2019-04-22 13:23:20 -07:00
Gilles LAMIRAL
6297de9492
it does build failures that are not explained... 2019-01-31 14:06:05 +01:00
Gilles LAMIRAL
04c3fac5cf
reread 2019-01-31 13:17:11 +01:00
Gilles LAMIRAL
038301b73a
Change message 2019-01-30 20:18:41 +01:00
Gilles LAMIRAL
9b2b7a9e90
Update README.md 2019-01-24 19:18:33 +01:00
Gilles LAMIRAL
61629d8bef
Update _config.yml 2019-01-22 21:39:22 +01:00
Gilles LAMIRAL
9631cd96fd Set theme jekyll-theme-cayman 2019-01-22 21:37:41 +01:00
Gilles LAMIRAL
d8b989e1b6
Delete _config.yml 2019-01-22 21:05:53 +01:00
Gilles LAMIRAL
f345d20b36
symlink 2019-01-22 20:56:33 +01:00
Gilles LAMIRAL
2c691674e5
Delete imapsync_abortme.log 2019-01-22 20:55:38 +01:00
Gilles LAMIRAL
057c10d462
Delete dist 2019-01-22 20:54:53 +01:00
Gilles LAMIRAL
4c08e7ffd7
No exe here! 2019-01-22 20:54:32 +01:00
Gilles LAMIRAL
68b17074a8
Options are in the README file now. (since August 2017) 2019-01-22 20:53:18 +01:00
Gilles LAMIRAL
2f10a175f0
Added README verbatim inside. 2019-01-22 19:21:29 +01:00
Gilles LAMIRAL
e28e79feec
Added a donation invitation 2019-01-22 18:57:07 +01:00
Gilles LAMIRAL
5a34fa97eb Set theme jekyll-theme-cayman 2019-01-22 13:02:56 +01:00
Gilles LAMIRAL
bdb1e744b6 Set theme jekyll-theme-minimal 2019-01-22 13:02:20 +01:00
Gilles LAMIRAL
9fcfea9905
Delete page_tva_mois.ps
broken link
2019-01-22 12:58:36 +01:00
Gilles LAMIRAL
0cdb6de7e6 Set theme jekyll-theme-cayman 2019-01-22 12:49:32 +01:00
Gilles LAMIRAL
ffbb0af5a1 Set theme jekyll-theme-cayman 2019-01-22 12:46:07 +01:00
Gilles LAMIRAL
95a972f0e5
Merge pull request #159 from jazzzz/link-dlls-on-win64
Link Win64 exe with DLLs
2018-11-20 22:49:31 +01:00
Gilles LAMIRAL
57f9629d16
Merge pull request #160 from jazzzz/fix-lose-spelling
Fix "lose" spelling
2018-11-20 22:46:13 +01:00
Ludovic Perrine
4bad4a8f5b Fix "lose" spelling 2018-11-20 15:19:24 +01:00
Ludovic Perrine
55ed2ad833 Link Win64 exe with DLLs 2018-11-20 15:08:44 +01:00
Gilles LAMIRAL
dd9a644e90
Merge pull request #152 from ZsZs73/master
added missing dependency: libssl-dev
2018-10-19 02:20:50 +02:00
ZsZs73
e24e651f36 added missing dependency 2018-10-11 09:38:28 +02:00
Gilles LAMIRAL
166e03b37f
Removed double-quotes around several modules names 2018-09-28 13:38:01 +02:00
Damian Szymczuk
5ba8dc1dab
Update INSTALL.Docker_build.txt
Resolve error:

Can't locate IO/Socket/INET6.pm in @INC (you may need to install the IO::Socket::INET6 module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/bin/imapsync line 713.
BEGIN failed--compilation aborted at /usr/bin/imapsync line 713.
2018-09-04 19:15:36 +02:00
Gilles LAMIRAL
e2daa4f91f
Add files via upload 2018-08-29 21:52:12 +02:00
Gilles LAMIRAL
da64bc0111
Merge pull request #145 from daveserpa/patch-1
Allow --maxsleep to override --gmail{1,2}
2018-08-21 23:09:13 +02:00
Dave Serpa
cdfc599350
Allow --maxsleep to override --gmail{1,2}
While backing up large gmail accounts, sleeping for longer than 2 seconds can help avoid hitting bandwidth quotas.
2018-08-18 01:08:59 -04:00
Nick Bebout
c6f6da4ef1 Disable releasecheck in 1.882 2018-05-07 09:04:51 -05:00
Nick Bebout
9a927be251 1.882 2018-05-07 09:04:23 -05:00
Gilles LAMIRAL
65f8552a91
Merge pull request #126 from pkirk/patch-1
Update INSTALL.Debian.txt
2018-03-06 17:36:57 +01:00
Paolo Gabrielli
22db0f9370
Update INSTALL.Debian.txt
In Debian 9, just tried with a fresh server, if you omit "libio-socket-inet6-perl", you get

"""
root@jarjar:~# ./imapsync
Can't locate IO/Socket/INET6.pm in @INC (you may need to install the IO::Socket::INET6 module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at ./imapsync line 676.
BEGIN failed--compilation aborted at ./imapsync line 676.
"""

Maybe because my machine has IPv6?
2018-03-06 16:07:08 +01:00
Nick Bebout
31115d0e16 Disable releasecheck in 1.836 2017-09-23 16:55:44 -05:00
Nick Bebout
8d76e44c5e 1.836 2017-09-23 16:54:48 -05:00
Gilles LAMIRAL
3afeea4a16 Merge pull request #101 from tux2000/patch-1
Package libtest-mockobject-perl needed as well
2017-06-26 09:59:03 +02:00
tux2000
5c3bf942e2 Package libtest-mockobject-perl needed as well
One more package to be installed from the repository on Ubuntu...
2017-06-25 22:19:39 +02:00
Nick Bebout
2ab41adc87 Disable releasecheck in 1.727 2016-09-19 10:17:50 -05:00
Nick Bebout
137242e609 1.727 2016-09-19 10:17:24 -05:00
Nick Bebout
3eaac56812 Disable releasecheck in 1.684 2016-09-19 10:16:03 -05:00
Nick Bebout
8d24f07718 1.684 2016-09-19 10:15:41 -05:00
Nick Bebout
d0595b7dfd Disable releasecheck in 1.678 2016-01-22 10:56:14 -06:00
Nick Bebout
629adbb8db 1.678 2016-01-22 10:52:28 -06:00
Nick Bebout
c16227350f Disable releasecheck in 1.670 2015-12-03 11:18:49 -06:00
Nick Bebout
b7c835d670 1.670 2015-12-03 11:16:32 -06:00
Nick Bebout
f1987d5e52 Disable releasecheck in 1.644 2015-11-18 08:54:51 -06:00
Nick Bebout
4a1d71d8fd 1.644 2015-08-03 20:44:40 -05:00
Nick Bebout
9ff89e7df2 Disable releasecheck in 1.637 2015-05-28 12:05:46 -05:00
Nick Bebout
e2bfc931f4 1.637 2015-05-28 12:04:57 -05:00
Nick Bebout
4a536871fe Disable releasecheck in 1.607 2014-11-19 16:17:09 -06:00
Nick Bebout
65fd3eac5d 1.607 2014-11-19 16:16:16 -06:00
Nick Bebout
e195ab3906 Disable releasecheck in 1.596 2014-11-19 16:11:24 -06:00
Nick Bebout
dc5f6a1605 1.596 2014-11-19 16:08:41 -06:00
Nick Bebout
2e764e7987 Merge pull request #17 from rzr/master
Please cleanup your tree :)
2014-08-08 02:10:54 -05:00
Nick Bebout
5a8a853033 Disable releasecheck in 1.592 2014-05-29 20:57:26 -05:00
Nick Bebout
459f6a81cb 1.592 2014-05-29 20:56:21 -05:00
Philippe Coval
1a2358abc2 cleanup
Signed-off-by: Philippe Coval <rzr@gna.org>
2014-05-02 09:58:50 +02:00
Philippe Coval
e1abb47952 ignore generated files
Signed-off-by: Philippe Coval <rzr@gna.org>
2014-05-02 09:58:50 +02:00
Nick Bebout
428581af1e Disable releasecheck - CVE-2013-4279 2014-03-04 19:29:43 -06:00
Nick Bebout
7d2043f95f 1.584 2014-02-13 11:34:20 -06:00
Nick Bebout
6a0efa6bc8 1.580 2013-12-29 20:03:13 -06:00
Nick Bebout
8f266abab8 1.569 2013-10-17 19:11:27 -05:00
461 changed files with 75918 additions and 93662 deletions

12
.github/FUNDING.yml vendored Normal file
View File

@ -0,0 +1,12 @@
# These are supported funding model platforms
github: [gilleslamiral] # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

21
.github/workflows/docker-image.yml vendored Normal file
View File

@ -0,0 +1,21 @@
name: Docker Image CI
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
buildx:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
- name: Available platforms
run: echo ${{ steps.buildx.outputs.platforms }}

10
.gitignore vendored
View File

@ -1 +1,11 @@
imapsync_elf_x86.bin
imapsync_bin_Darwin
imapsync_bin_Linux_i686
imapsync_bin_Linux_i686_petite
*~
debian/*.log
debian/*.subvars
debian/*/*/*
debian/files
packaging/*/*
W/tmp/*

66
CREDITS
View File

@ -1,25 +1,19 @@
#!/bin/cat
# $Id: CREDITS,v 1.175 2013/08/28 22:44:09 gilles Exp gilles $
# $Id: CREDITS,v 1.193 2019/07/31 23:05:10 gilles Exp gilles $
If you want to make a donation to the author, Gilles LAMIRAL,
If you want to make a donation to me, imapsync author, Gilles LAMIRAL,
use any of the following ways:
a) you can use the imapsync wishlist :
http://www.amazon.com/gp/registry/wishlist/1C9UNDIH3P7R7/
(Use the lowest postal cost)
b) If you can read french, please use the following wishlist :
http://amazon.fr/gp/registry/wishlist/37RZF7PPCD7YL
(books will be send with free postal cost)
c) its paypal account : gilles.lamiral@laposte.net
http://www.linux-france.org/prj/imapsync/paypal.shtml
d) If you prefer making your donation with
cash or cheque then my postal address is:
c) my paypal account is gilles@lamiral.info
http://imapsync.lamiral.info/S/donate.shtml
My postal address is:
Gilles LAMIRAL
La Billais
22 La Billais
35580 Baulon
FRANCE
@ -30,6 +24,51 @@ I thank very much all of these people.
I thank also very much all people who bought imapsync from the homepage
but I don't cite them here.
Sean McDougall from New Brunswick, Ian Thomas & Matt Wilks from Toronto
Sean found the solution and wrote the FAQ item in FAQ.Exchange.txt
"NO Maximum size of appendable message has been exceeded"
and Ian & Matt reported it to me.
Tomasz Makara
Gave FAQ.Massive.txt fixes about network monitoring on Windows any release.
Joe Pruett
Bugfix about delete_message_on_host1() not using --noexpungeaftereach
Mike Salonia
Contributed by using and validating the multi-archive Gmail
destination account. See FAQ.d/FAQ.Gmail.txt
Sebastian Lemke
Contributed by moving hosts from sync_loop_unix.sh to the
credentials file "file.txt".
Ingo Wichmann
Contributed by giving the book
14.95 USD "Rambles Through My Library"
David Karnowski.
Suggested --disarm_read_receipts for its regex.
Erik Torsner.
Gave patch for create_folder() and nested folders.
http://erik.torgesta.com/2013/05/gmail-and-google-apps-mail-migration/
Ralf Hauber
Gave installation tips for Centos and a bug report about
final statistics for newly created folders.
Dave Murray
Simon Savva
Found, reported and solved the "Flagged" flag Exchange bug.
Krul, Patrick
Found bug about trailing blanc on Win32 cache dir.
Jonathan Daley
Contributed by giving the book
13.33 EUR "Cartoon Guide to Statistics"
Bowman Gillianne
Contributed by giving the book
8.60 EUR "L'Épreuve de l'étranger: Culture et traduction dans l'Allemagne romantique"
@ -58,7 +97,6 @@ Peter Fuchs
Contributed by giving the book
30.86 "Designed for Use: Create Usable Interfaces for Applications and the Web"
Pierre GUILLAUME // e-Lixir
Fixed 2 bugs about [Gmail] folder names.
one is about $cache_dir and bsd_globs() that needs escaped characters.
@ -86,7 +124,7 @@ Suggested default --useheader 'Message-ID' --useheader 'Received'
Ameir Abdeldayem
Gave a patch to sync messages lacking Message-ID
http://www.linux-france.org/prj/imapsync_list/msg01151.html
https://web.archive.org/web/20160913193638/http://www.linux-france.org/prj/imapsync_list/msg01151.html
Yousaf Shah
Contributed by giving the book

2670
ChangeLog

File diff suppressed because it is too large Load Diff

1918
FAQ

File diff suppressed because it is too large Load Diff

1
FAQ Symbolic link
View File

@ -0,0 +1 @@
FAQ.d/FAQ.General.txt

1
FAQ.d/.htaccess Symbolic link
View File

@ -0,0 +1 @@
htaccess.txt

166
FAQ.d/FAQ.APPEND_errors.txt Normal file
View File

@ -0,0 +1,166 @@
#!/bin/cat
$Id: FAQ.APPEND_errors.txt,v 1.13 2021/05/29 07:58:44 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.APPEND_errors.txt
======================================================================
Dealing with Imapsync APPEND errors.
======================================================================
Questions answered in this FAQ are:
Q. I have an "could not append" error with
"Message contains invalid header" at the end, like:
Err 1/20: - msg INBOX/6 {40666} could not append ... NO Message contains invalid header
What can I do to transfer these emails?
Q. For some messages, the imapsync log says
"could not append", sometimes followed by an explicit message
describing what went wrong, or sometimes followed by a not very
useful message "socket closed while reading data from server"
What can I do?
Q. The append error message is "NO Message contains NUL characters"
What can I do?
Now the questions again with their answers.
======================================================================
Q. I have an "could not append" error with
"Message contains invalid header" at the end, like:
Err 1/20: - msg INBOX/6 {40666} could not append ... NO Message contains invalid header
What can I do to transfer these emails?
R0. Append in the IMAP protocol is the command to add a message in the folder.
The error string "NO. Message contains invalid header" comes from the
destination IMAP server at host2, it doesn't like the message and rejects it.
R1. Header lines must be in 7bit encoding.
If they aren't in 7bit the IMAP server can refuse them as they break
the IMAP RFC specification.
A solution is to transform all 8bit characters to the Z character with:
imapsync ... --regexmess "tr [\x80-\xff] [Z]"
Caveat: this transformation transforms both the email header and its body.
It's not perfect but I guess it's better than no message transferred.
So consider using it in a second pass, not the first, as 8-bit
characters in the body message are allowed and changing them when
not needed is not a good thing.
R2. Some crappy email systems, like virus checkers or any other crappy
software tool dealing with your messages, can add headers to tell
the world that they've done a great job. But sometimes they didn't,
and they fucked up your messages by not respecting the standard
and added ugly non-rfc compliant headers.
To remove an ugly header, let's call it "X-Spam-Report", that spreads over
several lines beginning or, that's the ugly part, not beginning with a space:
imapsync ... --regexmess 's{X-Spam-Report:.*?\n(^[a-zA-Z0-9\-]+:|^\r?\n)}{$1}xms'
Thanks to Damien SAUTEREAU for reporting and solving this issue.
======================================================================
Q. For some messages, the imapsync log says
"could not append", sometimes followed by an explicit message
describing what went wrong, or sometimes followed by a not very
useful message "socket closed while reading data from server"
What can I do?
R0. Append errors are host2 problems, append is the imap term to copy
a message to the destination account.
A problem is that the associated "socket closed ..." error message happens
for several different issues. So I list here several potential issues
and their solutions if they exist.
R1 deals about too long lines in messages on Windows.
R2 deals about too long lines in messages on Unix.
R3 deals with quota reached.
R1. On Windows, add --regexmess "s,(.{9900}),$1\r\n,g"
Some messages have too long lines; for example,
Exchange supports only 9900 characters line length.
Use this option to add "new line" characters (also called CRLF)
to wrap lines longer than 9900 characters.
The regex means "add one CRLF every 9900".
imapsync.exe ... --regexmess "s,(.{9900}),$1\r\n,g"
R2. On Unix, add --pipemess "reformime -r7". The command reformime
usually belongs to the package called "maildrop".
imapsync ... --pipemess "reformime -r7"
I reproduce here the "reformime" manual part explaining what does
the option "-r7"
$ man reformime |more
REFORMIME(1) Double Precision, Inc. REFORMIME(1)
NAME
reformime - MIME E-mail reformatting tool
SYNOPSIS
reformime [options...]
DESCRIPTION
reformime is a utility for reformatting MIME messages.
Generally, reformime expects to see an RFC 2045[1] compliant message on
standard input
...
OPTIONS
...
-r
Rewrite message, adding or standardizing RFC 2045[1] MIME headers.
-r7
Like -r but also convert 8bit-encoded MIME sections to
quoted-printable.
...
Adding RFC 2045 MIME headers
The -r option performs the following actions:
If there is no Mime-Version:, Content-Type:, or
Content-Transfer-Encoding: header, reformime adds one.
If the Content-Transfer-Encoding: header contains 8bit or raw, but only
seven-bit data is found, reformime changes the
Content-Transfer-Encoding header to 7bit.
-r7 does the same thing, but also converts 8bit-encoded content that
contains eight-bit characters to quoted-printable encoding.
R3. Look for the quota limit on host2. If it is the current mailbox size, don't
search elsewhere. Increase the quota on the destination account.
======================================================================
Q. The append error message is "NO Message contains NUL characters"
What can I do?
R. Use:
imapsync ... --skipmess "m/(\x00)+\Z/"
The option --skipmess skips messages matching a pattern, the
messages are not synced, you just avoid the error messages concerning
those messages.
If you want to sync them, use instead:
imapsync ... --regexmess "s/(\x00)+\Z//g"
======================================================================
======================================================================

View File

@ -0,0 +1,55 @@
#!/bin/cat
$Id: FAQ.Admin_Authentication.txt,v 1.13 2022/04/06 10:27:52 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Admin_Authentication.txt
=======================================================================
Imapsync tips about imap admin authentication.
=======================================================================
It can be useful to authenticate without knowing each user password.
Using an admin account to authenticate is not a standard feature
supported by all imap servers. Sometimes it is implemented via the
--authuser1 option, sometimes not. It depends on the imap software
server and its configuration.
You have to follow this road:
* Determine what is the imap software server
* Find out whether it supports an admin account or not
* If no, leave this reading, take a handkerchief and cry
* If yes, search how the imap software server supports
an admin account (see below)
* Figure out whether an admin account is actually configured or not,
* If not then configure it. If you can't configure it: handkerchief.
* Finally, check an imapsync authentication via this admin account for
a standard user, the option --justlogin is nice for doing this.
* Use it for all users.
* Next, read their email messages, enter their personal life etc.
I'm kidding.
Good luck!
Known imap software servers supporting imap admin authentication:
* Exchange 2003/2007/2010/2013/2016. See the file FAQ.Exchange.txt
* Office365. See the file FAQ.Exchange.txt
* Gmail. See the file FAQ.XOAUTH2.txt
* Dovecot. See the file FAQ.Dovecot.txt
* Zimbra. See the file FAQ.Zimbra.txt
* Kerio See the file FAQ.Kerio.txt
* Cyrus-imap. See the file FAQ.Cyrus.txt
* James. See the file FAQ.James.txt
* UW-imap. See the file FAQ.General.txt
* Sun Java Enterprise System/SunOne/iPlanet. See the file FAQ.General.txt
Known imap software servers NOT supporting imap admin authentication:
* Domino Lotus Notes. See FAQ.Handkerchief.txt
* SmartCloud Notes. Same joke.
=======================================================================
=======================================================================

180
FAQ.d/FAQ.Archiving.txt Normal file
View File

@ -0,0 +1,180 @@
#!/bin/cat
$Id: FAQ.Archiving.txt,v 1.27 2022/05/19 08:56:06 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Archiving.txt
=======================================================================
Imapsync issues and tips about archiving
=======================================================================
Questions answered in this FAQ are:
Q. How can I archive different accounts on the same destination account,
each account on a separate folder?
Q. How can I restore a complete account backuped in a subfolder, like
a backup done with the option --subfolder2?
Q. How to move emails from one IMAP folder to another, either on the
same IMAP server or on a different one? Example:
How to move all messages older than 1 year, from INBOX to Archive?
Q. Can imapsync be used to maintain and restore a local offline copy
of a mailbox, eg for backup purposes, using Mbox or Maildir
format, so that if the server fails, then the mailbox could be
reinstated?
Q. How to archive a whole year in a subfolder?
Q. How to archive a whole year in subfolders but without subsubfolders?
Now the questions again with their answers.
https://imapsync.lamiral.info/FAQ.d/FAQ.Archiving.txt
=======================================================================
Q. How can I archive different accounts on the same destination account,
each account on a separate folder?
R. Use --subfolder2
--subfolder2 str : Syncs the whole host1 folders hierarchy under the
host2 given folder str.
(It does it internally by adding two --regextrans2
options before all others.)
Examples:
To backup the account Foo in a subfolder called FooArchiv:
imapsync ... --user1 Foo --subfolder2 FooArchiv
To backup the account Bar in a subfolder called BarArchiv:
imapsync ... --user1 Bar --subfolder2 BarArchiv
In case you need a very strict sync, add options
--delete2 --delete2foldersonly "FooArchiv"
(or "BarArchiv" for the second example).
It will delete on account2 what is not on account1 but only in the right place,
ie, the subfolder "FooArchiv".
Examples:
To mirror the account Foo in a subfolder called FooArchiv:
imapsync ... --user1 Foo --subfolder2 FooArchiv --delete2 --delete2foldersonly "FooArchiv"
To mirror the account Bar in a subfolder called BarArchiv:
imapsync ... --user1 Bar --subfolder2 BarArchiv --delete2 --delete2foldersonly "BarArchiv"
See how to restore those with the next Q/R.
=======================================================================
Q. How can I restore a complete account backuped in a subfolder, like
a backup done with the option --subfolder2?
R. Use --subfolder1
--subfolder1 str : Syncs the host1 folders hierarchy under folder str
to the root hierarchy of host2.
It's the counterpart of a sync done by --subfolder2
when doing it in the reverse order.
Backup/Restore scenario:
Use --subfolder2 str for a backup to the folder str
on host2. Then use --subfolder1 str for restoring
from the folder str, after inverting
host1/host2 user1/user2 values.
Examples:
To restore the whole folder FooArchiv hierarchy into the Foo account:
imapsync ... --user2 Foo --subfolder1 FooArchiv
To restore the whole folder BarArchiv hierarchy into the Bar account:
imapsync ... --user2 Bar --subfolder1 BarArchiv
Read the previous FAQ item "How can I archive different accounts on the same
destination account, each account on a separate folder?"
=======================================================================
Q. How to move emails from one IMAP folder to another, either on the
same IMAP server or on a different one? Example:
How to move all messages older than 1 year, from INBOX to Archive?
R1. Solution:
imapsync ... --minage 365 --folder INBOX --f1f2 INBOX=Archive --delete1 --noexpungeaftereach
Use the option --noexpungeaftereach if speed is a concern.
Use the option --delete1 if you want to move messages, instead of just copy/sync them.
R2. See also file FAQ.Folders_Mapping.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Archiving.txt
=======================================================================
Q. Can imapsync be used to maintain and restore a local offline copy
of a mailbox, eg for backup purposes, using Mbox or Maildir
format, so that if the server fails, then the mailbox could be
reinstated?
R1. No. Imapsync plays with IMAP servers only.
R2. Imapsync can't but rick-sanders-imap-tools can!
See https://github.com/andrewnimmo/rick-sanders-imap-tools
Back up and restore IMAP accounts with imapdump.pl and dumptoIMAP.pl
It looks like imapdump.pl has options to avoid duplicates and be
in an update mode. There are options -u -U
[-u] Don't dump messages already dumped
[-U] Don't dump message if it already exists in the dump directory
I wonder the difference, I've never used Rick's tools.
Remember that doing backups without trying the restore process
is more dangerous than doing no backups at all. No backups makes
people act in a safer way, well, usually...
R3. Use isync or mbsync
https://isync.sourceforge.io/
R4. Use ImportExportTools for Mozilla Thunderbird by Paolo Kaosmos.
ImportExportTools does not do IMAP as a destination.
R5. There is a free online service (French language) allowing you to do
a Maildir or EML or HTML backup at
https://www.powermail.fr/sauvegarder-une-boite-mail/
Prefer Maildir or EML format if you want to be able to restore the
archive. HTML format will be ok to only view the messages, not to
restore them to an imap server account.
I have never used any of this tools so asking me for support with them
won't go far away.
=======================================================================
Q. How to backup a whole year in a subfolder?
R. To backup the whole year 2018 in a subfolder called ARCHIVE_2018:
imapsync ... --search "SENTSINCE 1-Jan-2018 SENTBEFORE 31-Dec-2018" \
--subfolder2 ARCHIVE_2018
=======================================================================
Q. How to archive a whole year in subfolders but without subsubfolders?
R. Instead of the solution with --subfolder2, use:
imapsync ... --search "SENTSINCE 1-Jan-2018 SENTBEFORE 31-Dec-2018" \
--sep2 _ --regextrans2 's{(.*)}{Archive_$1_2018}'
For each folder "FFFF" in the source account, it will create a destination
folder called "Archive_FFFF_2018", at the top level hierarchy.
=======================================================================
=======================================================================

View File

@ -0,0 +1,134 @@
#!/bin/cat
$Id: FAQ.Authentication_failure.txt,v 1.16 2022/08/17 09:18:17 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Authentication_failure.txt
=======================================================================
Imapsync authentication issues
=======================================================================
Questions answered in this FAQ are:
Q. Imapsync fails with the following error, what can I do?
Host1 failure: Error login on [imap.example.com] with user [foo] auth [LOGIN]: 2 NO [AUTHENTICATIONFAILED] Authentication failed
Q. Imapsync fails with the following error, what can I do?
Host2 failure: Error login on [imap.example.com] with user [foo] auth [LOGIN]: 2 BAD Invalid characters in atom
Now the questions again with their answers.
=======================================================================
Q. Imapsync fails with the following error, what can I do?
Host1 failure: Error login on [imap.example.com] with user [foo] auth [LOGIN]: 2 NO [AUTHENTICATIONFAILED] Authentication failed
R. One over four imapsync synchronizations end up quickly with the error message
"Authentication failed" or "NO LOGIN failed" or a similar message.
Authentication failure is the primary failure with imapsync
and since nothing tangible can be done without authentication,
this stage must succeed to go further.
Here is some advice to get you to pass this difficult stage of authentication:
* Triple-check each credential parameter. It's not always a password problem,
it can be the wrong host or the wrong user.
There are three parameters on each side:
* triple check --host1
* triple check --user1
* triple check --password1
* triple check --host2
* triple check --user2
* triple check --password2
* If you can authenticate successfully with another imap client software
like Thunderbird or Outlook or Sparrow then it is a very good sign to
authenticate successfully with imapsync. Examine the parameters of
this other imap client and copy them as is for imapsync.
* Use option --showpasswords
At the beginning of the output, imapsync dumps all its command
line parameters; it's the line after "Command line used:".
With --showpasswords, imapsync prints the passwords received
instead of the string MASKED. It helps with debugging quoting issues.
Option --showpasswords shows passwords again when the IMAP dialog
is dumped by --debugimap option. Search for a line like
"Sending: 2 LOGIN test1 secret1" (secret1 is the password here).
* It is sometimes very hard to quote correctly unusual characters,
especially on Windows (and Mac sometimes). See
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Mac.txt
The quickest trick may be to change the password temporarily
with easy characters like the classical alphabet, one long
string still ensures strong security.
* Sometimes some servers announce they support LOGIN but it
fails because the authentication mechanism
working is something else like CRAM-MD5 or PLAIN.
So:
* Try --authmech1 CRAM-MD5 (or --authmech2 CRAM-MD5)
* Try --authmech1 PLAIN (or --authmech2 PLAIN)
* Manually test a login via ssl.
Use either ncat or telnet-ssl or openssl commands, like in the
following examples with imap.gmail.com server:
ncat --ssl -C imap.gmail.com 993
telnet-ssl -z ssl imap.gmail.com 993
openssl s_client -crlf -connect imap.gmail.com:993
Replace imap.gmail.com by your imap server.
The three commands do the same work here.
Typical dialog for an imap LOGIN command:
* OK Gimap ready for requests from 78.196.254.58 q1mb175739668wix
a LOGIN "gilles.lamiral@gmail.com" "secret"
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE ... ESEARCH
a OK gilles.lamiral@gmail.com Gilles Lamiral authenticated (Success)
b LOGOUT
* BYE LOGOUT Requested
b OK 73 good day (Success)
The client part you have to type is
a LOGIN ...
b LOGOUT
while replacing ... by your credentials values,
other lines are the server responses. Letters
a and b are identifiers of IMAP commands, you have to
type them but you can chose any string instead of a or b.
Some other examples:
ncat --ssl test.lamiral.info 993
c1 LOGIN test1 "secret1"
c2 LOGOUT
openssl s_client -crlf -connect test.lamiral.info:993
c1 LOGIN test1 "secret1"
c2 LOGOUT
ncat --ssl -C outlook.office365.com 993
a LOGIN gilles.lamiral@outlook.com topsecret
b LOGOUT
telnet-ssl -z ssl test.lamiral.info 993
c1 LOGIN test1 "secret1"
c2 LOGOUT
=======================================================================
Q. Imapsync fails with the following error, what can I do?
Host2 failure: Error login on [imap.example.com] with user [foo] auth [LOGIN]: 2 BAD Invalid characters in atom
R. It might be a Dovecot imap server. The password string might
contain special characters that Dovecot doesn't like. Change them.
=======================================================================
=======================================================================

48
FAQ.d/FAQ.Bandwidth.txt Normal file
View File

@ -0,0 +1,48 @@
$Id: FAQ.Bandwidth.txt,v 1.3 2021/05/20 11:46:07 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Bandwidth.txt
=======================================================================
Imapsync bandwidth used
=======================================================================
Questions answered in this FAQ are:
Q. What is the bandwidth used by imapsync?
Now the questions again with their answers.
=======================================================================
Q. What is the bandwidth used by imapsync?
R. From the host where imapsync runs, imapsync opens two imap
connections, one with the source account at host1, one with the
destination account at host2.
So, the global bandwidth used by an imapsync transfer is twice the
volume of the source account, one volume to download the messages from
host1, one volume to upload those messages to host2.
If the host2 is already filled with the messages, imapsync doesn't
transfer them and then the volume transferred is small, this volume is
just made up of the IMAP commands needed to identify the messages on
both sides.
There is no local cache of the email messages, except when a message
is very big; it is then temporarily saved locally between the download
and the upload.
The biggest message seen so far on the online service I call /X is
3.08 GiB while the biggest message transferred is 1.51 GiB. So I
suspect a bug here. Drop me a note if you encounter the same issue,
I'll then dig into it, ie, I'll create a 2 or 3 GiB big message and
play with it :-)
=======================================================================
=======================================================================

56
FAQ.d/FAQ.Big_Mailbox.txt Normal file
View File

@ -0,0 +1,56 @@
#!/bin/cat
$Id: FAQ.Big_Mailbox.txt,v 1.8 2022/04/23 13:45:48 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Big_Mailbox.txt
=====================================================================
Imapsync tips to deal with huge mailboxes
=====================================================================
Questions answered in this FAQ are:
Q. How to deal with huge mailboxes, whose size is over dozens of GB?
Now the questions again with their answers.
=====================================================================
Q. How to deal with huge mailboxes, whose size is over dozens of GB?
R1. It should be ok with imapsync.
Sometimes with big folders, imapsync seems to stall when sizing the
folders or maybe before syncing a huge folder. So you wonder whether
imapsync is doing something or being just frozen. A solution to
increase the verbosity is to add the option --debugimap. Option
--debugimap shows what is currently done, it's quite a big output but
it helps to wait, saying to ourselves "ok it's long but it's working,
let's wait a little more". The real purpose of --debugimap is to show
genuine IMAP commands used and their responses. The problem with
--debugimap is that it is very verbose and so with a big mailbox it
generates a very BIG logfile.
R2. If it is not ok with imapsync, the sync finally fails, then
consider splitting the whole sync into smaller chunks using
the option --search
For example, select messages only for a given year:
...
..imapsync ... --search "SENTSINCE 1-Jan-2017 SENTBEFORE 31-Dec-2017"
imapsync ... --search "SENTSINCE 1-Jan-2018 SENTBEFORE 31-Dec-2018"
imapsync ... --search "SENTSINCE 1-Jan-2019 SENTBEFORE 31-Dec-2019"
R3. Use option --useuid
Option --useuid speeds up the process but --useuid implies --usecache
and --usecache may trigger some issues sometimes. Read:
https://imapsync.lamiral.info/FAQ.d/FAQ.Use_cache.txt
Don't use --useuid in Docker context, the cache will be destroyed
each time, or put it in a stable place, I don't know how to do it,
I just know it's possible and not very difficult.
=====================================================================
=====================================================================

View File

@ -0,0 +1,103 @@
$Id: FAQ.Capacity_Planning.txt,v 1.6 2022/04/23 13:45:59 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Capacity_Planning.txt
======================================================================
Imapsync tips for Capacity Planning.
======================================================================
I plan to go to a distributed architecture for the online service. I'm
doing some capacity planning. Imapsync takes memory, cpu and bandwidth
in a relatively deterministic values.
My current question is: Shall I use
* N 2GB hosts
* N/2 4GB hosts
* N/4 8GB hosts
Let's do some observations and maths
The observations are done on the standalone imapsync online which
characteristics are:
CPU: Intel i5-2300 with 4 cores
RAM: 16 GB
NET: 100 Mbps symmetrical, 12.5 MBytes/s symmetrical,
so 25 MBytes/s max for a global imapsync rate.
Disks: I don't know.
System: FreeBSD 11.4
===== CPU =====
The CPU can be an issue. On average, an imapsync run takes 5% of the
overall cpu time for a Intel i5-2300 with 4 cores. It implies 20
imapsync runs is ok on the current online host before the cpus become
the bottleneck. As a rule of thumb, imapsync takes 20% of a cpu core.
In the Intel i5-2300 4 cores, so far, the maximum number of imapsync
processes has been 68, which is 3 times what the system should
support in a standard imapsync workload. For this workload, the
imapsync performamces were not good, the server could not handle the load
and was even out of order for a while. For 40 imapsync processes, the
the performances are ok.
===== RAM =====
The RAM can be an issue. On average, an imapsync run takes 250 MB. So
4 imapsync processes per GB is the limit before swapping to disk,
which is a known phenomenon telling when memory becomes the bottleneck.
16 GB allows 64 imapsync processes.
On the 16GB, so far, the maximum memory usage taken by imapsync processes was
13 GB. For that value, the one minute load was 10 or more, the number of
imapsync processes was around 40, the total bandwidth was around 19 MiB/s.
===== LINK =====
The Bandwidth I/O can be an issue. The "Average bandwidth rate" value
given by imapsync at the end of a transfer, and also the bandwidth
rate given during the sync on the ETA line, is the total size of all
messages copied divided by the time passed. If imapsync is run between
two foreign imap servers then the total size transferred on the
network link is twice this value, one time when getting the message
from host1, rx, and one time sending the message to host2, tx.
On average, an imapsync runs at 3 Mbps both ways, rx and tx, so 6 Mbps
in total. So a 100 Mbps symmetric link allows 33 imapsync processes
before the link becomes the bottleneck.
The best minutes observed so far are a global 187 Mpbs rate (~23 MiB/s)
on a 100 Mbps symmetrical link done by 28 imapsync processes in parallel,
6.7 Mbps per process at that minute. But the best minutes observed for a
single imapsync process is at 103 Mbps (~13 MiB/s), the rx/tx link limit.
===== DISKS =====
The harddisk I/O can be an issue. I don't measure it yet because
imapsync doesn't perform heavy I/0 where it runs. Well, I don't know,
no measure is no knowledge, just guesses.
===== Repartition ======
Let's consider we have a bunch of different hosts able to run imapsync
processes. How should we distribute imapsync jobs among them?
A simple rule is "do not add more load to a host when one of the resources
has reached its maximal. The resources are memory, bandwidth, cpu, disk.
I ignore the disk part for now.
maximum number of imapsync processes for a host
= min( 4 * RAM_in_GB, 10 * nb_cores, bandwidth_in_mbps / 3 )

View File

@ -0,0 +1,107 @@
#!/bin/cat
$Id: FAQ.Contacts_Calendars.txt,v 1.17 2022/05/31 22:21:55 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Contacts_Calendars.txt
=======================================================================
Imapsync issues about syncing Contacts & Calendars.
=======================================================================
Questions answered in this FAQ are:
Q. Can I copy or sync Contacts, Calendars, Tasks or Chat messages with imapsync?
Q. How can I copy or synchronize Calendars or Contacts?
Q. How can I avoid copying Calendar or Contacts folders?
Now the questions again with their answers.
=======================================================================
Q. Can I copy or sync Contacts, Calendars, Tasks or Chat messages with imapsync?
R. No, Imapsync can't migrate Contacts, Calendars, Tasks nor Chat messages.
It's because most IMAP servers don't get Contacts and Calendar
events via IMAP. In other words, messages synced by imapsync from
Contacts/Calendars/Tasks/Chat folders are not used by email servers to set
or get the contacts, calendars, tasks, or chat messages.
No way via IMAP, no way via imapsync.
So it's a good idea to avoid syncing contacts and calendars when
using imapsync. See at the end of this document
"How can I avoid copying Calendar or Contacts folders?"
But see also the question below
"How can I copy or synchronize Calendars or Contacts?".
=======================================================================
Q. How can I copy or synchronize Calendars or Contacts?
R1. Synchronizing Calendars or Contacts can't be done with imapsync.
For detailed explanations, see the question above
"Can I copy or sync Calendars or Contacts with imapsync?".
R2. Synchronizing Calendars or Contacts can be done, not
with imapsync, depending on the imap server software tool used.
a) From Exchange to Exchange, export contacts and calendar to
PST format files on host1 and import them on host2.
b) From Gmail to Gmail, export and import calendars in ical format,
extension for those files is .ics.
Contacts can be copied using a csv file. See the help page
http://support.google.com/mail/bin/topic.py?hl=en&topic=1669027
c) For Outlook, use caldavsynchronizer.
Homepage: http://caldavsynchronizer.org/
Outlook CalDav Synchronizer is a free Outlook Plugin, which synchronizes
events, tasks, and contacts between Outlook and Google, SOGo, Nextcloud or
any other CalDAV or CardDAV server.
Supported Outlook versions are Office 365, 2019, 2016, 2013, 2010, and 2007.
Free and Open Source software tool.
d) Etc. Search the web. There are also specific tools and paid services.
There's no silver bullet to migrate Calendars and Contacts,
if you find one, tell me!
Also, consider asking experts at the Sumatra company
https://twitter.com/sumatra_dev
http://sumatra.com/
e) I plan to make tools for that but so far nothing has begun
more than this discussion:
http://linux-france.tk/prj/imapsync_list/msg01797.html
http://linux-france.tk/prj/imapsync_list/msg01811.html
=======================================================================
Q. How can I avoid copying Calendar or Contacts folders?
R. You can avoid synchronizing Calendar or Contacts folders with
the --exclude option. First, you have to search what is their
exact name. The listing of folders printed by imapsync at the
beginning of a run will surely help to find their names.
Assuming their names are "Calendars" and "Contacts", use:
imapsync ... --exclude "Calendar" --exclude "Contacts"
Be aware that this syntax will exclude all folders that have
Calendar or Contacts in their names, as a substring,
like "MyCalendarZ" or "TheContactsImDoneWith".
Also, case matters: "calendar" does not match "Calendar".
To strictly avoid only "Calendar" or "Contacts" use
the following
On windows:
imapsync ... --exclude "^Calendar$" --exclude "^Contacts$"
On Unix:
imapsync ... --exclude '^Calendar$' --exclude '^Contacts$'
or
imapsync ... --exclude "^Calendar\$" --exclude "^Contacts\$"
=======================================================================
=======================================================================

127
FAQ.d/FAQ.Cyrus.txt Normal file
View File

@ -0,0 +1,127 @@
#!/bin/cat
$Id: FAQ.Cyrus.txt,v 1.2 2021/02/19 13:39:03 gilles Exp gilles $
=======================================================================
Imapsync tips for the Cyrus imap server
=======================================================================
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Cyrus.txt
Questions answered in this FAQ are:
Q. How to migrate from cyrus with an admin account?
Q. I'm migrating from WU to Cyrus, and the mail folders are under
/home/user/mail but the tool copies everything in /home/user, how
can i avoid that?
Q. I'm migrating from WU to Cyrus, and the mail folders are under
/home/user/mail directory. When imapsync creates the folders in
the new cyrus imap server, it makes a folder "mail" and below that
folder puts all the mail folders the user have in /home/user/mail,
i would like to have all those folders directly under INBOX.
Q. Migrating from Groupwise to Cyrus
Now the questions again with their answers.
======================================================================
Q. How to migrate from cyrus with an admin account?
R. Use:
imapsync ... \
--authuser1 admin_user ----password1 admin_user_password \
--user1 foo_user --ssl1
Instead of --ssl1 the alternative --tls1 can be used.
With --authuser1, the option --authmech1 PLAIN is set
automatically, you don't have to add it.
PLAIN authentication is the only way to go with --authuser1 for now.
So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
Do not forget the option --ssl1 or --tls1 since PLAIN auth is only
supported with ssl encryption most of the time. But it can
work without --ssl1 nor --tls1 if PLAIN is permitted in clear text
transmissions (the normal mode).
Add the AdminAccount to admins line in /etc/imapd.conf
Give AdminAccount lrswipkxtecda to the Cyrus Imap account
being migrated from, "joe" here.
Here is an example:
imapsync \
--host1 server1 \
--user1 joe \
--authuser1 AdminAccount \
--password1 AdminAccountPassword \
--ssl1 \
--host2 server2 \
--user2 joe \
--password2 joespassonserver2 \
--exclude "^user\."
=======================================================================
Q. I'm migrating from WU to Cyrus, and the mail folders are under
/home/user/mail but the tool copies everything in /home/user, how
can i avoid that?
Two solutions:
R. Use
imapsync ... --include '^mail'
R. or (better)
imapsync ... --subscribed --subscribe
=======================================================================
Q. I'm migrating from WU to Cyrus, and the mail folders are under
/home/user/mail directory. When imapsync creates the folders in
the new cyrus imap server, it makes a folder "mail" and below that
folder puts all the mail folders the user have in /home/user/mail,
i would like to have all those folders directly under INBOX.
R. Use
imapsync ... --regextrans2 's/^mail/INBOX/' --dry
look at the simulation and if all transformations seem
good then remove the --dry option.
=======================================================================
Q. Migrating from Groupwise to Cyrus
R. By Jamie Neil:
I eventually managed to get the mail to migrate without errors using the
following options:
--maxage 3650
- some messages just don't seem to want to transfer and produce the
perl errors I mentioned before. This prevents the errors, but the
bad messages don't transfer.
Even though the mail migrated OK, there are a couple of gotchas with
Groupwise IMAP:
1) Some of the GW folders are not real folders and are not available
to IMAP, the main problem one being "Sent Items". I could find no way
of coping the contents of these folders. The nearest I got was to
create a "real" folder and copy/move the sent items into it, but
imapsync still didn't see the messages (I think because there is
something funny about the reported dates/sizes).
It think this problem has been rectified in GW6.5.
2) Version 6.0.1 of the Groupwise Internet Connector sucks. I was
getting server aborts when I pushed it a bit hard! I eventually had to
upgrade to 6.0.4 which seems to be a lot more stable.
=======================================================================

22
FAQ.d/FAQ.DBmail.txt Normal file
View File

@ -0,0 +1,22 @@
#!/bin/cat
$Id: FAQ.DBmail.txt,v 1.1 2020/11/04 08:43:05 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.DBmail.txt
=======================================================================
Imapsync tips for DBmail accounts.
=======================================================================
=======================================================================
Q. How to migrate from or to DBmail?
R. In order to avoid duplicates, use:
imapsync ... --useheader Message-Id
More explanations and details at
https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

122
FAQ.d/FAQ.Dates.txt Normal file
View File

@ -0,0 +1,122 @@
#!/bin/cat
$Id: FAQ.Dates.txt,v 1.11 2019/12/23 12:46:50 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Dates.txt
=======================================================================
Imapsync tips about dates.
=======================================================================
Questions answered in this FAQ are:
Q. We have found that the time and date displayed have been changed to
the time at which the file was synchronized. What happened? Any fix?
Q. Is there a way to set any message with
"Date: (Invalid)" to a valid one like
"Date: Thu, 1 Jun 2017 23:59:59 +0000"?
=======================================================================
Q. We have found that the time and date displayed have been changed to
the time at which the file was synchronized. What happened? Any fix?
R. This is the case by default with some email readers like:
- Outlook 2003
- Ipad
- Iphone
- spark https://sparkmailapp.com/ (iphone app)
- Edison mail https://apps.apple.com/us/app/email-edison-mail/id922793622 (iphone app)
- ...
but not with:
- Mutt
- Thunderbird
- Zimbra
- Gmail
First, a thing to keep in mind.
Imapsync does not touch any byte of messages unless told to do so by
special and rarely used options like --regexmess or --pipemess.
Messages on both parts should be exactly identical because imapsync
does a verbatim sync of all messages, by default.
Now I explain the whole picture about dates of messages.
There are several different dates attached to any message.
First, there is the "Date:" header, which belongs to the message itself.
You can see this "Date:" header line by asking "View->Message Source"
in most email client software (Ctrl-u in Thunderbird).
Most of the time, this "Date:" header date is set by the MUA,
MUA stands for "Mail User Agent".
MUA can be Outlook, Thunderbird, a webmail or Mutt etc.
The "Date:" header is usually the date the message was written
or sent the first time, at the writer side.
This "Date:" header line is never changed by any transfer or copy.
If an email reader uses the "Date:" header for displaying the date of
a message then no problem should arise.
Another date is the "internal date". In IMAP, the internal date is
handled by the imap server and normally it corresponds to the arrival
date in the mailbox. This date is not part of the email message
itself, it is a metadata kept by the imap server.
The IMAP protocol allows the internal date to be set by an email client
when a message is delivered by the imap APPEND command.
Imapsync synchronizes these internal dates, by default, so internal dates
on host2 should then be the same as the internal dates on host1.
If an email reader uses the internal date for displaying the date of a
message then the sync date problem only occurs when the host2 imap
server ignores the internal date given by imapsync during the APPEND
imap command. It happens with some imap servers. Bad luck.
There are also the "Received:" header lines. Each time a message
travels a SMTP server, this one adds a "Received:" header
line. Sometimes some email clients use the last "Received:" header
date as the date of the message. And some IMAP servers software add a
"Received:" line after and imap transfer. If those both conditions are
met then the date displayed become the transfer date even if imapsync
has done its best to keep all the dates synchronized. Bad luck.
Solutions:
a) Use another email client or configure it in order that it sorts messages
by sent dates, the "Date:" header line.
b) Use a imap server that respects the imap RFC and accepts
the internal date set by imapsync.
c) Try to understand why the email client software shows another date
than the "Date:" header line.
=======================================================================
Q. Is there a way to set any message with
"Date: (Invalid)" to a valid one like
"Date: Thu, 1 Jun 2017 23:59:59 +0000"?
R. Yes, there is a way with option --regexmess
First, let's select only messages with a buggy Date header:
--search "HEADER Date Invalid"
Second, let's change this line by a valid one,
on windows:
--regexmess "s{\A(.*?(?! ^$))^Date:\ \(Invalid\)(.*?)$}{$1Date: Thu, 1 Jun 2017 23:59:59 +0000}xms"
on Unix (replaced enclosing double-quotes " by single quotes ' ):
--regexmess 's{\A(.*?(?! ^$))^Date:\ \(Invalid\)(.*?)$}{$1Date: Thu, 1 Jun 2017 23:59:59 +0000}xms'
=======================================================================
=======================================================================

54
FAQ.d/FAQ.David_Tobit.txt Normal file
View File

@ -0,0 +1,54 @@
#!/bin/cat
$Id: FAQ.David_Tobit.txt,v 1.5 2022/04/05 13:48:52 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.David_Tobit.txt
=======================================================================
Imapsync tips for David Tobit imap server
=======================================================================
=======================================================================
Q. Migrating to David Tobit V12
(Banner: "* OK IMAP4rev1 David.fx Mail Access Server MA-12.00a (0271)")
imapsync ... --sep2 / --create_folder_old
Thanks to Ludwig Behm
=======================================================================
Q. Migrating from David Tobit V10 (DvISE Mail Access Server MA-...)
R. Use the following options:
imapsync ... --prefix1 "" --sep1 / --idatefromheader ^
--nofoldersizes --useuid --nocheckmessageexists
=======================================================================
Q. Migrating from David Tobit V8
(Banner: "* OK IMAP4rev1 DvISE Mail Access Server MA-8.10a (0126)")
First try above V10 solution since improvments have been made
to support Tobit.
R. Use the following options :
imapsync ... --prefix1 INBOX. --sep1 / --subscribe --subscribed
=======================================================================
Q. Migrating from Tobit David Server 6
(Banner: "* OK IMAP4rev1 DvISE Mail Access Server MA-6.60a (0118)")
First try above V10 solution since improvments have been made
to support Tobit.
R. Look at the discussion:
http://linux-france.tk/prj/imapsync_list/msg00583.html
http://linux-france.tk/prj/imapsync_list/threads.html#00583
patch saved in ./patches/imapsync-1.337_tobit_V6.patch
=======================================================================

160
FAQ.d/FAQ.Docker.txt Normal file
View File

@ -0,0 +1,160 @@
#!/bin/cat
# $Id: FAQ.Docker.txt,v 1.10 2021/07/12 21:22:14 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Docker.txt
======================================================================
Installing and using imapsync docker image
======================================================================
Questions answered in this FAQ are:
Q. How can I install and use the imapsync Docker image on my amd64 system?
Q. How can I install and use the imapsync Docker image on my i386 system?
Q. How can I install or update only the imapsync Docker image on my system?
Q. Where is the imapsync Docker hub location?
Q. How can I mask the passwords on the command line without mounting
inside the container?
Q. Why imapsync isn't logging to a file in Docker context.
How can I allow it?
Q. Any tips for the Docker Mailcow distribution?
Q. How can I build my own image?
Q. What do you do to update the docker hub image?
Now the questions again with their answers.
======================================================================
Q. How can I install and use the imapsync Docker image on my amd64 system?
R. Install Docker on your system. Once Docker is installed on your system,
all you have to do in order install and run imapsync is the command line:
docker run gilleslamiral/imapsync imapsync <usual imapsync arguments>
======================================================================
Q. How can I install and use the imapsync Docker image on my i386 system?
R1. You can't do it directly for now. It will throw this error when running it:
docker run gilleslamiral/imapsync imapsync
standard_init_linux.go:211: exec user process caused "exec format error"
I plan to make the image multi-plateform. Drop me a note if you are
eager to have it.
R2. Build the image on a i386 system with the exact same Dockerfile provided
and run it on your i386 system.
======================================================================
Q. How can I install or update only the imapsync Docker image on my system?
R. To install or update the imapsync image, run:
docker pull gilleslamiral/imapsync
======================================================================
Q. Where is the imapsync Docker hub location?
R. Here:
https://hub.docker.com/r/gilleslamiral/imapsync/
======================================================================
Q. How can I mask the passwords on the command line without mounting
anything inside the container?
R. Use a file defining the environment variables
IMAPSYNC_PASSWORD1 and IMAPSYNC_PASSWORD2
cat ./secret.txt
IMAPSYNC_PASSWORD1=secret1
IMAPSYNC_PASSWORD2=secret2
Use that file like this for a run:
docker run --env-file ./secret.txt gilleslamiral/imapsync imapsync ...
See:
https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file
https://stackoverflow.com/a/30494145/491175
======================================================================
Q. Why imapsync isn't logging to a file in Docker context.
How can I allow it?
R. In Docker context, writing the log is disabled by default because you
don't have an easy access to the logfile after the sync. The logfile is
inside the docker and no longer available after the sync, unless
a special mount is done before.
Starting with imapsync release 2.113 the logging can be turned on
by using the option --log. You can add access to the logfile with a mount
or similar. See https://docs.docker.com/storage/volumes/"
See also
https://github.com/imapsync/imapsync/issues/283
======================================================================
Q. Any tips for the Docker Mailcow distribution?
R. With the Mailcow distribution, imapsync is running inside a Docker
container. Quoting Mathilde:
To make it work with Mailcow, options should be added like this:
--regexflag=s/\\Indexed//gi
Without spaces, without quotes.
See the original Mathilde's comment:
https://github.com/imapsync/imapsync/issues/201#issuecomment-559500077
======================================================================
Q. How can I build my own image?
R. Use the following command to build the image:
docker build -t gilleslamiral/imapsync .
with the Dockerfile in the current directory.
The latest Dockerfile can be found at
https://imapsync.lamiral.info/INSTALL.d/Dockerfile
or
https://hub.docker.com/r/gilleslamiral/imapsync
or
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Docker_build.txt
======================================================================
Q. What do you do to update the docker hub image?
R. I do in my imapsync directory:
make docker_build
make docker_upload_docker_hub
Then I go to
https://hub.docker.com/r/gilleslamiral/imapsync
I edit and update the docker "Readme" with the last Dockerfile I used.
I also update INSTALL.Docker_build.txt with the last Dockerfile
I also update the last Dockerfile on the imapsync site with:
make upload_index
======================================================================
======================================================================

142
FAQ.d/FAQ.Domino.txt Normal file
View File

@ -0,0 +1,142 @@
#!/bin/cat
# $Id: FAQ.Domino.txt,v 1.13 2020/09/06 20:33:46 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Domino.txt
=======================================================================
Imapsync tips for Domino.
=======================================================================
Questions answered in this FAQ are:
Q. How to use imapsync from Domino Notes to XXX?
Q. How to use imapsync from XXX to Domino Notes?
Q. Whan I use --maxage imapsync sees no messages at all.
What can I do?
Q. My usual email software client like Thunderbird sees more messages
than imapsync. How can it be possible?
Now the questions again with their answers:
=======================================================================
Q. How to use imapsync from Domino Notes to XXX?
R1. Use --domino1, like this:
imapsync ... --domino1
R2. Or use the following options:
On Windows use:
imapsync.exe ... --sep1 "\\" --prefix1 "" --messageidnodomain
On Unix use:
imapsync ... --sep1 '\' --prefix1 '' --messageidnodomain
=======================================================================
Q. How to use imapsync from XXX to Domino Notes
R1. Use --domino2, like this:
imapsync ... --domino2
R2. Or follow this:
Domino doesn't accept INBOX subfolders.
On Windows:
imapsync.exe ... ^
--sep2 "\\" --prefix2 "" --messageidnodomain ^
--regextrans2 "s,^Inbox\\(.*),$1,i"
On Unix:
imapsync ... \
--sep2 '\' --prefix2 '' --messageidnodomain \
--regextrans2 's,^Inbox\\(.*),$1,i'
If you want to sync the complete host1 mailbox in a subfolder called OLDBOX use:
On Windows:
imapsync.exe ... ^
--sep2 "\\" --prefix2 "" --messageidnodomain ^
--subfolder2 "OLDBOX" --justfolders --dry
On Unix:
imapsync ... \
--sep2 '\' --prefix2 '' --messageidnodomain \
--subfolder2 'OLDBOX' --justfolders --dry
If the output is correct for you then remove --dry and have a run.
Verify the folder hierarchy is good on host2 then remove --justfolders to
sync the messages.
For Domino with imapsync and Domino on Linux:
(Thanks to Zafer)
R. We've migrated about 5000 users from cyrus to lotus domino 853!
Mails are being copied into the new created lotus domino mailbox
retaining the existing folder structure from the old mail database.
This means that sent mails can be found in the sent folder and
mails from the old inbox are copied into the new inbox
We also marked migrated mails with the header Migratedbyus
For Domino 853FP6 on Linux, we used this command on Unix:
imapsync \
--host1 hhh1 --user1 uuu1 --password1 ppp1 \
--host2 hhh2 --user2 uuu2 --password2 ppp2 \
--exclude "^INBOX/Trash" --exclude 'Junk|Drafts' \
--regextrans2 's#^INBOX/Sent$#^Sent#' \
--prefix2 "" --sep2 "\/" --messageidnodomain \
--regexmess 's{\A(.*?(?! ^$))^Date:(.*?)$}{$1Migratedbyus:$2\nx-MailDate:$2}gxms'
=======================================================================
Q. When I use --maxage option, imapsync sees no messages at all.
What can I do?
R. Use --noabletosearch
imapsync ... --noabletosearch
Same thing with --minage
=======================================================================
Q. On Domino this errors occurs:
"Login failure, cannot convert Mailbox: code 02:46"
R. Follow
https://atnotes.de/index.php?topic=44888.0
Use:
load convert -e mail\mailfilename.nsf
=======================================================================
Q. My usual email software client like Thunderbird sees more messages
than imapsync. How can it be possible?
R. (Issue given and solved by Falko Veith)
"As you might know, it is possible to replicate databases, aka mailboxes,
across domino servers. THAT is the explanation. Thunderbird and some Notes
clients may be connected to one server and imapsync to another.
As funny and simply the solution is, as mysterious is why those two
databases are not completely in sync - new emails are replicated but not
the old ones. One mystery solved - and a new one to go.
Thanks to Falko Veith for this question and the answer!
=======================================================================

103
FAQ.d/FAQ.Dovecot.txt Normal file
View File

@ -0,0 +1,103 @@
#!/bin/cat
$Id: FAQ.Dovecot.txt,v 1.14 2022/06/16 09:06:34 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Dovecot.txt
=======================================================================
Imapsync tips for Dovecot. Specific issues and solutions.
=======================================================================
Questions answered in this FAQ are:
Q. Is there imapsync specific options to deal with Dovecot?
Q. The login fails with the error:
Err 1/1: Host2 failure: Error login on [...] with user [...] auth [LOGIN]: timeout waiting 120s for data from server (5x)
What can it be?
Q. Subfolders are not created by imapsync and the error is
"Mailbox doesn't allow inferior mailboxes"
Q. How to migrate to Dovecot with an admin/MasterUser account?
Q. I have a mailbox that "dovecot backup" gives errors on that
I can't get rid of: "UID inserted in the middle of mailbox".
What can I do?
Q. Imapsync encounters timeouts and Dovecot log says
"mailpiler dovecot: imap(xxx): Error: Timeout (180s) while waiting for dotlock for transaction log file ..."
What can I do?
Now the questions again with their answers.
=======================================================================
Q. Is there imapsync specific options to deal with Dovecot?
R. Dovecot is ok with imapsync, nothing special is needed, just the
basic options, ie, the credentials, are needed.
=======================================================================
Q. The login fails with the error:
Err 1/1: Host2 failure: Error login on [...] with user [...] auth [LOGIN]: timeout waiting 120s for data from server (5x)
What can it be?
R. It can be special characters in the password. Change it with only
normal characters, characters from a-z A-Z 0-9
=======================================================================
Q. How to migrate to Dovecot with an admin/MasterUser account?
R. Dovecot uses the same syntax as uw-imap
imapsync ... --user2="loginuser*admin_user" --password2 "admin_user_password"
To setup a Dovecot MasterUser follow the document
https://doc.dovecot.org/configuration_manual/authentication/master_users/
=======================================================================
Q. I have a mailbox that "dovecot backup" gives errors on that
I can't get rid of: "UID inserted in the middle of mailbox".
What can I do?
R. Solution, run:
doveadm -v deduplicate -m -u martin mailbox A/Posted
it will get rid of the duplicates (it can also be run without "-m")
and this also allows "dovecot backup" to work.
Thanks to Martin Ward for this input!
=======================================================================
Q. Subfolders are not created by imapsync and the error is
"Mailbox doesn't allow inferior mailboxes"
R. Dovecot usually supports subfolders with messages and subfolders,
it's a Dovecot configuration issue.
If you can change the Dovecot configuration dovecot.conf,
then try this:
mail_location = maildir:~/Maildir:LAYOUT=fs
See details at:
https://superuser.com/questions/813362/sub-folders-in-dovecot
https://wiki2.dovecot.org/MailLocation/Maildir
=======================================================================
Q. Imapsync encounters timeouts and Dovecot log says
"mailpiler dovecot: imap(xxx): Error: Timeout (180s) while waiting for dotlock for transaction log file ..."
What can I do?
R. It looks like an NFS issue with Dovecot.
Solve it with Dovecot support or use something else than NFS.
I'll be glad to share here any solution you find.
(Thanks to Mate Skoblar for reporting this issue)
=======================================================================
=======================================================================

279
FAQ.d/FAQ.Duplicates.txt Normal file
View File

@ -0,0 +1,279 @@
#!/bin/cat
$Id: FAQ.Duplicates.txt,v 1.25 2021/10/06 20:21:13 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt
=======================================================================
Imapsync tips about duplicated messages issues.
=======================================================================
Questions answered in this FAQ are:
Q. Without imapsync, I made several copies that partially failed and it
ended with many duplicates/triplicates messages or more. Can I clean
up the account with imapsync and how?
Q. How does imapsync identify messages and duplicates?
Q. How can I know if imapsync will generate duplicates on a second run?
Q: I found multiple copies, duplicates, when I run imapsync twice or
more. What the hell is happening?
Q. imapsync calculates 479 messages in a folder but only transfers 400
messages. What is happening?
Q. imapsync doesn't synchronize duplicates by default but I want to.
How can I synchronize duplicates?
Q. How can I remove duplicates in a lone account?
Now the questions again with their answers.
=======================================================================
Q. Without imapsync, I made several copies that partially failed and it
ended with many duplicates/triplicates messages or more. Can I clean
up the account with imapsync and how?
R. Yes.
See the Q/R "How can I remove duplicates in a lone account?" below.
=======================================================================
Q. How does imapsync identify messages and duplicates?
R. Imapsync by default identify messages by their headers "Message-Id"
and "Received". Usually, for a given message, "Message-Id" appears one
time while multiple "Received" headers are common.
For imapsync, messages with the same "Message-Id" and "Received" headers
are consider identical, ie, duplicates.
=======================================================================
Q. How can I know if imapsync will generate duplicates on a second run?
R. To see if imapsync will generate duplicates on a second run, start
a second run with --dry option added. With --dry, imapsync will
show whether it would mistakenly copy messages again, but without
really copying them:
imapsync ... --dry
The final stats should also show a positive value for the line
"Messages skipped:" since most of the skipped messages are skipped
because they are already on host2. Example of final stats:
++++ Statistics
Transfer started on : Thu Aug 31 04:28:32 2017
Transfer ended on : Thu Aug 31 04:28:44 2017
Transfer time : 11.7 sec
Folders synced : 1/1 synced
Messages transferred : 0
Messages skipped : 1555
=======================================================================
Q: I found multiple copies, duplicates, when I run imapsync twice or
more. What the hell is happening?
R0. First, some explanations to understand the issue.
Normally and by default, imapsync doesn't generate duplicates.
So, if it does generate duplicates it means a problem occurs
with message identification. It happens sometimes with IMAP
servers changing the "Message-Id" header line or one or more
of the "Received:" header lines in the header part of messages.
By default, Imapsync uses "Message-Id" header line and
"Received:" header lines to identify messages on both sides.
R1. This solution is R3 simplified.
A quick practical solution is to change the way imapsync
identify messages that works most of the time. But since
you're reading this because you encountered duplicates issue,
let's check this solution in a safe way.
First use the same commmand with additionnal options:
imapsync ... --useheader "Message-Id" --dry
The previous command does nothing real but it will show you
if imapsync handles duplicates in a better way.
The criterium is to search at the end of the sync for a line
like this one:
Messages skipped : 1555
where 1555 is an example but reflects mostly the number
of all messages already transferred.
If you end with:
Messages skipped : 0
then don't go on, it means imapsync is still suffering to
identify messages.
If you end with many messages skipped then it's very
good and now you can safely resync the mailbox
and get rid of the duplicate messages on host2 with:
imapsync ... --useheader "Message-Id" --delete2duplicates
End of the problem!
R2.
A second solution is to use option --useuid.
With option --useuid, imapsync doesn't use header lines
to identify and compare messages in folders.
Instead of some headers, --useuid tell imapsync to use
the imap UIDs given by imap servers on both sides.
To avoid duplicates on next runs, imapsync uses a local cache
where it keeps UIDs already transferred.
imapsync ... --useuid
There is an issue when --useuid is not used the first time.
A big issue with --useuid is that it doesn't generate duplicates if
used from the first time but it does generate duplicates after a previous
run without --useuid (because it then uses a different method to identify
the messages).
A solution? Two solutions.
The easiest is --delete2 if you are permitted to use it.
Option --delete2 removes messages on host2
that are not on host1. So, with --delete2 you go for resyncing all
messages again. All previously transferred messages are deleted,
but also messages previously there without imapsync.
So --useuid --delete2 is an easy way to remove duplicates but it
is not suitable in all contexts. The good context is that the host2
account must be considered as a strict replication of the host1
account, ie, host2 not active yet.
A second solution, better if R3 works (see R3 below), is to build
the cache before using --useuid
First sync:
imapsync ... --useheader "Message-Id" --addheader --usecache
Next syncs:
imapsync ... --useuid
imapsync ... --useuid
...
R3.
Best way if you can follow it.
Multiple copies of the emails on the destination server. Some IMAP
servers (Domino for example) change some headers for each message
transferred. All messages are transferred again and again each time you
run imapsync. This is bad of course. The explanation is that imapsync
considers messages are not the same on each side, default headers used
to identify the messages have changed.
You can look at the headers found by imapsync by using the --debug
option (and search for the message on both part), Header lines from
the source server begin with a "FH:" prefix, Header lines from the
destination server begin with a "TH:" prefix. Since --debug is very
verbose I suggest to isolate a email in a specific folder in case you
want to forward me the output.
A way to avoid this problem is by using option --useheader with
a different set than the default ones used by imapsync.
The default set is equivalent to:
imapsync ... --useheader "Message-Id" --useheader "Received"
The problem now is that what can be used instead of Message-Id
and Received lines? Often standalone Message-Id works:
imapsync ... --useheader "Message-Id"
Once imapsync does not generate duplicates, the previous duplicates
can be deleted with option --delete2duplicates
imapsync ... --useheader "Message-Id" --delete2duplicates
Another good way toward a solution is to isolate two or three messages
in a BUG folder and send me the --debug output by email to
gilles@lamiral.info
imapsync ... --debug --folder BUG
I will take a close look at the log and modify imapsync to fix
this faulty duplicate behavior.
Remark. (Trick found by Tomasz Kaczmarski)
Option --useheader "Message-Id" asks the server to send only header
lines beginning with "Message-Id". Some (buggy) servers send the whole
header (all lines) instead of the "Message-Id" line. In that case, a
trick to keep the --useheader filtering behavior is to use
--skipheader with a negative lookahead pattern:
imapsync ... --skipheader "^(?!Message-Id)"
Read it as "skip every header except Message-Id".
=======================================================================
Q. imapsync calculates 479 messages in a folder but only transfers 400
messages. What is happening?
R1. Unless --useuid is used, imapsync considers a header part
of a message to identify a message on both sides.
By default the header part used is lines "Message-Id:" "Message-ID:"
and "Received:" or specific lines depending on --useheader
--skipheader. Whole header can be set by --useheader ALL
Consequences:
1) Duplicate messages on host1 (identical header) are not transferred.
The result is that you can have more messages on host1 than on host2.
R2. With option --useuid imapsync doesn't use headers to identify
messages on both sides but it uses their imap uid identifier.
In that case duplicates on host1 are also transferred on host2.
=======================================================================
Q. imapsync doesn't synchronize duplicates by default but I want to.
How can I synchronize duplicates?
R1. Use the option --syncduplicates
R2. Use the option --useuid
If you have already synchronized two mailboxes without --useuid then
using it right away will generate duplicates on host2. To avoid that
behavior, you have to perform a first run with --usecache to build
the local UID cache. Then the next runs with --useuid
There are potentially issues with --usecache. They can be solved.
Read the document FAQ.Use_cache.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Use_cache.txt
So to finalise how to synchronize duplicates:
imapsync ... --tmpdir . --usecache
imapsync ... --tmpdir . --useuid
imapsync ... --tmpdir . --useuid
...
Here --tmpdir value is the dot "." meaning "current directory".
Surrounding it with double-quotes is optional.
If the two mailboxes haven't been already synchronized then the
first run with --usecache is useless.
=======================================================================
Q. How can I remove duplicates in a lone account?
R. In order to remove duplicates in a lone account, just run imapsync
on the same account as source and destination, plus the
option --delete2duplicates, ie, with
host1 == host2, user1 == user2, password1 == password2
imapsync ... --delete2duplicates
=======================================================================
=======================================================================

153
FAQ.d/FAQ.Emptying.txt Normal file
View File

@ -0,0 +1,153 @@
#!/bin/cat
$Id: FAQ.Emptying.txt,v 1.14 2022/06/08 11:08:39 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Emptying.txt
=======================================================================
Imapsync tips about deletions.
=======================================================================
Questions answered in this FAQ are:
Q. How to delete all emails of all folders of an account with imapsync?
Q. How to delete all folders of an account with imapsync?
Q. How to delete some folders of an account with imapsync?
Q. How to revert a sync that went to the wrong account?
Q. How to delete emails older than 90 days with imapsync?
Q. How to delete emails matching a certain criterium with imapsync?
Q. Is there a way to delete the destination folders when the source
folders are no longer there?
Now the questions again with their answers.
=======================================================================
Q. How to delete all emails of all folders of an account with imapsync?
R. Use the same account on both sides with option --delete1
--noexpungeaftereach
Option --noexpungeaftereach is just to speed up the deletions.
Example:
imapsync \
--host1 test.lamiral.info --user1 foo --password1 secret \
--host2 test.lamiral.info --user2 foo --password2 secret \
--delete1 --noexpungeaftereach
=======================================================================
Q. How to delete all folders of an account with imapsync?
R. Use the same account on both sides with the options
--delete1 --noexpungeaftereach --delete1emptyfolders
imapsync ... --delete1 --noexpungeaftereach --delete1emptyfolders
It won't delete the folder INBOX since INBOX is mandatory in imap.
=======================================================================
Q. How to delete some folders of an account with imapsync?
R. Use the same account on both sides with the options
--delete1 --noexpungeaftereach --delete1emptyfolders
like when deleting all folders BUT select only the folders
you want to delete by using options to select folders,
like --folder or --folderrec or --include or --exclude
Example, to delete the folder Blabla
imapsync ... --delete1 --noexpungeaftereach --delete1emptyfolders --folder Blabla
To delete the four folders Kalender Kontakte Aufgaben Notizen:
imapsync ... --include 'Kalender|Kontakte|Aufgaben|Notizen' \
--delete1 --noexpungeaftereach --delete1emptyfolders
I strongly recommend to use --dry first!
To get an overview of how to select folders, see
https://imapsync.lamiral.info/FAQ.d/FAQ.Folders_Selection.txt
=======================================================================
Q. How to revert a sync that went to the wrong account?
R. If all the wrong emails have some common characteristic that is not
shared by the good one, then you can search for them and delete them
using the same account as source and destination. For example, if all
the messages have the same email destination address as
"john_smith@example.com", then you can remove them with the following
command:
imapsync ... --search "TO john_smith@example.com" --delete1
Be very careful with deletions! Since this section deals with something
that was made wrong, you're in good place to make a new one!
There is many possibilities to select messages with different criterium.
See some of them by reading the document
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt
If the wrongly emails messages synced have no special common
characteristic then take an handkerchief and cry.
=======================================================================
Q. How to delete emails older than 90 days with imapsync?
R. Use the same account on both sides with option --delete1
--noexpungeaftereach --minage 90
Example:
imapsync \
--host1 test.lamiral.info --user1 foo --password1 secret \
--host2 test.lamiral.info --user2 foo --password2 secret \
--delete1 --noexpungeaftereach --minage 90
=======================================================================
Q. How to delete emails matching a certain criterium with imapsync?
R. Use the --search option to select the messages you want to delete
and then use the same account on both sides with the suplementary
options --delete1 --noexpungeaftereach
To select the messages read the document FAQ.Messages_Selection.txt
at https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt
Example:
imapsync \
--host1 test.lamiral.info --user1 foo --password1 secret \
--host2 test.lamiral.info --user2 foo --password2 secret \
--delete1 --noexpungeaftereach --search "SENTBEFORE 31-Dec-2010"
======================================================================
Q. Is there a way to delete the destination folders when the source
folders are no longer there?
R. Yes, use --delete2folders
--delete2folders : Delete folders in host2 that are not in host1.
For safety, first try it like this, it is safe:
--delete2folders --dry --justfolders --nofoldersizes
and see what folders will be deleted.
--delete2foldersonly reg : Delete only folders matching the regex reg.
Example: --delete2foldersonly "/^Junk$|^INBOX.Junk$/"
This option activates --delete2folders
--delete2foldersbutnot reg : Do not delete folders matching the regex rex.
Example: --delete2foldersbutnot "/Tasks$|Contacts$|Foo$/"
This option activates --delete2folders
=======================================================================
=======================================================================

675
FAQ.d/FAQ.Exchange.txt Normal file
View File

@ -0,0 +1,675 @@
$Id: FAQ.Exchange.txt,v 1.80 2022/04/05 13:48:52 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Exchange.txt
======================================================================
Imapsync tips for Exchange 2000 2003 2007 2010 2013 2016 2019
======================================================================
Questions answered in this FAQ are:
Q. Can I use imapsync to transfer from or to Exchange accounts?
Q. How to sync from XXX to Exchange 2010/2013/2016/2019
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
Q. Exchange throttles the sync and says:
"Request is throttled. Suggested Backoff Time: 299961 milliseconds".
What can I do with that?
Q. For Exchange I have double and triple checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
Q. Exchange fails with "User is authenticated but not connected".
Q. Exchange fails with "BAD Command received in Invalid state".
Q. Exchange fails with "NO Server Unavailable. 15" and imapsync says
"could not append ... NO Server Unavailable. 15"
Q. From XXX to Exchange 2013, read receipts are all resent again
after a sync. Even for old messages. How can I fix that?
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
Q. From XXX to Exchange 2010/2013 I get this error message
sometimes: "BAD Command Argument Error 11". What does it mean?
Q. From XXX to Exchange 2010 or 2013, the flag Flagged does
not seem to be well synced. What can I do?
Q. How to migrate from or to Exchange 2007/2010/2013 with an
admin/authuser account?
Q. How to migrate from or to Exchange 2016 with an admin/authuser account?
Q. How to migrate from or to Exchange 2003 with an admin/authuser account?
Q. Couldn't create folder [trash] "Mailbox already exists".
Q. Migrating to Exchange 201O, messages get date of the transfer,
this is bad for sorting and listing. What can I do?
Q. How to sync from any to Exchange 2007?
Q. Exchange 2007 folders ending with a space cause an imapsync crash.
"Could not select: socket closed while reading data from server"
Q. How to sync from Microsoft Exchange 2000 IMAP4rev1 server?
Now the questions again with their answers.
======================================================================
Q. Can I use imapsync to transfer from or to Exchange accounts?
R. Yes. But IMAP access to an Exchange account is not always
allowed by default so it has to be allowed in the server configuration
part.
Exchange: https://docs.microsoft.com/en-us/exchange/enable-imap4-in-exchange-2013-exchange-2013-help
======================================================================
Q. How to sync from XXX to Exchange 2010/2013/2016/2019
R0. IMAP is not enabled by default on Exchange, see how to enable it:
https://docs.microsoft.com/en-us/exchange/enable-imap4-in-exchange-2013-exchange-2013-help
Also read
http://clintboessen.blogspot.com/2018/03/binding-certificate-breaks-imap-or-pop.html
R1. Following is a command line resume that solves most encountered
issues when migrating to Exchange. To fully understand or change
the details you have to read the next Q/R sections.
On Windows, use:
imapsync.exe ... --exchange2
which is equivalent in imapsync release 1.970 to:
imapsync.exe ... ^
--maxsize 10_000_000 ^
--maxmessagespersecond 4 ^
--disarmreadreceipts ^
--regexflag "s/\\Flagged//g" ^
--regexmess "s,(.{10239}),$1\r\n,g"
On Unix, use:
imapsync ... --exchange2
which is equivalent in imapsync release 1.970 to:
imapsync ... \
--maxsize 10_000_000 \
--maxmessagespersecond 4 \
--maxlinelength 10239 \
--regexflag 's/\\Flagged//g' \
--disarmreadreceipts \
--regexmess 's,(.{10239}),$1\r\n,g'
On Linux, you can also try the "reformime" command
that can be used like:
imapsync ... --maxlinelengthcmd "reformime -r7"
To get reformime, install the "maildrop" package.
On Linux again, there is a good Python script in the tarball that can
fix several things that Exchange or O365 have issues with.
Use it like this:
./imapsync ... --pipemess W/tools/fix_email_for_exchange.py
It often does some miracles on messages.
======================================================================
Q. For Exchange I have double and triple checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
R1. Disable double-step authentication, also known as 2-factor,
2-step authentication on the Azure/Active Directory portal.
R2. Microsoft introduced something called "security defaults" which is
enabled by default for new tenants. One of the rules blocks IMAP
access as of imapsync. The funny thing is that you can't disable
single rules of this security package without buying additional
licenses. Switching the whole thing off allows the login.
Also, disable double-step authentication on the Azure/Active Directory
portal.
See here:
https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults
Thanks to Stephan Buhre for this R2 answer.
R3. I want to migrate to Exchange 2019. I haven't been able to sync
using admin credentials even if I've tried and made all the
configuration mentioned in the FAQ.Exchange.
The error was always "LOGIN failed".
My problem was that I have been used an admin account which haven’t
attached a mailbox in Exchange. This is my admin account which I use
only for configuration in Exchange, not for emails.
So, after I’ve created in Exchange a mailbox for this user, the sync
started to work. I can confirm that imapsync is working for Exchange
2019, with plain authentication, and full permissions of admin account
(this account must have mailbox attached) applied to regular user
mailbox account.
It doesn't work with the --authuser2 parameter. The syntax that
works for me is:
imapsync ... --host2 192.168.xxx.yyy --user2 domain.local\admin.account\regular.user --password2 "xxxxxxx"
or
imapsync ...--host2 192.168.xxx.yyy --user2 admin.account@mycompany.com\regular.user --password2 "xxxxxxx"
Thanks to Ioan Nechita for this R3 answer.
R4. Are there special characters in the password?
https://imapsync.lamiral.info/FAQ.d/FAQ.Authentication_failure.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Mac.txt
======================================================================
Q. Exchange fails with "User is authenticated but not connected".
R1. "The message User is authenticated but not connected is due to a
bug in the Exchange server's IMAP implementation. If the client
presents a valid user name but an invalid password, the server
accepts the login, but subsequent commands fail with the
aforementioned error message." Source:
https://unix.stackexchange.com/questions/164823/user-is-authenticated-but-not-connected-after-changing-my-exchange-password
Thanks to James Abbottsmith for this link and explanation at
https://github.com/imapsync/imapsync/issues/32#issuecomment-153561647
R2. Miguel Alameda reported understanding and solving this issue
like this, the context was admin/authuser:
"The admin user had not permission in the target mailbox."
R3. Jason from magiclamp.net adds this tip (2021_11_23).
"In GoDaddy Exchange admin center, under recipients, you can
double click on a mailbox, and in the "mailbox delegation" scrolling
down to the bottom it shows a input field named "Full Access".
Add the admin account there."
======================================================================
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
R1. With Exchange 2019
Change the recipient Receive/Send size:
Set-mailbox -Identity user@contoso.com -MaxReceiveSize 100mb -MaxSendSize 100mb
Thanks to Pasternak Denis for this Exchange 2019 tip.
R2. With Exchange 2016
Thanks to Torsten Bergemann from Germany:
In order to have exchange accept big e-mails via imap, several size
limitations have to be removed on several levels of the exchange
software: For the organization (within exchange admin panel),
for each connector (with exchange admin panel), for each Postfach
(using exchange management shell).
For details and instructions, check
https://docs.microsoft.com/en-us/Exchange/mail-flow/message-size-limits?view=exchserver-2019
https://docs.microsoft.com/en-us/Exchange/architecture/client-access/client-message-size-limits?view=exchserver-2019
Unfortunately, the second link does not give instructions to remove the limit from imap4 service.
To solve it for Exchange 2013/2016, add the line
<add key="MaxReceiveSize" value="1024000000" />
to the file
C:\Program Files\Microsoft\Exchange Server\V14 [or V15]\ClientAccess\PopImap\Microsoft.Exchange.Imap4.exe.config
Please note that this change may be lost each time you install an
cumulative update. After the change, restart the services exchange
transport role, exchange information store and exchange imap4 Backend.
Source in Deutch:
https://social.technet.microsoft.com/Forums/de-DE/2c07cf43-fe0e-4d22-b061-676a67c6d58d/exchange-2013-max-mailgre-per-imap?forum=exchange_serverde
This change made our Exchange installation accept bigger e-mails
in order to remove the imapsync Exchange error message
"NO Maximum size of appendable message has been exceeded"
======================================================================
Q. Exchange throttles the sync and says:
"Request is throttled. Suggested Backoff Time: 299961 milliseconds".
What can I do with that?
R0. Exchange has throttle mechanisms to limit any huge
usage. Sometimes imapsync transfers are too stressful for servers.
The message
"Request is throttled. Suggested Backoff Time: 299961 milliseconds"
comes from the imap Office365 server, imapsync just reports it
before being disconnected from it.
R1. Play with options --maxbytespersecond or --maxmessagespersecond
or --exitwhenover
imapsync ... --maxbytespersecond 100_000
imapsync ... --maxmessagespersecond 2
imapsync ... --exitwhenover 1_000_000_000
I don't know the upper value that avoid the default throttling from 0365
and I guess it changes over time.
R2. For Exchange, in case throttle appears anyway, fix them with:
https://docs.microsoft.com/en-us/exchange/change-user-throttling-settings-for-specific-users-exchange-2013-help
See also:
http://linux-france.tk/prj/imapsync_list/msg02072.html
(Subject: Re: [imapsync] From Exchange to Dovecot. Speed
Date: Mon, 03 Nov 2014 00:14:59 +0100)
Sometimes restarting the Exchange server is needed to take
into account the change in the configuration.
======================================================================
Q. Exchange fails with "BAD Command received in Invalid state".
R1. This message might happen when authenticating without ssl nor tls,
wich is rare now since imapsync tries to do ssl then tls by default.
Add --tls1 or else --ssl1 if this error message comes from host1.
Add --tls2 or else --ssl2 if this error message comes from host2.
R2. This message might also happen when authenticating with ssl and tls.
Don't use --tlsX and --sslX on the same side X (1 or 2), since it
asks for a double encryption, an encryption inside an encryption.
It won't work, you can't do a direct double encryption, few
servers allow that.
======================================================================
Q. Exchange fails with "NO Server Unavailable. 15" and imapsync says
"could not append ... NO Server Unavailable. 15"
R1. The problem could be a mailbox quota limitation.
In the Exchange log file, if you see many
"Microsoft.Exchange.Data.Storage.QuotaExceededException",
it looks like "Quota Exceeded" could be the issue.
The solution is to increase the quota for this mailbox.
Quoting
https://docs.microsoft.com/en-us/exchange/recipients/user-mailboxes/storage-quotas?view=exchserver-2019
In the Exchange admin center (EAC), navigate to Recipients > Mailboxes.
In the list of user mailboxes, click the mailbox that you want to change the storage quotas for,
and then click Edit icon, then change "Prohibit send and receive at (GB)", then save your changes.
Or use the Exchange Management Shell, aka PowerShell:
Set-Mailbox -Identity "Joe Healy" -IssueWarningQuota 499gb -ProhibitSendQuota 500gb -ProhibitSendReceiveQuota 500gb -UseDatabaseQuotaDefaults $false
R2. A user fixed it by enabling IMAP logging on Exchange which is
quite crazy as a solution. But why not, here it is:
https://www.mantisbt.org/forums/viewtopic.php?p=68835&sid=8b1ecd2d976e85218fbf33fcd38c209d#p68835
by TerranceM
Set-ImapSettings -Server <server-name> -ProtocolLogEnabled $true
======================================================================
Q. From XXX to Exchange 2013, read receipts are all resent again
after a sync. Even for old messages. How can I fix that?
R. imapsync can remove the header containing this read receipt request.
On Unix or Windows use:
imapsync ... --disarmreadreceipts
Since read receipts should be sent for unseen messages that will go
to a seen state after the migration, you could be strict and apply
the regex only to seen messages.
Selecting seen message can be done with:
imapsync ... --search1 "SEEN" --disarmreadreceipts
Of course a second run has to be done without the --disarmreadreceipts
for unseen messages:
imapsync ... --search1 "UNSEEN"
If fact --disarmreadreceipts is just an option equivalent to:
--regexmess 's{\A(.*?(?! ^$))^Disposition-Notification-To:(.*?)$}{$1X-Disposition-Notification-To:$2}igxms'
That regex changes the header Disposition-Notification-To. It prefixes
it with an X- so that it becomes inactive.
Disposition-Notification-To: blabla
becomes
X-Disposition-Notification-To: blabla
Thanks to David Karnowski for pointing and solving this issue.
======================================================================
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
R1. "SSL read or write error" happens sometimes, it isn't related to
imapsync directly but to the ssl underlying library when communicating
with Exchange in TLS/SSL encrypted mode.
Next runs should put the sync further, so rerun the syncs
until it is well completed.
R2. Another solution is to remove --tls or --ssl options for Exchange
and accept a clear text sync.
R3. See also the FAQ FAQ.SSL_errors.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.SSL_errors.txt
======================================================================
Q. From XXX to Exchange 2010/2013 I get this error message
sometimes: "BAD Command Argument Error 11". What does it mean?
R. This error message comes from Exchange IMAP server when it
encounters any problem. Most of the time it is one of the following:
* Some messages are bigger than the size limit. 10 MB by default
on Exchange. It can be upped by configuration for Exchange.
Sometimes a reboot is needed to take configuration changes
into account.
If you can't fix this limit on Exchange then use option
--maxsize 10000000 for 10 MB, change it if needed) to tell
imapsync to skip those messages.
imapsync ... --maxsize 10000000 # 10 MB for Exchange
* Quota reached. The whole account is full.
It can be upped by configuration.
* Some messages have some lines too long. Use option --maxlinelength
to skip messages whose max line length is over a number of bytes.
--maxlinelength 1000 is a RFC2822 must but most server support
higher values. Exchange supports 9900 characters line length:
imapsync ... --maxlinelength 9900
In case you prefer fixing messages with long lines the hard way,
instead of skipping them with --maxlinelength 9900, just use:
On Windows
imapsync ... --regexmess "s,(.{9900}),$1\r\n,g"
On Unix
imapsync ... --regexmess 's,(.{9900}),$1\r\n,g'
Have also in mind that Exchange closes the connection after 10 errors
encountered so you might also see "BYE Connection closed" errors from
Exchange, which means Exchange leaves the session and says goodbye,
come back later. Rerun a sync then.
On Linux, there is a good Python script in the tarball that can
fix several things that Exchange have issues with.
Use it like this:
./imapsync ... --pipemess W/tools/fix_email_for_exchange.py
It often does some miracles on messages.
======================================================================
Q. From XXX to Exchange 2010 or 2013, the flag Flagged does
not seem to be well synced. What can I do?
R. Use the following trick. Run imapsync twice, one with --regexflag
and one without, like this:
1) imapsync ... --regexflag "s/\\Flagged//g"
2) imapsync ...
You can add --debugflags if you want to see what imapsync gets and
does in details with flags.
The magic of this trick is on ignoring the \Flagged flag on the first
sync and setting it on the second sync, with STORE instead of APPEND.
This Exchange bug seems that Exchange gets and sets well the Flagged
flag with APPEND in IMAP but then it forgets it with other protocols;
With STORE it sets and gets the "\Flagged" flag everywhere.
Thanks to Dave Murray and Simon Savva for reporting and solving
this issue.
======================================================================
Q. How to migrate from or to Exchange 2007/2010/2013 with an
admin/authuser account?
R. The tricks comes from Michele Marcionelli and Benjamin Priestman:
This may works:
imapsync ... --user2 user2 --authuser2 admin2 --password2 adminpassword2 ...
or
imapsync ... --user2 "domain\admin2\user2" --password2 adminpassword2 ...
or
imapsync ... --user2 "admin2@domain\user2" --password2 adminpassword2 ...
where "domain" is set be the user's UPN in Active Directory
or the NETBIOS or DNS name of the domain.
The exact format might vary depending on local configuration and you
should experiment with the different formats.
PLAIN authentication is the only way to go with --authuser1 for now.
So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
A little note from Michael Scherer.
The previous workaround in the FAQ seems to be obsolete.
I can confirm that
imapsync ... --host2 exchange_server \
--authuser2 admin@domain --user2 user_alias
is working without any glitches, running Exchange 2010 SP2 here.
Explanation, Exchange knows a) about a username
and b) about a mailNickname which can be different to the username.
As it seems you need to use the mailNickname (e.g. alias) defined
for user2, not the username itself.
https://techcommunity.microsoft.com/t5/exchange-team-blog/understanding-login-strings-with-pop3-imap/ba-p/610683
https://web.archive.org/web/20191014050543/https://techcommunity.microsoft.com/t5/Exchange-Team-Blog/Understanding-login-strings-with-POP3-IMAP/ba-p/610683
the second link has the same content, I give it in case the first disappears,
thanks to archive.org!
A little note from John Becker.
After you pointed out that I had issues with authorization, I realized that
as an admin user I don't have all the rights on the mailboxes.
So I added myself to the user mailbox with full rights.
I also changed the login type on the Exchange CAS to type 1, plain text.
The command that I used for imapsync is:
imapsync ... ^
--host2 exchange-server ^
--user2 "windows-domain\admin-user\aaaa" ^
--password2 admin-password ...
======================================================================
Q. How to migrate from or to Exchange 2016 with an admin/authuser account?
R. This might work:
imapsync ... --user2 user2 --authuser2 admin2 --password2 adminpassword2 ...
Note from
https://github.com/imapsync/imapsync/issues/136#issuecomment-397020248
The key is to not use the built-in Administrator account which
Microsoft seems to have blocked access to in imap.
I'm doing a sync to Exchange 2016 accounts and this works:
imapsync ... --ssl2 --user2 account@domain.com ^
--authuser2 accountwfullaccess@domain.com ^
--password2 "accountwfullaccesspw"
The admin account you use needs to have full access to the account(s)
you want to sync which you can setup in the gui or via powershell
and again - don't use administrator@domain.com - as that will never
work because IMAP LOGIN is disabled for that account apparently
under all circumstances.
Note from Noxyron
https://github.com/imapsync/imapsync/issues/136#issuecomment-543761463
Hi guys. For a long time I could not start synchronization, I use admin rights
and the --authuser2 switch when I migrated from Mdaemon to Exchange2016.
I always got the error:
"Host2 failure: Error login on [my.exchange.com] with user [mydomainuser] auth [PLAIN]: 2 NO AUTHENTICATE failed."
In order for ImapSync to login to the user's mailbox using administrator rights,
you need to give for the administrator the permissions to the mailbox of this
user with the command:
Add-MailboxPermission -Identity DomainUser -User DomainAdmin -AccessRights FullAccess -InheritanceType All -AutoMapping $false
using Exchange Management Shell.
Without this, synchronization will not work.
Please, add this comment to the FAQ.Exchange.txt
(done!)
======================================================================
Q. How to migrate from or to Exchange 2003 with an admin/authuser account?
R. Thomas Edgar wrote the following
In case you can glean something from this snippet which allowed us
to migrate from Exchange 2003 t0 Cyrus Imap 2.4 (you would reverse
the flow to go the other way):
imapsync --dry --host1 ExchangeServer.mycompany.com \
--user1 Some.User@mycompany.com --authuser1 ExchangeAdminAccount \
--proxyauth1 --password1 '$pass_with_dollars$' \
--host2 cyrusimapbackend.mycompany.com \
--user2 SameOrDiffererentUser@mycompany.com \
--authuser2 CyrusAdminAccount --password2 CyrusAdminPassword
We also needed to:
- Ensure the Exchange Admin Acct had IMAP4 enabled in it's
profile (it initially didn't!)
- Ensure the Some.User Exchange Acct had IMAP4 enabled
in it's profile (it initially didn't!)
- Add the CyrusAdminAccount to admins line in /etc/imapd.conf
- Give CyrusAdminAccount lrswipkxtecda to the Cyrus Imap account
being migrated to (- or in your case, from)
In case you are not aware:
- It will prompt for a password if you don't supply it
- the domain part of the fully-qualified email could be
omitted in our case
- Use --debugimap when testing initial connectivity, if necessary
======================================================================
Q. Couldn't create folder [trash] "Mailbox already exists".
R. Some servers take care about character case in folder names,
some servers do not, like Exchange. Since non-respecting case
can merge two different folders into one then imapsync respects case.
For example, if a host1 server has a folder name called "trash"
and the host2 server already has a folder "Trash" or "TRASH"
then imapsync will try to create the folder "trash" on host2,
because trash and Trash are different strings. But if host2
does not respect character case it will consider folder "trash"
already exists and will say it, that's the error message
reported by imapsync: "Mailbox already exists", message coming
from the server.
The folder creation fails but messages are well transferred in
so take a look at this warning, understand why it happens
and it should be fine most of the time.
To avoid this warning use --regextrans2 to map the folder names
imapsync ... --regextrans2 "s/^trash$/Trash/"
If there are two folders Trash and trash on host1 then both
will be merged into only one Trash folder on host2.
In case option --delete2 is used the regextans2 above becomes
mandatory, otherwise imapsync will sync messages from the
first Trash and then delete them when syncing trash.
In order to avoid merging folders that are considered different
on host1 but the same on destination host2 because of case
sensitivities and insensitivities, use --nomixfolders
======================================================================
Q. Migrating to Exchange 201O, messages get date of the transfer,
this is bad for sorting and listing. What can I do?
R1. Be sure to have at least Exchange 2010 SP2 Rollup 5
https://tribalchicken.net/imap-sync-issues-with-exchange-2010/
R2. See also the Outlook side (thanks to Martin Hochreiter for this solution)
by changing Archive to sent/receive date
https://www.howto-outlook.com/faq/archivenotworking.htm
https://web.archive.org/web/20190914063435/http://www.howto-outlook.com/faq/archivenotworking.htm
It's often shorter to change one server than thousands clients
so R1 might be easier to do.
======================================================================
Q. How to sync from any to Exchange 2007?
Several problems:
- Big messages: increase the "send- and receive-connector"
in exchange2007 to 40 MB.
R. 2 solutions
R1. With imapsync (only partial success)
--skipheader 'Received' \
--regexmess 's{\A(.*?(?!^$))^Date:(.*?)$}{$1Date:$2\nReceived: From; $2}gxms'
Any user having time to spend to debug Exchange 2007 with imapsync
is welcome.
R2. Other solution
Two users succeeded by using "MS Transporter Suite" (which is closed
expensive non-free software).
======================================================================
Q. Exchange 2007 folders ending with a space cause an imapsync crash.
"Could not select: socket closed while reading data from server"
R. Using another imap tool like Thunderbird or Outlook, change the
folder name to exclude the trailing space.
See https://github.com/imapsync/imapsync/issues/154
======================================================================
Q. How to sync from Microsoft Exchange 2000 IMAP4rev1 server?
R. imapsync ... --prefix1 "INBOX."
======================================================================
======================================================================

68
FAQ.d/FAQ.FirstClass.txt Normal file
View File

@ -0,0 +1,68 @@
#!/bin/cat
$Id: FAQ.FirstClass.txt,v 1.1 2021/01/26 14:03:58 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.FirstClass.txt
=======================================================================
Imapsync tips for the FirstClass IMAP server.
=======================================================================
======================================================================
Q. From FirstClass to XXX
http://www.firstclass.com/
R. Migrating from FirstClass is not easy because FirstClass, strangely,
does not show all messages via IMAP. To make it show all messages,
a trick, painful to follow by hand, is moving emails
out and back in, for each folder. May be it can be done by a script.
FirstClass releases prior to release 12 do not shows the "Sent"
folder in IMAP but FirstClass release 12 shows it.
I advice you to upgrade to FirstClass release 12 before leaving it
with imapsync or another imap tool.
Here is a command line used to migrate from FirtClass release 12:
imapsync ... \
--tmpdir /var/tmp --usecache \
--useheader Message-ID \
--idatefromheader \
--addheader \
--regextrans2 "s,(/|^) +,\$1,g" --regextrans2 "s, +(/|$),\$1,g" \
--regextrans2 "s/[\^]/_/g" \
--regextrans2 "s/['\"\\\\]/_/g" \
--regextrans2 "s,&AC8-,-,g" \
--regextrans2 "s,&APg-,oe,g"
On Windows:
imapsync.exe ... ^
--automap ^
--usecache ^
--useheader Message-ID ^
--idatefromheader ^
--addheader ^
--regextrans2 "s,(/|^) +,$1,g" ^
--regextrans2 "s, +(/|$),$1,g" ^
--regextrans2 "s/[\^]/_/g" ^
--regextrans2 "s/['\\]/_/g" ^
--regextrans2 "s,^&AC8-,-,g" ^
--regextrans2 "s,^&APg-,oe,g"
Special thanks to Kristian Wind and Joey Alexander for helping me
writing this FAQ item.
See also this worth reading discussion in a Zimbra forum:
https://web.archive.org/web/20120627043731/http://www.zimbra.com/forums/migration/20349-help-needed-migrating-firstclass.html
Another advice with FirstClass as host1, do not use --delete1emptyfolders since
it will destroy all subfolders of any empty folder. This is a FirstClass bug:
when asked to list a non-empty folder it says it has no children while it does
have ones.
See the details and explanations at
https://github.com/imapsync/imapsync/issues/207

328
FAQ.d/FAQ.Flags.txt Normal file
View File

@ -0,0 +1,328 @@
#!/bin/cat
$Id: FAQ.Flags.txt,v 1.28 2022/06/22 17:07:35 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Flags.txt
=======================================================================
Imapsync tips about flags.
=======================================================================
Questions answered here are:
Q. How to debug flag issues?
Q. Is there a way to only sync messages with a specific flag set,
for example, the \Seen flag?
Q. How to convert flags?
Q. Exchange sends an email to any sender whose email is deleted
without reading. It's called "unread notifications".
How to set the \Seen flag on host1 (source system) before syncing?
Q. Every single mail synced to exchange online owns the
category $MDNSent after migration. How can I avoid this?
Q. Does imapsync retain the \Answered and $Forwarded flags?
Q. How to fix this error: BAD Invalid system flag \FORWARDED
Q. How to convert flags with $ to \ character?
Q. imapsync fails with the following error:
flags from : [\Seen NonJunk]
Error trying to append string: 58 NO APPEND Invalid flag list
Q. Flags are not well synchronized. Is it a bug?
Q. Flags are resynced at each run for already synced/copied messages,
how can I avoid this feature?
Q. Is it possible to sync labels and stars made by Thunderbird to
Exchange categories? Or a way in Outlook to show labels created by
Thunderbird?
Now the questions again with their answers.
=======================================================================
Q. How to debug flag issues?
R. Use --debugflags
imapsync ... --debugflags
=======================================================================
Q. Is there a way to only sync messages with a specific flag set,
for example, the \Seen flag?
R. use --search
imapsync ... --search SEEN
or
imapsync ... --search UNSEEN
or ...
The complete list of search things related to flags are listed below
http://www.faqs.org/rfcs/rfc3501.html
6.4.4. SEARCH Command
...
ANSWERED
Messages with the \Answered flag set.
DELETED
Messages with the \Deleted flag set.
DRAFT
Messages with the \Draft flag set.
FLAGGED
Messages with the \Flagged flag set.
KEYWORD <flag>
Messages with the specified keyword flag set.
NEW
Messages that have the \Recent flag set but not the \Seen flag.
This is functionally equivalent to "(RECENT UNSEEN)".
NOT <search-key>
Messages that do not match the specified search key.
OLD
Messages that do not have the \Recent flag set. This is
functionally equivalent to "NOT RECENT" (as opposed to "NOT
NEW").
OR <search-key1> <search-key2>
Messages that match either search key.
RECENT
Messages that have the \Recent flag set.
SEEN
Messages that have the \Seen flag set.
UNANSWERED
Messages that do not have the \Answered flag set.
UNDELETED
Messages that do not have the \Deleted flag set.
UNDRAFT
Messages that do not have the \Draft flag set.
UNFLAGGED
Messages that do not have the \Flagged flag set.
UNKEYWORD <flag>
Messages that do not have the specified keyword flag set.
UNSEEN
Messages that do not have the \Seen flag set.
=======================================================================
Q. How to convert flags?
R. use --regexflag
For example to convert flag IMPORTANT to flag CANWAIT
imapsync ... --regexflag "s/IMPORTANT/CANWAIT/g" --debugflags
option --debugflags is usefull to see in details what imapsync
does with flags.
=======================================================================
Q. Exchange sends an email to any sender whose email is deleted
without reading. It's called "unread notifications".
How to set the \Seen flag on host1 (source system) before syncing?
R. You can add \Seen (if missing) to the host1 account by applying a
first sync to the same account, same at source and destination,
and the help of option --regexflag. It can also be done on the fly
from account1 to account2 in case account1 has to stay as it is.
Add the \Seen flag to all messages like this:
On Winwows:
imapsync.exe ... --regexflag "s,\\Seen,," --regexflag "s,,\\Seen ,"
On Unix:
imapsync ... --regexflag 's,\\Seen,,' --regexflag 's,,\\Seen ,'
R2. You can also filter with --search1 UNSEEN and use only one
of the regexes:
On Unix:
imapsync ... --search1 UNSEEN --regexflag 's,,\\Seen ,'
On Winwows:
imapsync.exe ... --search1 UNSEEN --regexflag "s,,\\Seen ,"
R3. Fix it on the server Exchange:
Google translate:
https://translate.google.com/translate?sl=auto&tl=en&u=https%3A%2F%2Fwww.ci-solution.com%2Fblog%2Fartikel%2Fungelesen-geloescht-verhindern.html
German original:
https://www.ci-solution.com/blog/artikel/ungelesen-geloescht-verhindern.html
(Link from Oliver B.)
======================================================================
Q. Every single mail synced to exchange online owns the
category $MDNSent after migration. How can I avoid this?
R. To remove the flag $MDNSent from all messages, use:
imapsync ... --regexflag "s/\$MDNSent//g"
See also the document
https://imapsync.lamiral.info/FAQ.d/FAQ.Flags.txt
=======================================================================
Q. Does imapsync retain the \Answered and $Forwarded flags?
R. It depends on the destination server.
a) If the destination server honours the "PERMANENTFLAGS \*"
directive (meaning it accepts any flag) or no PERMANENTFLAGS at all
then imapsync synchronizes all flags except the \Recent flag
(RFC 3501 says about \Recent flag "This flag can not be
altered by the client.").
b) If the destination server honours the "PERMANENTFLAGS without the
special "\*" then imapsync synchronizes only the flags listed
in PERMANENTFLAGS.
Some imap servers have problems with flags not beginning with
the backslash character \
(see next question to find a solution to this issue)
=======================================================================
Q. How to fix this error: BAD Invalid system flag \FORWARDED
R. Filter flag \FORWARDED with --regexflag like this:
On Windows:
imapsync.exe ... --regexflag "s/\\FORWARDED//gi"
On Unix:
imapsync ... --regexflag 's/\\FORWARDED//gi'
or (on Unix)
imapsync ... --regexflag "s/\\\\FORWARDED//gi"
Other related flags to remove are \Indexed and \ATTACHED
Windows: imapsync.exe ... --regexflag "s/\\Indexed//gi"
Unix: imapsync ... --regexflag 's/\\Indexed//gi'
Windows: imapsync.exe ... --regexflag "s/\\ATTACHED//gi"
Unix: imapsync ... --regexflag 's/\\ATTACHED//gi'
Removing usually problematic flags in one line:
Windows: imapsync.exe ... --regexflag "s/\\FORWARDED|\\Indexed|\\ATTACHED|\\RECEIPTCHECKED//gi"
Unix: imapsync ... --regexflag 's/\\FORWARDED|\\Indexed|\\ATTACHED|\\RECEIPTCHECKED//gi'
Removing all problematic flags I've seen so far in one line:
Windows: imapsync.exe ... --regexflag "s/\\RECEIPTCHECKED|\\Indexed|\\X-EON-HAS-ATTACHMENT|\\UNSEEN|\\ATTACHED|\\X-HAS-ATTACH|\\FORWARDED|\\FORWARD|\\X-FORWARDED|\\\$FORWARDED|\\PRIORITY|\\READRCPT//g"
Unix: imapsync ... --regexflag 's/\\RECEIPTCHECKED|\\Indexed|\\X-EON-HAS-ATTACHMENT|\\UNSEEN|\\ATTACHED|\\X-HAS-ATTACH|\\FORWARDED|\\FORWARD|\\X-FORWARDED|\\\$FORWARDED|\\PRIORITY|\\READRCPT//g'
Removing all flags but the standard ones from the IMAP RFC:
Windows: imapsync.exe ... --regexflag "s/.*?(?:(\\(?:Answered|Flagged|Deleted|Seen|Draft)\s?)|$)/defined($1)?$1:q()/eg"
Unix: imapsync ... --regexflag 's/.*?(?:(\\(?:Answered|Flagged|Deleted|Seen|Draft)\s?)|$)/defined($1)?$1:q()/eg'
=======================================================================
Q. How to convert flags with $ to \ character?
R. $ and \ are special characters we have to "escape" them.
For example to convert flag $label1 to \label1
imapsync ... --regexflag "s/\$label1/\\label1/g" --debugflags
======================================================================
Q. imapsync fails with the following error:
flags from : [\Seen NonJunk]
Error trying to append string: 58 NO APPEND Invalid flag list
R. For some servers, flags have to begin with a \ character.
The flag "NonJunk" may be a invalid flag for your server
so use for example:
imapsync ... --regexflag "s/NonJunk//g"
Remark (thanks to Arnt Gulbrandsen):
IMAP system flags have to begin with \ character.
Any other flag must begin with another character.
System flags are just flags defined by an RFC instead of by users.
Conclusion, some imap server coders don't read the RFCs (so do I).
Recent imapsync deals with this issue by filter with PERMANENTFLAGS
automatically.
=======================================================================
Q. Flags are not well synchronized. Is it a bug?
R. It happens with some servers on the first sync.
Also, it was a bug from revision 1.200 to revision 1.207
Two solutions:
* Run imapsync a second time. imapsync synchronizes flags on each run.
* Use option --syncflagsaftercopy. With this option imapsync will
also sync flags after each message transfer. Flags are already
synced during the transfer with the imap APPEND command but
option --syncflagsaftercopy does it again using the imap STORE
command.
=======================================================================
Q. Flags are resynced at each run for already synced/copied messages,
how can I avoid this feature?
R. Use the option --noresyncflags
=======================================================================
Q. Is it possible to sync labels and stars made by Thunderbird to
Exchange categories? Or a way in Outlook to show labels created by
Thunderbird?
R. Imapsync syncs all flags possible by default, so if it doesn't do
that it might means there are not on the server but stay only on the
client or that the host2 server claims to accept only a given set
of flags.
Check those claims by accessing the same mailbox on the same server
from another thunderbird on another host, you should not retrieve
those labels. If you do find them it then might mean that host2
server don't want them, try --nofilterflags
imapsync ... --nofilterflags
You can try --nofilterflags straightaway without the
"other thunderbird" proposal.
There is also the possibility to map flags across servers
with the option --regexflag. See some --regexflag examples
above.
=======================================================================
=======================================================================

View File

@ -0,0 +1,490 @@
#!/bin/cat
$Id: FAQ.Folders_Mapping.txt,v 1.23 2021/04/28 23:42:04 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Folders_Mapping.txt
=======================================================================
Imapsync tips about changing folders names.
=======================================================================
Questions answered in this FAQ are:
Q. What is the imapsync default behaviour with folders?
Q. What encoding imapsync uses to deal with folder names?
Q. How can I encode folder names for imapsync without getting a headache?
Q. What are the easy ways to change folder names?
Q. What should I know before playing with the --regextrans2 option?
Q. How to remove the string "INBOX." in the name of destination folders?
Q. How to change only INBOX name to Inbox_Migrated?
Q. How to sync all folders to a single INBOX folder?
Q. How to sync all folders to a single WHOLE folder?
Q. How to sync a complete account in a subfolder called FOO?
Q. How to substitute all characters dot "." by underscores "_" ?
Q. How to substitute all doublequotes " by underscores _ ?
Q. How to substitute all characters * % . and # by underscores _ ?
Q. How to change folder names like this:
[mail/Sent Items] -> [Sent]
[mail/Test] -> [INBOX/Test]
[mail/Test2] -> [INBOX/Test2]
Q. Is it possible to synchronize all messages from one server to
another without recreating the folder structure and the target server.
Q. I would like to move emails from InBox to a sub-folder called,
say "2010-INBOX" based on the date. Like all emails received in the
year 2010 should be moved to the folder called "2010-INBOX".
Q. How to sync a complete account in a subfolder called FOO,
the hard way?
Q. How to transform all folder names to lowercase?
Q. How to transform all folder names to uppercase?
Q. How to transform all folder names to lowercase, except the first
character in uppercase?
Now the questions again with their answers.
=======================================================================
Q. What is the imapsync default behaviour with folders?
R. Imapsync syncs all folders, one by one, in alphanumeric order,
reproducing their name identical from the source account at host1 to
the destination account at host2, except for the prefix and the
separator which are automatically adapted for host2.
=======================================================================
Q. What encoding imapsync uses to deal with folder names?
R. The IMAP protocol has a specific way to code folders names,
especially when these names use non-ascii 7bit characters.
This encoding is called utf7imap.
Imapsync uses the same encoding as IMAP, it uses utf7imap.
=======================================================================
Q. How can I encode folder names for imapsync without getting a headache?
R. In order to well specify folders names on the command line, imapsync
prints the complete folder list of both sides at the beginning
of each run.
The left column is the encoding you have to use, without the first enclosing
square brackets [], the right column is the human utf8 view.
=======================================================================
Q. What are the easy ways to change folder names?
R. This document gives most examples with the powerful but complex
option --regextrans2. Before using --regextrans2 you should
consider using --automap and --f1f2 because they are simpler
to understand and use.
--automap : guesses folders mapping, for folders like
"Sent", "Junk", "Drafts", "All", "Archive", "Flagged".
--f1f2 str1=str2 : Force folder str1 to be synced to str2,
--f1f2 overrides --automap and --regextrans2.
--subfolder2 str : Syncs the whole host1 folders hierarchy under the
host2 folder named str.
(It does it internally by adding two
--regextrans2 options before all others.
Add --debug to see what's really going on.)
=======================================================================
Q. What should I know before playing with the --regextrans2 option?
The things to know and understand before playing with --regextrans2 are:
*) --regextrans2 is used to transform folders names.
*) --regextrans2 applies after the default
inversion prefix1 <-> prefix2 and sep1 <-> sep2.
So, when elaborating the regex you should focus on
the right part of the default mapping printed by
imapsync, the part showing the host2 folder name.
The section to look at is either the folder size section
or within the folder loop located "Here" between the lines
++++ Looping on each folder
Here
++++ End looping on each folder
*) Several --regextrans2 are possible, they will be applied in the order
of the command line, each one on the result of the previous one.
*) --regextrans2 uses Perl regex mechanism so it may be hard to master
this part. It is powerful but not very simple.
*) Windows vs Unix quotes.
On windows don't use single quotes ' around the regex string,
use double quotes instead, like --regextrans2 "myregex"
On Linux/Unix use single quotes ' around the regex string, it is
easier to get what we want with single quotes since the shell
won't change the inner string. Use --regextrans2 'myregex'
*) Good method to elaborate any --regextrans2 string
First, elaborate the --regextrans2 string with --dry --justfolders options.
imapsync ... --dry --justfolders
With --dry imapsync shows the transformations it will do without
really doing them, --dry is the "do nothing" mode.
With --justfolders imapsync will work only with folders,
messages won't be taken into account, so it will be fast
and focused on the folders names.
When the output shows what you expect imapsync to do with folders
names, you can remove the --dry option. Keep the --justfolders
option in order to see if the destination server host2 accepts
to create the folders.
When everything is ok with folders you might remove --justfolders,
imapsync will also transfer messages.
Showing folders sizes is good then transferring messages, it allows
ETA calculation and it's a supplementary check on folders.
=======================================================================
Q. How to remove the string "INBOX." in the name of destination folders?
R. To remove "INBOX." in the name of destination folders:
On Linux:
imapsync ... --regextrans2 's/^INBOX\.(.+)/$1/'
On Windows:
imapsync ... --regextrans2 "s/^INBOX\.(.+)/$1/"
=======================================================================
Q. How to change only INBOX name to Inbox_Migrated?
R.
imapsync ... --regextrans2 's{^INBOX$}{Inbox_Migrated}'
=======================================================================
Q. How to sync all folders to a single INBOX folder?
R.
imapsync ... --regextrans2 "s/.*/INBOX/"
=======================================================================
Q. How to sync all folders to a single WHOLE folder?
R.
imapsync ... --regextrans2 "s,.*,WHOLE,"
=======================================================================
Q. How to sync a complete account in a subfolder called FOO?
R. With imapsync release 1.641 and former, simply use:
imapsync ... --subfolder2 FOO
Examples to this subfolder problem for previous releases
can be seen below, in another Q/R section.
=======================================================================
Q. How to substitute all characters dot "." by underscores "_" ?
R.
--regextrans2 "s,\.,_,g"
=======================================================================
Q. How to substitute all doublequotes " by underscores _ ?
On Linux/Unix:
--regextrans2 's,\",_,g'
On Windows:
--regextrans2 s,\^",_,g
=======================================================================
Q. How to substitute all characters * % . and # by underscores _ ?
On Linux/Unix:
--regextrans2 'tr,*%.#,_,'
On Windows:
--regextrans2 "tr,*%.#,_,"
You can increase the *%.# list by any unwanted character,
plus, the order doesn't count. See below for &
It is a bad idea to substitute & characters since &
is a character to encode non-ascii characters in IMAP folder names.
=======================================================================
Q. How to change folder names like this:
[mail/Sent Items] -> [Sent]
[mail/Test] -> [INBOX/Test]
[mail/Test2] -> [INBOX/Test2]
On Linux/Unix:
--regextrans2 's,^mail/Sent Items$,Sent,' \
--regextrans2 's,^mail/,INBOX/,'
a) I want folders 'Sent', 'Sent Messages', 'Gesendet' from host1
to be merged to folder 'Sent' on host2.
--regextrans2 "s{Sent|Sent Messages|Gesendet}{Sent}"
b) I do not want this rule a) to apply to any subfolders,
e.g. 'folder/Sent' or 'folder/Sent Messages' etc. but only
basefolders.
So you have to add an anchor character ^ meaning "beginning of the string",
and an anchor character $ meaning "end of the string" like this:
--regextrans2 's{^Sent$|^Sent Messages$|^Gesendet$}{Sent}'
c) I want just a) to be combined with a subfolder, for example:
INBOX -> Subfolder/INBOX
Sent -> Subfolder/Sent
Sent Messages -> Subfolder/Sent
Gesendet -> Subfolder/Sent
folder/Sent -> Subfolder/folder/Sent
folder/Sent Messages -> Subfolder/folder/Sent
folder/Gesendet -> Subfolder/folder/Sent
To achieve this, use:
--subfolder2 Subfolder --regextrans2 's{Sent|Sent Messages|Gesendet}{Sent}'
d) I want a) and b) to be combined with a subfolder, for example:
INBOX -> Subfolder/INBOX
Sent -> Subfolder/Sent
Sent Messages -> Subfolder/Sent
Gesendet -> Subfolder/Sent
folder/Sent -> Subfolder/folder/Sent
folder/Sent Messages -> Subfolder/folder/Send Messages
folder/Gesendet -> Subfolder/folder/Gesendet
...
This one is trickier to achieve:
--subfolder2 Subfolder --regextrans2 's{^Subfolder/Sent$|^Subfolder/Sent Messages$|^Subfolder/Gesendet$}{Subfolder/Sent}'
======================================================================
Q. Is it possible to synchronize all messages from one server to
another without recreating the folder structure and the target server.
R. Yes. This question has already been answered but a little of
redundancy can't hurt, can he?
For example, to synchronize all messages in all folders on host1
to folder INBOX only on host2:
1) First try (safe mode):
--regextrans2 "s/.*/INBOX/" --dry --justfolders
2) See if the output says everything you want imapsync to do,
--dry option is safe and does nothing real.
3) Remove --dry
Check the imap folder tree on the target side, you should
only have one: the classical INBOX.
4) Remove --justfolders
=======================================================================
Q. I would like to move emails from InBox to a sub-folder called,
say "2010-INBOX" based on the date. Like all emails received in the
year 2010 should be moved to the folder called "2010-INBOX".
R. 2 ways :
a) With imapsync:
-----------------
imapsync ... \
--search "SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010"
--f1f2 "INBOX=2010-INBOX" \
--folder INBOX
See also the FAQ FAQ.Messages_Selection.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt
to get more examples about how to select messages.
b) Manually:
------------
1) You create a folder INBOX.2010-INBOX
2) Mostly every email software allow sorting by date. In INBOX, you
select from 1 January to 31 December 2010 messages with the shift key.
(in mutt, use ~d)
3) Cut/paste in INBOX.2010-INBOX
=======================================================================
Q. How to sync a complete account in a subfolder called FOO,
the hard way?
R0. See the easy above using --subfolder2
R1. Several ways to do it.
a) Separator is dot character "." and "INBOX" prefixes every folder
On Linux/Unix:
--regextrans2 's,^INBOX(.*),INBOX.FOO$1,'
On Windows:
--regextrans2 "s,^INBOX(.*),INBOX.FOO$1,"
or:
b) Separator is the slash character "/" and there is no prefix
On Linux/Unix:
--regextrans2 's,(.*),FOO/$1,'
or with doublequotes
--regextrans2 "s,(.*),FOO/\$1,"
On Windows:
--regextrans2 "s,(.*),FOO/$1,"
=======================================================================
Q. How to transform all folder names to lowercase?
R.
First, some documentation to understand how the regexes work.
perldoc perlre says:
\l lowercase next char (think vi)
\u uppercase next char (think vi)
\L lowercase till \E (think vi)
\U uppercase till \E (think vi)
\E end case modification (think vi)
\Q quote (disable) pattern metacharacters till \E
On Linux:
If the separator on host2 is the character slash "/", use:
--regextrans2 's,([^/]+),\L$1\E,g'
If the separator on host2 is the character dot ".", use:
--regextrans2 's,([^.]+),\L$1\E,g'
On Windows:
If the separator on host2 is the character slash "/", use:
--regextrans2 "s,([^/]+),\L$1\E,g"
If the separator on host2 is the character dot ".", use:
--regextrans2 "s,([^.]+),\L$1\E,g"
=======================================================================
Q. How to transform all folder names to uppercase?
R.
On Linux:
If the separator on host2 is the character slash "/", use:
--regextrans2 's,([^/]+),\U$1\E,g'
If the separator on host2 is the character dot ".", use:
--regextrans2 's,([^.]+),\U$1\E,g'
On Windows:
If the separator on host2 is the character slash "/", use:
--regextrans2 "s,([^/]+),\U$1\E,g"
If the separator on host2 is the character dot ".", use:
--regextrans2 "s,([^.]+),\U$1\E,g"
=======================================================================
Q. How to transform all folder names to lowercase, except the first
character in uppercase?
R.
On Linux:
If the separator on host2 is the character slash "/", use:
--regextrans2 's,([^/]+),\u\L$1\E,g'
If the separator on host2 is the character dot ".", use:
--regextrans2 's,([^.]+),\u\L$1\E,g'
On Windows:
If the separator on host2 is the character slash "/", use:
--regextrans2 "s,([^/]+),\u\L$1\E,g"
If the separator on host2 is the character dot ".", use:
--regextrans2 "s,([^.]+),\u\L$1\E,g"
=======================================================================
=======================================================================

View File

@ -0,0 +1,184 @@
#!/bin/cat
$Id: FAQ.Folders_Selection.txt,v 1.12 2022/03/22 11:12:32 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Folders_Selection.txt
=======================================================================
Imapsync tips to select folders.
=======================================================================
Questions answered in this FAQ are:
Q. What is the default imapsync behaviour about folders?
Q. What are the options to select folders?
Q. Is it ok to use --subfolder1 and --subfolder2 to just select folders?
Q. How can I sync only one folder?
Q. What are --subscribe and --subscribed for, and how can they be used?
Q. How to exclude a folder hierarchy like "public"?
Q. How to exclude only the INBOX folder?
Q. How to exclude folders matching SPAM no matter the case,
aka, how to be case insensitive?
Q. How to make option --folder "MyFolder" be recursive?
Q. How to change folders names or move them elsewhere?
Now the questions again with their answers.
=======================================================================
Q. What is the default imapsync behaviour about folders?
By default, Imapsync syncs all folders, one by one, in alphanumeric order.
The IMAP protocol has a specific way to code folders names,
especially when these names use non-ascii 7bit characters.
This encoding is called utf7imap.
Imapsync uses the same encoding as IMAP, utf7imap.
In order to well specify folders names on the command line, imapsync
prints the complete folder list of both sides at the beginning of each run.
The left column is the encoding you have to use, without the first enclosing
square brackets [], the right column is the human utf8 view.
=======================================================================
Q. What are the options to select folders?
R. From https://imapsync.lamiral.info/README
OPTIONS/folders
--folder str : Sync this folder.
--folder str : and this one, etc.
--folderrec str : Sync this folder recursively.
--folderrec str : and this one, etc.
--folderfirst str : Sync this folder first. Ex. --folderfirst "INBOX"
--folderfirst str : then this one, etc.
--folderlast str : Sync this folder last. --folderlast "[Gmail]/All Mail"
--folderlast str : then this one, etc.
--nomixfolders : Do not merge folders when host1 is case-sensitive
while host2 is not (like Exchange). Only the first
similar folder is synced. Example: with folders
"Sent", "SENT" and "sent" on host1, only "Sent"
will be synced to host2.
--skipemptyfolders : Empty host1 folders are not created on host2.
--include reg : Sync folders matching this regular expression
--include reg : or this one, etc.
If both --include --exclude options are used, then
include is done before.
--exclude reg : Skips folders matching this regular expression
Several folders to avoid:
--exclude 'fold1|fold2|f3' skips fold1, fold2 and f3.
--exclude reg : or this one, etc.
=======================================================================
Q. Is it ok to use --subfolder1 and --subfolder2 to just select folders?
R. No.
If you do:
imapsync ... --subfolder1 Foo --subfolder2 Foo
then it ends up with all messages from Foo going to Foo/INBOX
=======================================================================
Q. How can I sync only one folder?
R. Use --folder option.
imapsync ... --folder MyFolder
If you have more specific folders to sync just add several --folder
imapsync ... --folder MyFolder --folder ThisFolder --folder ThatFolder
=======================================================================
Q. What are --subscribe and --subscribed for, and how can they be used?
R. In the IMAP protocol each user can subscribe to one or more folders.
Then one can configure his email software to just see his subscribed
folders. That's an IMAP feature.
Imapsync can use this imap feature to select subscribed folders
and also subscribe to folders on host2. Here are the options:
--subscribed : Transfers subscribed folders.
--subscribe : Subscribe to the folders transferred on the
host2 that are subscribed on host1. On by default.
--subscribeall : Subscribe to the folders transferred on the
host2 even if they are not subscribed on host1.
=======================================================================
Q. How to exclude a folder hierarchy like "public"?
R. Use:
--exclude "public"
=======================================================================
Q. How to exclude only the INBOX folder?
R. Use:
imapsync ... --exclude "^INBOX$"
A good way to see what will be done is to first use:
imapsync ... --exclude "^INBOX$" --justfolders --nofoldersizes --dry
=======================================================================
Q. How to exclude folders matching SPAM no matter the case,
aka, how to be case insensitive?
R. Use:
imapsync ... --exclude "(?i)spam"
A good way to see what will be done is to first use:
imapsync ... --exclude "(?i)spam" --justfolders --nofoldersizes --dry
=======================================================================
Q. How to make option --folder "MyFolder" be recursive?
Two solutions:
R1. Use
--folderrec "MyFolder"
R2. Use --include "^MyFolder"
Then the folder "MyFolder" and all its subfolders will be handled
and only them.
=======================================================================
Q. How to change folders names or move them elsewhere?
R. Read the document FAQ.Folders_Mapping.txt also available at
https://imapsync.lamiral.info/FAQ.d/FAQ.Folders_Mapping.txt
=======================================================================
=======================================================================

View File

@ -0,0 +1,86 @@
#!/bin/cat
$Id: FAQ.Folders_Sizes.txt,v 1.7 2022/04/23 13:45:15 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Folders_Sizes.txt
=======================================================================
Imapsync tips to know folders sizes.
=======================================================================
Questions answered in this FAQ are:
Q. How can I know the size of all folders before launching a sync?
Q. After a sync, the folders sizes and the number of messages don't
match between the source mailbox on host1 and the destination
mailbox on host2.
Why? What can I do about it?
Now the questions again with their answers.
=======================================================================
Q. How can I know the size of all folders before launching a sync?
R. Use --justfoldersizes option.
With the --justfoldersizes option, imapsync evaluates and displays the
folders sizes on both sides and then exits without doing any transfer.
It also presents the total size of each account, the number of messages,
the biggest message size, the number of folders.
Example:
imapsync ... --justfoldersizes
=======================================================================
Q. After a sync, the folders sizes and the number of messages don't
match between the source mailbox on host1 and the destination
mailbox on host2.
Why? What can I do about it?
R. There are several reasons to explain the different counts between
source and destination after a sync.
1) The sync ended prematurely, before syncing all wanted folders.
Solution: run new syncs until completion.
2) There are messages on host2 that are not on host1, so host2 is
bigger than host1. Imapsync doesn't delete any message by default.
Solution: use --delete2 to have a strict sync by folder, ie,
delete messages on host2 that are not on host1.
3) There are folders on host2 that are not on host1, so host2 is
bigger. Imapsync doesn't delete any folder by default.
Solution: use --delete2folders to have a strict sync, ie,
delete folders on host2 that are not on host1.
4) There are duplicate messages on host1. Imapsync doesn't sync
duplicates by default. But it counts them and presents the number
found at the end on the line "Messages found duplicate on host1".
To sync duplicates, use the option --syncduplicates
5) There are cross-duplicate messages on host1, same messages in different
folders, and you use --gmail1 or --skipcrossduplicates, which avoid
copying cross-duplicate messages in more than one folder on host2.
Use --noskipcrossduplicates if you want imapsync to sync cross-duplicates.
6) There are duplicate messages on host2 so host2 is
bigger than host1. Solution: use --delete2duplicates to delete
the host2 duplicates.
7) Cosmic rays...
The important part to take into account for a good and well done sync
is the last lines of imapsync. A good sync is when you encounter the
three final lines:
"The sync looks good, all xxxx identified messages in host1 are on host2."
"There is no unidentified message on host1."
"Detected 0 errors"
=======================================================================
=======================================================================

80
FAQ.d/FAQ.Fun_Things.txt Normal file
View File

@ -0,0 +1,80 @@
#!/bin/cat
$Id: FAQ.Fun_Things.txt,v 1.2 2021/10/15 13:37:05 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Fun_Things.txt
=======================================================================
Imapsync fun things to do with
=======================================================================
Questions answered in this FAQ are:
Q. How can I have a daily digest of my mailbox?
Now the questions again with their answers.
=======================================================================
Q. How can I have a daily digest of my mailbox?
R. Use the same account as source and destination (not mandatory)
imapsync \
--host1 imap.example.com --user1 foo --password1 secret \
--host2 imap.example.com --user2 foo --password2 secret \
--folder INBOX --maxage 1 --dry --nodry1 --debugcontent \
--truncmess 1000 --f1f2 INBOX=Unused
Some explanations on the options suggested:
--folder INBOX to only focus on the folder INBOX
--maxage 1 to select messages less old than 1 day
--dry --nodry1 to avoid really syncing the messages
--debugcontent to display the content
--truncmess 1000 to display only the first 1000 bytes
--f1f2 INBOX=Unused to force imapsync to do something.
Since it is the same mailbox as source and destination,
imapsync usually doesn't sync the messages as they are
already at the destination. The folder Unused has to
exist, otherwise imapsync won't display the messages.
If you pipe this command with a filter like egrep, it becomes like
magic:
imapsync \
--host1 imap.example.com --user1 foo --password1 secret \
--host2 imap.example.com --user2 foo --password2 secret \
--folder INBOX --maxage 1 --dry --nodry1 --debugcontent \
--truncmess 1000 --f1f2 INBOX=Unused \
| egrep 'From:|To:|Subject:|Date:|=====|msg '
Output example:
$RCSfile: FAQ.Fun_Things.txt,v $ $Revision: 1.2 $ $Date: 2021/10/15 13:37:05 $
msg INBOX/22359 copying to INBOX.Fake (not really since --dry mode) ETA: vendredi 15 octobre 2021-10-15 15:31:38 +0200 CEST 57 s 38/38 msgs left
================================================================================
Subject: test: lei7eeCh4Fulahei
To: <tata@localhost.info>
Date: Mon, 11 Oct 2021 11:56:05 +0200 (CEST)
From: gilles@localhost.info (Gilles LAMIRAL)
================================================================================
msg INBOX/22360 copying to INBOX.Fake (not really since --dry mode) ETA: vendredi 15 octobre 2021-10-15 15:31:38 +0200 CEST 57 s 37/38 msgs left
================================================================================
Subject: test: yahShee3Ohvu9mai
To: <tata@localhost.info>
Date: Mon, 11 Oct 2021 11:56:05 +0200 (CEST)
From: gilles@localhost.info (Gilles LAMIRAL)
================================================================================
Thanks to John Fedewa for this trick!
=======================================================================
=======================================================================

65
FAQ.d/FAQ.GDPR.txt Normal file
View File

@ -0,0 +1,65 @@
$Id: FAQ.GDPR.txt,v 1.4 2019/09/12 10:07:27 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.GDPR.txt
=======================================================================
Imapsync and the GDPR, General Data Protection Regulation UE 2016/679
=======================================================================
Questions answered in this FAQ are:
Q. Can you explain the compliance of imapsync with GDPR?
[General Data Protection Regulation UE 2016/679]
Now the questions again with their answers.
=======================================================================
Q. Can you explain the compliance of imapsync with GDPR?
[General Data Protection Regulation UE 2016/679]
R. 1) To know whether a newer imapsync exists or not,
imapsync does a http GET to the file VERSION at
http://imapsync.lamiral.info/VERSION
Via the HTTP User-agent header it also sends:
* the imapsync release
* the Perl version
* the Mail::IMAPClient version
* the Operating System
* the context ( Standard, CGI, or Docker )
You can remove this behavior by adding the option --noreleasecheck
on the command line or by setting $releasecheck = 0 in the source code.
Check also https://nvd.nist.gov/vuln/detail/CVE-2013-4279
See also https://imapsync.lamiral.info/#NUMBERS
2) Each imapsync run generates a logfile that corresponds exactly
to the output printed on the console during the run.
This log file contains data from the mailboxes,
not the content messages, not the passwords but
it contains the folders names and the login names.
Remove the log or use the option --nolog to avoid logging.
3) In case you purchase imapsync:
3.1) the data collected during the purchase,
Name, Company Name, Postal Address, VAT number, Price,
will be used to edit the invoice. I (Gilles LAMIRAL) keep a copy
of the invoice for accounting.
3.2) Once in a while, I send an email to the buyers announcing
a new imapsync is released and where to get it.
3.3) If the customer is a professional inside the EU,
I also use his company VAT number and the amount to declare it
to the Customs each month, before the 10th of the month, at
https://pro.douane.gouv.fr/
It's mandatory, it's the law inside EU and the fine is 750 EUR/month
when it is not done or not well done a month.
=======================================================================
=======================================================================

690
FAQ.d/FAQ.General.txt Normal file
View File

@ -0,0 +1,690 @@
#!/bin/cat
# $Id: FAQ.General.txt,v 1.246 2022/06/08 10:57:52 gilles Exp gilles $
=======================================================================
General FAQ for imapsync
=======================================================================
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.General.txt
Questions answered in this FAQ are:
Q. Do I need to create IMAP mailboxes at the destination platform?
Q. Am I forced to publish the IMAP service on the Internet since the two
environments are not in the same location or same LAN?
Q. What are the most important differences between the Unix shell syntax
and the Windows batch syntax.
Q. How to install imapsync?
Q. How to upgrade imapsync?
Q. How to use imapsync?
Q. Can you give some configuration examples?
Q. How can I have commercial support?
Q. How can I have gratis support?
Q. Where can I find old imapsync releases?
Q. Where can I find free open and gratis imapsync releases?
Q. Is it legal to find imapsync gratis (or not) elsewhere?
Q. How "Facts and figures" are known
https://imapsync.lamiral.info/#NUMBERS
Q. I use --useuid which uses a cache in /tmp or --tmpdir, the hostnames
host1 or host2 has changed but mailboxes are the same. Will imapsync
generate duplicate messages on next runs?
Q. How can I speed up transfers?
Q. I see warning messages like the following:
"Host1 Sent/15 size 1428 ignored (no header so we ignore this message.
To solve this: use --addheader)".
What can I do to transfer those messages?
Q. How can I try imapsync with latest Mail::IMAPClient 3.xx perl module?
Q. How can I use imapsync with Mail::IMAPClient 2.2.9 perl module?
Q. How to verify imapsync.exe I got is the right file bit per bit?
Q. Folders are not created on host2. What happens?
Q. I am interested in creating a local clone of the IMAP on a LAN
server for faster synchronizations, email will always be delivered
to the remote server and so the synchronization will be one way - from
remote to local. How suited is imapsync for continuous one-way
synchronization of mailboxes? Is there a better solution?
Q. I need to log every output on a file named log.txt
Q. Quantifier in {,} bigger than 32766 in regex; marked by <-- HERE in
m/(.{ <-- HERE 1,49947})(?:,|$)/ at Mail/IMAPClient.pm line 2121.
Q. Couldn't create [INBOX.Ops/foo/bar]: NO Invalid mailbox name:
INBOX.Ops/foo/bar
Q. Is it possible to sync also the UIDL of the POP3 server?
Q. Is it possible to sync also the UIDs of the IMAP server?
Q. The option --subscribe does not seem to work
Q. Can Imapsync filter Spam during the sync?
Q. How to migrate from uw-imap with an admin/authuser account?
Q: How to migrate from Sun Java Enterprise System / Sun One / iPlanet /
Netscape servers with an admin account?
Q. Is there a way to delete the destination folder when the source
folder is no longer there?
Q. I would love to have a function to inject lines in the header.
Things like "X-migrated-from-foo: 20100617"
Q. I want to play with headers line and --regexmess but I want to leave
the body as is.
Q. My imap server does not accept a message and warns
"Invalid header". What is the problem?
Q: How can I write an .rpm with imapsync
Q. Where I can read up on the various IMAP RFCs?
Now the questions again with their answers.
=======================================================================
Q. Do I need to create IMAP mailboxes at the destination platform?
R. Yes!
Imapsync does only IMAP and there is no way to create an account
with the standard IMAP protocol. So you have to create them first.
=======================================================================
Q. Am I forced to publish the IMAP service on the Internet since the two
environments are not in the same location or same LAN?
R. The host where you run imapsync has to contact both servers via imap.
You are not obliged to publish the imap service on the Internet if
the host running imapsync can contact both imap servers via imap.
Use their names or their IP addresses.
=======================================================================
Q. What are the most important differences between the Unix shell syntax
and the Windows batch syntax.
R. There are several differences between Unix and Windows
in the command line syntax.
- Character \ on Unix versus ^ on Windows
- Character ' on Unix versus " on Windows
Details:
A) \ versus ^
On Unix shells you can write a single command on multiple lines
by using the escape character \ at the end of each line
(except the last one). On Windows this character is ^
Unix example:
./imapsync \
--host1 imap.truc.org --user1 foo --password1 secret1 \
--host2 imap.trac.org --user2 bar --password2 secret2
Windows example:
imapsync ^
--host1 imap.truc.org --user1 foo --password1 secret1 ^
--host2 imap.trac.org --user2 bar --password2 secret2
Of course you can write the command on a single line without
characters \ nor ^. I use them because the output is
better, no truncation, pretty print. It's just sugar!
In this FAQ I use \ for examples. Transcript to ^ if
you're on a Windows system.
B) Quote vs Double-quote, ie ' versus "
On Windows the single quote character ' doesn't work
like on Unix so in the examples of this FAQ the
command containing single quotes ' will fail on Windows.
To fix this, just replace single quotes ' by double quotes "
Also on Windows, in examples with \$1, replace
any \$1 by $1 (remove the \ before $).
=======================================================================
Q. How to install imapsync?
R. Read the INSTALL files in the tarball. They are also available at
https://imapsync.lamiral.info/#install
and
https://imapsync.lamiral.info/INSTALL.d/
=======================================================================
Q. How to upgrade imapsync?
R. Do like a fresh installation.
=======================================================================
Q. How to use imapsync?
R. Read the TUTORIAL_Unix.txt file, maybe the README and, if you
encounter problems, the FAQ.d/* files in the tarball.
All are also available and updated at:
https://imapsync.lamiral.info/#doc
=======================================================================
Q. Can you give some configuration examples?
R1. Basic usage is described there:
https://imapsync.lamiral.info/#DOC_BASIC
It is:
imapsync --host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
R2. Some FAQ files contains many examples for several scenarios
https://imapsync.lamiral.info/#doc
(Gmail, Office365, Exchange, Darwin, etc.)
=======================================================================
Q. How can I have commercial support?
R. Buy support from imapsync author: Gilles LAMIRAL
https://imapsync.lamiral.info/#buy_all
=======================================================================
Q. How can I have gratis support?
R. Write to the imapsync author: Gilles LAMIRAL
https://imapsync.lamiral.info/#AUTHOR
I help all users as long as I have time to help them all,
users who bought the support get my help first.
=======================================================================
Q. Where can I find old imapsync releases?
R. Search the internet or ask the author.
=======================================================================
Q. Where can I find free open and gratis imapsync releases?
R. Search the internet or ask the author.
Q. Is it legal to find imapsync gratis (or not) elsewhere?
R. Yes, the license permits it
https://imapsync.lamiral.info/NOLIMIT
=======================================================================
Q. How "Facts and figures" are known
https://imapsync.lamiral.info/#NUMBERS
R. To know wether a newer imapsync exists or not, imapsync does a http
GET to http://imapsync.lamiral.info/VERSION
Via the User-agent parameter it also send:
* imapsync release
* Perl version
* Mail::IMAPClient version
* Operating System
You can remove this behavior by adding option --noreleasecheck on the
command line (or by setting $releasecheck = 0 in the source code)
or by using the github release.
=======================================================================
Q. I use --useuid which uses a cache in /tmp or --tmpdir, the hostnames
host1 or host2 has changed but mailboxes are the same. Will imapsync
generate duplicate messages on next runs?
R. Yes
Q. How can I fix this?
R. The cache path reflects exactly hostnames or ip addresses given via
--host1 and --host2 values. So just change the directory names
of host1 or host2. Use --dry to see if next runs will generate
duplicates.
By default on Unix the cache is like
/tmp/imapsync_cache/host1/user1/host2/user2/...
=======================================================================
Q. How can I speed up transfers?
R. By using --useuid imapsync avoid getting messages headers and build
a cache. On Unix a good thing is to add also --tmpdir /var/tmp
to keep the cache since /tmp is often cleared on reboot.
imapsync ... --useuid
On Unix:
imapsync ... --useuid --tmpdir /var/tmp/
R. Add also --nofoldersizes since the default behavior is to compute
folder sizes. Folder sizes are useless for the transfer, just
useful to see what has to be done on each folder and guess when
the transfer will end (ETA).
R. Add also --noexpungeaftereach if you use --delete1.
But be warned that an interrupted transfer can lose messages
on host2 in a second run if you use a (silly) combination like:
imapsync ... --delete1 --noexpunge1 --noexpungeaftereach --expunge2
Notes about --useuid
Case where building the cache first is necessary (to avoid multiples transfers)
If you run again imapsync with --useuid on a transfer already done without
--useuid then, to avoid messages be copied again, first run imapsync
with --usecache but without --useuid, example scenario:
A] Running with the default options, I began without --useuid
1) First run with default options
imapsync ...
Too slow, I want to speed up!
2) Build the cache
imapsync ... --usecache
3) Speed up now
imapsync ... --useuid
B] I began with --useuid from the first time
1) First run and next runs with --useuid
imapsync ... --useuid
Inodes number issue.
The cache is simple, it uses the file-system natively,
it's just an empty file per message transfered.
When mailboxes are huge the cache can exhaust the number of inodes
allowed in the filesystem, that's a limitation like limitation
size but it's less often encountered.
On Unix, to predict whether your tmpdir filesystem used by imapsync
will support the whole cache, just run the command "df -i /var/tmp",
if /var/tmp is the --tmpdir argument.
On windows, search and drop me a note about how to count the number
of files allowed in the filesystem.
It seems FAT32 supports 268 435 445 clusters.
Choosing the number of inodes allowed by a filesystem can be done
at the creation of it with "mkfs -N number-of-inodes ..."
imapsync can predict how many messages have to be synced with the
option --justfoldersizes (no transfer will be done)
imapsync ... --justfoldersizes
=======================================================================
Q. I see warning messages like the following:
"Host1 Sent/15 size 1428 ignored (no header so we ignore this message.
To solve this: use --addheader)".
What can I do to transfer those messages?
R1. Like suggested inline, use --addheader option.
Option --addheader will add an header line like
Message-Id: <15@imapsync>
where 15 is the message UID number on host1.
Then imapsync will transfer the changed message on host2.
Duplicates won't happen on next runs.
imapsync ... --addheader
R2. Other solution.
Use --useuid then imapsync will avoid dealing with headers.
imapsync ... --useuid
=======================================================================
Q. How can I try imapsync with latest Mail::IMAPClient 3.xx perl module?
R1 Run:
cpanm Mail::IMAPClient # this uses cpanminus
or
cpan -i Mail::IMAPClient
or
perl -MCPAN -e "install Mail::IMAPClient"
R2 If you want to install the Perl module locally in a directory
- Download latest Mail::IMAPClient 3.xx at
http://search.cpan.org/dist/Mail-IMAPClient/
- untar it anywhere:
tar xzvf Mail-IMAPClient-3.xx.tar.gz
- Get any imapsync (latest is better).
- run imapsync with perl and -I option tailing to use the perl
module Mail-IMAPClient-3.xx. Example:
perl -I./Mail-IMAPClient-3.39/lib ./imapsync ...
or if imapsync is in directory /path/
perl -I./Mail-IMAPClient-3.39/lib /path/imapsync ...
=======================================================================
Q. How can I use imapsync with Mail::IMAPClient 2.2.9 perl module?
R. Mail::IMAPClient 2.2.9 is no longer supported.
Help yourself.
=======================================================================
Q. How to verify imapsync.exe I got is the right file bit per bit?
R. Use md5sum to check integrity of the file.
Get md5sum.exe at http://etree.org/md5com.html
md5sum imapsync.exe
Then compare the checksum with the one given by the author.
=======================================================================
Q. Folders are not created on host2. What happens?
R. Do you use IMAP or POP3 with your client software?
It looks like you use POP3 instead of IMAP, POP3 sees only INBOX.
=======================================================================
Q. I am interested in creating a local clone of the IMAP on a LAN
server for faster synchronizations, email will always be delivered
to the remote server and so the synchronization will be one way - from
remote to local. How suited is imapsync for continuous one-way
synchronization of mailboxes? Is there a better solution?
R. If messages are delivered remotely and you play locally with the
copy, in order to have fast access, then the synchronization can't
be one way. You may change flags, you may move messages in
different folders etc. The issue described is clearly
two-ways sync.
A better tool with this scenario is offlineimap,
designed for this issue, and faster than imapsync.
=======================================================================
Q. I need to log every output on a file named log.txt
R1. imapsync logs on a file by default, its name is given at the
beginning and the end of each run. This name is unique since
it is compound of the current date and time and user2 value.
R2. To change this default name, use --logfile log.txt
imapsync ... --logfile log.txt
=======================================================================
Q. Quantifier in {,} bigger than 32766 in regex; marked by <-- HERE in
m/(.{ <-- HERE 1,49947})(?:,|$)/ at Mail/IMAPClient.pm line 2121.
R. Do not use a bigger value than 3276 with --split1 or --split2
=======================================================================
Q. Couldn't create [INBOX.Ops/foo/bar]: NO Invalid mailbox name:
INBOX.Ops/foo/bar
Let begin by an explanation.
Example:
sep1 = /
sep2 = .
imapsync reverts each separator automatically.
a) All / character coming from host1 are converted to . (convert the separator)
b) All . character coming from host1 are converted to / (to avoid
intermediate unwanted folder creation).
So
INBOX/Ops.foo.bar (Ops.foo.bar is just one folder name) will be translated to
INBOX.Ops/foo/bar
Sometimes the sep1 character is not valid on host2 (character "/" usually)
R. Try :
--regextrans2 "s,/,X,g"
It'll convert / character to X
Choose X as you wish: _ or SEP or
any string (including the empty string).
This issue is automatically fixed by default since imapsync
release 1.513, use --nofixslash2 to suppress the fix.
=======================================================================
Q. Is it possible to sync also the UIDL of the POP3 server?
R. imapsync doesn't talk POP3 but I think you mean UID in IMAP.
See next question.
=======================================================================
Q. Is it possible to sync also the UIDs of the IMAP server?
R. UIDs in IMAP are chosen and created by the server, not by the
client software. imapsync is a client software command.
So UIDs cannot be synced by any imap method.
UIDs may be synced via a rsync command between the imap servers but
it implies they are the same software, among other constraints.
=======================================================================
Q. The option --subscribe does not seem to work
R1. Use it with --subscribed
R2. There is also the --subscribe_all option that subscribe
to all folders on host2.
======================================================================
Q. Can Imapsync filter Spam during the sync?
R. No, imapsync doesn't detect Spam by itself. But I've seen blogs and
Spamassassin documentation explaining solutions to apply Spamassassin
to a imap mailbox. So you can apply one of these solutions on the host1
source mailbox before the imapsync run or on the destination host2
mailbox after the imapsync transfer.
http://www.stearns.org/doc/spamassassin-setup.current.html#isbg
http://euer.krebsco.de/using-spamassassin-on-a-remote-imap-host.html
https://github.com/ook/isbg
Imapsync can delegate this job during its IMAP syncs via the
--pipemess option but the underlying spam tool has to be written.
======================================================================
Q. How to migrate from uw-imap with an admin/authuser account?
R. Use the following syntax:
imapsync ... --user1="loginuser*admin_user" --password1 "admin_user_password"
======================================================================
Q: How to migrate from Sun Java Enterprise System / Sun One / iPlanet /
Netscape servers with an admin account?
R: Those imap servers don't allow the typical use of --authuser1 to use an
administrative account. They expect the use of an IMAP command called
proxyauth that is issued after login in as an administrative account.
For example, consider the administrative account 'administrator' and your
real user 'real_user'. The IMAP sequence would be:
OK [CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS
CHILDREN BINARY UNSELECT LANGUAGE STARTTLS XSENDER X-NETSCAPE XSERVERINFO
AUTH=PLAIN] imap.server IMAP4 service (Sun Java(tm) System Messaging
Server ...))
1 LOGIN administrator password
1 OK User logged in
2 PROXYAUTH real_user
2 OK Completed
In imapsync, you can achieve this by using the following options:
--host1 source.imap.server \
--user1 real_user \
--authuser1 administrator \
--proxyauth1 \
--passfile admin.txt
=======================================================================
Q. I would love to have a function to inject lines in the header.
Things like "X-migrated-from-foo: 20100617"
R. You can do that with:
imapsync ... --regexmess 's/\A/X-migrated-from-foo: 20100617\n/'
It will insert a first header line containing "X-migrated-from-foo: 20100617"
=======================================================================
Q. I want to play with headers line and --regexmess but I want to leave
the body as is.
R. The header/body separation is a blank line so an example:
--regexmess 's{\A(.*?(?! ^$))^Date:(.*?)$}{$1Date:$2\nX-Date:$2}gxms'
Will replace the next three lines
Message-ID: <499EF800.4030002@blabla.fr>
Date: Fri, 20 Feb 2009 19:35:44 +0100
From: Gilles LAMIRAL <lamiral@linux-france.org>
by the next four lines
Message-ID: <499EF800.4030002@blabla.fr>
Date: Fri, 20 Feb 2009 19:35:44 +0100
X-Date: Fri, 20 Feb 2009 19:35:44 +0100
From: Gilles LAMIRAL <lamiral@linux-france.org>
This example just add an header line "X-Date:" based on "Date:" line.
=======================================================================
Q. My imap server does not accept a message and warns
"Invalid header". What is the problem?
R. You fall in the classical mbox versus Maildir/ format
problem. May be you use a misconfigured procmail rule.
A header beginning like the following one is in the mbox
format, header line 1 has no colon behind "From", header
lines 2 through N do have a colon :
From foo@yoyo.org Sat Jun 22 01:10:21 2002
Return-Path: <foo@yoyo.org>
Received: ...
Any Maildir/ configured imap server may refuse this message since its
header is invalid. The first "From " line is not valid. It lacks a
colon character ":". To solve this issue you have several solutions
a) Remove manually this first "From " line for each message before
using imapsync.
b) Replace manually the whitespace by a colon in string "From " but you
might end with two "From:" lines (just have a look at the other
header lines of the message)
c) Run imapsync with the following option (this replaces "From "by "From:"):
--regexmess 's/\AFrom /From: /'
or may be better (no other "From:" collision):
d) Run imapsync with the following option (this replaces "From "by "X-om:"):
--regexmess 's/\AFrom /X-From: /'
e) Run imapsync with the following option (this removes the whole "From " line):
--regexmess 's{\AFrom\ [^\n]*(\n)?}{}gxms'
Solution e) is solution a) made by imapsync itself.
Solutions c) and d) keep "From " lines information
(normally it's useless to keep them)
Best solutions are e) or d).
I prefer the d) solution because it fixes the issue
and keeps the old "From" value while the e) solution removes
the old "From" value definitively.
======================================================================
Q: How can I write an .rpm with imapsync
R. You'll find an RPM imapsync.spec file in the directory learn/rpm/
It has been downloaded from
https://svn.fysik.dtu.dk/projects/rpmbuild/trunk/SPECS/imapsync.spec
It has been tested with imapsync 1.434 (May 2011) on CentOS5
and RedHat RHEL5 Linux. (Thanks to Ole Holm Nielsen).
This imapsync.spec is coming from Neil Brown work in 2007.
=======================================================================
Q. Where I can read up on the various IMAP RFCs?
R. Here:
RFC 3501 - INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1
http://www.faqs.org/rfcs/rfc3501.html
RFC 2683 - IMAP4 Implementation Recommendations
http://www.faqs.org/rfcs/rfc2683.html
RFC 2595 - Using TLS with IMAP, POP3 and ACAP
http://www.faqs.org/rfcs/rfc2595.html
RFC 2822 - Internet Message Format
http://www.faqs.org/rfcs/rfc2822.html
RFC 2342 - IMAP4 Namespace
http://www.faqs.org/rfcs/rfc2342.html
RFC 2180 - IMAP4 Multi-Accessed Mailbox Practice
http://www.faqs.org/rfcs/rfc2180.html
RFC 4549 - Synchronization Operations for Disconnected IMAP4 Clients
http://www.faqs.org/rfcs/rfc4549.html
=======================================================================
=======================================================================

856
FAQ.d/FAQ.Gmail.txt Normal file
View File

@ -0,0 +1,856 @@
#!/bin/cat
$Id: FAQ.Gmail.txt,v 1.92 2022/06/28 11:03:40 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
=======================================================================
Imapsync tips for Gmail accounts.
=======================================================================
Questions answered in this FAQ are:
Q. Can I use imapsync to transfer from or to Gmail accounts?
Q. Even after several syncs the number of messages differs
largely between imapsync and the Gmail web interface.
What can I do?
Q. How many days does it take to transfer X GB?
Q. What is the Gmail free maximum storage size?
R. 15 GB in 2020. Need more?
https://gsuite.google.com/pricing.html
Q. How to synchronize from Gmail to Gmail?
Q. How to synchronize from XXX to Gmail?
Q. How to synchronize from Gmail to XXX?
Q. I can't authenticate with Gmail via IMAP
and Gmail says
"Please log in via your web browser"
or it says
"Application-specific password required: https://support.google.com/accounts/answer/185833 (Failure)"
Q. Can not open imap connection on [imap.gmail.com]
Unable to connect to imap.gmail.com
Q. I want to sync the Sent folder only, what is it on Gmail?
Q. From Gmail to Gmail, I want to keep Categories. What can I do?
Q. The context is a Gmail account at the source, but not Gmail at the
destination. I want cross-duplicates, different Gmail folders/labels
generating duplicate messages across folders on the destination,
except for the "All Mail" folder.
Q. I can't sync more than 1000 messages in each Gmail folder.
What can I do?
Q. What are the Gmail imap names of the folders
"Sent" "Trash" "Drafts" "Junk" "Flagged" and "All"?
Q. Can I transfer Gmail "Chat" messages with imapsync?
Q. Can I safely use --useuid or --usecache for Gmail transfers?
Q. Can I use --syncduplicates Gmail transfers?
Q. Gmail does not really delete messages in folder [Gmail]/All Mail
What happens? What can I do?
Q. Can I use the Extension of the SEARCH command: X-GM-RAW described at
https://support.google.com/mail/answer/7190?hl=en
https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw
Q. How to avoid the [IMAP] prefix on Gmail side?
Q. Does imapsync have the capability to do 2 stage authentication?
Q. How to use XOAUTH2 to globally authenticate gmail users?
Q. How to use XOAUTH to globally authenticate gmail users?
Q. How to use a Gmail account to backup several different imap accounts?
Q. How to migrate email from gmail to google apps?
Now the questions again with their answers.
=======================================================================
Q. Can I use imapsync to transfer from or to Gmail accounts?
R. Yes. But IMAP access to a Gmail account is not allowed by default so
it has to be allowed in the Gmail configuration part:
-> Do not go to "Manage your Google Account" (the upper right icon with your Google face)
-> Instead, go to Gmail at https://mail.google.com/
-> Click on Settings, the gear button on the upper right of Gmail.
-> Click on "See all settings" at the top of the pull down menu.
-> Go to the tab "Forwarding and POP/IMAP"
-> Go down to "IMAP Access"
-> Activate the radio button "Enable IMAP"
-> Click on the button "Save changes" at the bottom of the page.
That's not finished, there is also the authentication issue to consider.
There are two different ways to authenticate to Gmail with Imapsync,
either:
1) "App password with 2-step verification"
or
2) XOauth2 ready for groups on any OS or
for individual Gmail authentication on Linux.
For the second one, XOauth2, see the document
https://imapsync.lamiral.info/FAQ.d/FAQ.XOAUTH2.txt
1) "App password with 2-step verification" is the first way to
authenticate to Gmail with Imapsync.
Go to
https://myaccount.google.com/security
and sign in with the account you want to transfer email to or from.
Go to the "Signing in to Google" part of the security page and follow
the process to activate the "2-step verification" until it is ok.
Then, just below "2-step verification" click on the "App passwords"
and follow the instruction to get a 16 characters password, on the
pull down button "Select app", choose "Other (Custom name)" and call
it imapsync for example. Now use this this password to access the
mailbox with imapsync.
The following document details what I've just described:
https://support.google.com/accounts/answer/185833?hl=en
Even if Google says "Every App Password is only used once",
it's false and the password can be used many times.
=======================================================================
Q. Even after several syncs the number of messages differs
largely between imapsync and the Gmail web interface.
What can I do?
R. Turned off "Conversation views" on Gmail.
With "Conversation views" turned off, Gmail shows the exact message
count for each folder and it matches what imapsync shows.
With "Conversation views" on, a single conversation can have
multiple messages and it doesn't match what imapsync presents.
(Thanks to Karthik.V for this input)
=======================================================================
Q. How many days does it take to transfer X GB?
R. Basically it takes X days to transfer X GB per account.
Gmail has usage limits per day and use throttles when
they are overtaken
https://support.google.com/a/answer/1071518?hl=en
From the previous link:
* Upload with IMAP: 500 MB per day.
It's 2X days to upload X GB to Gmail, it's why I suggest to add
--maxbytespersecond 20000 --maxbytesafter 1_000_000_000
for uploading messages to Gmail
(should be --maxbytespersecond 5787 --maxbytesafter 500_000_000
but the higher values given work well)
* Download with IMAP: 2500 MB per day.
It's X/2.5 days to download X GB from Gmail, it's why I suggest to add
--maxbytespersecond 40_000 --maxbytesafter 3_000_000_000
for downloading messages from Gmail
(should be --maxbytespersecond 28_935 --maxbytesafter 2_500_000_000
That's empirical values that work in practice.
Try upper values and see if they still work.
I would be grateful if you report them to me.
How Gmail says limits are reached?
This is either a disconnection with
"BYE Session expired, please login again"
or a very small rate, less than 1 Kib/s
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
=======================================================================
Q. How to synchronize from Gmail to Gmail?
R. Use the following example:
imapsync \
--user1 account1@gmail.com \
--password1 gmailsecret1 \
--user2 account2@gmail.com \
--password2 gmailsecret2 \
--gmail1 --gmail2
Or, replacing what does --gmail1 --gmail2 (values are from imapsync 2.178):
imapsync \
--host1 imap.gmail.com \
--ssl1 \
--user1 account1@gmail.com \
--password1 gmailsecret1 \
--host2 imap.gmail.com \
--ssl2 \
--user2 account2@gmail.com \
--password2 gmailsecret2 \
--maxbytespersecond 20_000 \
--maxbytesafter 1_000_000_000 \
--automap \
--maxsleep 2 \
--useheader X-Gmail-Received --useheader Message-Id \
--synclabels \
--resynclabels \
--exclude "\[Gmail\]$" \
--folderlast "[Gmail]/All Mail"
With --gmail1 --gmail2 you can override the parameters
that are activated by them, for example:
imapsync ... --gmail1 --gmail2 --maxbytespersecond 50_000 --noautomap
Explanations:
--maxbytespersecond 20_000 ( 20 kBytes/s ) option is here to avoid
locking or errors when imap transfers exceed Gmail maximum limit.
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--maxbytespersecond 20_000 is not mandatory in the sense Gmail may
allow you to use an upper value than 1 GBytes per 24h without
disconnections. If you count well, it should be 10_000, not 20_000.
--maxbytesafter 1_000_000_000 option is there because
option --maxbytespersecond 20000 is unnecessarily slow if the Gmail
account size is less than 1 GB.
--maxsleep 2 is to avoid sleeping more than 2 seconds when
the --maxbytespersecond value implies a long sleep between 2 copies.
--automap is not mandatory but it's a feature to automatically
map folder names based on the Gmail user configuration itself, par account.
It will save manual folder names changes or the use of --regextrans2
or --f1f2 to map folder names.
For example, imap folder "[Gmail]/Sent Mail" may be mapped as one of
E-mails enviados
Enviada
Enviado
Gesendet
Gönderildi
Inviati
Sendt
Skickat
Verzonden
etc.
on both sides, host1 or host2, maybe differently, sometimes in
incomprehensible alphabets, a headache for imap sysadmins.
See a listing here:
http://stackoverflow.com/questions/2185391/localized-gmail-imap-folders/2185548#2185548
--exclude "\[Gmail\]$" is just there to avoid a warning error
when selecting this not used folder.
Be aware that --gmail1 --gmail2 is a special case, it's not what
--gmail1 does plus what --gmail2 does when they are not invoked
together.
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
=======================================================================
Q. How to synchronize from XXX to Gmail?
R. Use the following example:
imapsync --host1 mail.oldhost.com \
--user1 my_email@oldhost.com \
--password1 passwordold \
--user2 my_email@gmail.com \
--password2 gmailsecret \
--gmail2
Or, replacing what does --gmail2 (values are from imapsync 2.178):
(For Linux only because of the quotes)
imapsync --host1 mail.oldhost.com \
--user1 my_email@oldhost.com \
--password1 passwordold \
--host2 imap.gmail.com \
--ssl2 \
--user2 my_email@gmail.com \
--password2 gmailsecret \
--maxbytespersecond 20_000 \
--maxbytesafter 1_000_000_000 \
--automap \
--expunge1 \
--addheader \
--maxsleep 2 \
--maxsize 35_651_584
--exclude "\[Gmail\]$" \
--regextrans2 's,\[Gmail\].,,' \
--regextrans2 's,^ +| +$,,g'
--regextrans2 's,/ +| +/,/,g'
--regextrans2 's/['"'"'\\^"]/_/g'
--folderlast "[Gmail]/All Mail"
Starting from imapsync 1.926 a --regextrans2 replacing
all blanks, was:
--regextrans2 "s/[ ]+/_/g"
With --gmail2 you can override the parameters
that are activated by them, for example:
imapsync ... --gmail2 --maxbytespersecond 50_000 --noautomap --maxsleep 0
Explanations:
--maxbytespersecond 20_000 ( 20 kBytes/s ) option is here to avoid
locking or errors when imap transfers exceed Gmail maximum limit.
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--maxbytespersecond 20_000 is not mandatory in the sense Gmail may
allow you to use an upper value than 1 GBytes per 24h without
disconnections. If you count well, it should be 10_000, not 20_000.
--maxbytesafter 1_000_000_000 option is there because
option --maxbytespersecond 20000 is unnecessarily slow if the Gmail
account size is less than 1 GB. Counting well, it should
be 500_000_000, not 1_000_000_000, but 1GB is usually ok.
--maxsleep 2 is to avoid sleeping more than 2 seconds when
the --maxbytespersecond value implies a long sleep between 2 copies.
--maxsleep 0 cancels any imapsync bandwidth restriction given by
the options --maxbytespersecond or --maxmessagespersecond
It does not cancel the imap server bandwidth restrictions.
--automap is optional but it will save manual folders names
changes or the use of --regextrans2 or --f1f2 to map folder names.
--expunge1 is optional. It deletes messages marked \Deleted on host1.
Imapsync syncs messages with all their flags, Gmail takes the messages
marked \Deleted but deletes or moves them just after.
Option --expunge1 really removes messages marked \Deleted on host1
so they are not synced at all.
The --addheader option is there because "Sent" folder messages
sometimes lack the "Message-Id:" and "Received:" headers needed
by imapsync to identify messages (only when --useuid is not used).
So option --addheader adds a "Message-Id" header consisting of
the imap UID of the message on the host1 folder, like
"Message-Id: 12345@imapsync".
--exclude "\[Gmail\]$" is there to avoid a small examine/select error:
"Could not examine: 43 NO [NONEXISTENT] Unknown Mailbox: [Gmail]
(now in authenticated state) (Failure)".
--regextrans2 "s/[ ]+/_/g" is there to convert blank characters not
accepted by gmail to character _ underscore. In fact only leading
and trailing blank characters are problems with gmail, and
also successive blanks ending with the IMAP error
"NO [CANNOT] Folder contains excess whitespace (Failure)"
If you want to change only leading and trailing blank characters
then use the following instead
On Linux/Unix:
--regextrans2 "s,(/|^) +,\$1,g" --regextrans2 "s, +(/|$),\$1,g"
On Windows:
--regextrans2 "s,(/|^) +,$1,g" --regextrans2 "s, +(/|$),$1,g"
--regextrans2 "s/[\^]/_/g" is mandatory. It converts, since
not accepted by gmail, character ^ to character _ underscore.
--regextrans2 's/['"'"'\\^"]/_/g' is optional. It converts
characters ' or " or \ to character _ underscore.
You can select folders exported to imap within the gmail preferences.
Select or unselect some "System labels", depending on your needs.
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
=======================================================================
Q. How to synchronize from Gmail to XXX?
R. Use this example:
imapsync \
--user1 foo@gmail.com \
--password1 gmailsecret \
--host2 imap.example.org \
--user2 tata \
--password2 tatasecret \
--gmail1
Or, replacing what does --gmail1 (values are from imapsync 2.178):
imapsync \
--host1 imap.gmail.com \
--ssl1 \
--user1 foo@gmail.com \
--password1 gmailsecret \
--host2 imap.example.org \
--user2 tata \
--password2 tatasecret \
--maxbytespersecond 40_000 \
--maxbytesafter 3_000_000_000 \
--automap \
--maxsleep 2 \
--skipcrossduplicates \
--useheader="X-Gmail-Received" \
--useheader "Message-Id" \
--regextrans2 "s,\[Gmail\].,," \
--folderlast "[Gmail]/All Mail"
With --gmail1 you can override the parameters
that are activated by them, for example:
imapsync ... --gmail1 --maxbytespersecond 50_000 --noskipcrossduplicates --noautomap
Explanations:
--maxbytespersecond 40_000 ( 40 kBytes/s ) option is here to avoid
locking or errors when imap transfers exceed Gmail maximum limit.
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--maxbytespersecond 40_000 is not mandatory in the sense Gmail may
allow you to use an upper value than 2 GBytes per 24h without
disconnections.
--useheader="X-Gmail-Received" --useheader "Message-Id" are not
mandatory. I use them because I found (several years ago, it may have
changed) that Gmail always adds a different header "X-Gmail-Received:"
to all messages it gets. So the identification by imapsync can not
fail using this header. "Message-Id" is there for safety about this
Gmail rule.
--automap is optional but it will save manual folder names changes or
the use of --regextrans2 to map folder names.
--regextrans2 "s,\[Gmail\].,,"
If your destination imap server doesn't like the "[Gmail]" name,
get rid of this "[Gmail]" part with that.
You can select folders exported to imap within the Gmail preferences,
for example, you may unselect all "System labels".
--skipcrossduplicates is optional but it can save Gigabytes of hard
disk memory. Within imap protocol, Gmail presents Gmail labels as
folders, so a message labelled "Work" "ProjectX" "Urgent" ends up in
three different imap folders "Work" "ProjectX" and "Urgent" after an
imap sync. Option --skipcrossduplicates prevent this behavior.
An issue with --skipcrossduplicates is that the first label synced by
imapsync goes to its corresponding folder but other labels are then
ignored. You can choose what labels have priority by using the
--folderfirst option. For example, --folderfirst "Work" will sync
messages labelled "Work" before messages labelled "CanWait" or
"Urgent". By default imapsync syncs folders (Gmail labels) using the
classical alphanumeric order.
Option --folderlast "CanWait" will sync only messages that only have
the label CanWait.
Option --folderlast "[Gmail]/All Mail", in conjunction with option
--skipcrossduplicates will only put in "[Gmail]/All Mail" the messages
that are not labelled at all.
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
=======================================================================
Q. I can't authenticate with Gmail via IMAP
and Gmail says
"Please log in via your web browser"
or it says
"Application-specific password required: https://support.google.com/accounts/answer/185833 (Failure)"
R0. My current settings on two Gmail accounts are made
with three steps 0) 1) 2)
0) Login via a web browser to the Gmail account concerned.
1) The 2-step verification is OFF
https://myaccount.google.com/security?hl=en&pli=1&nlr=1#signin
2) Access for less secure apps is turned ON
https://www.google.com/settings/security/lesssecureapps
https://support.google.com/accounts/answer/6010255?hl=en
R1. See Coert Grobbelaar solution:
https://web.archive.org/web/20150906230041/http://security.stackexchange.com/questions/86404/how-do-i-interact-with-google-to-import-email-via-imapsync
https://support.google.com/accounts/answer/185833
R2. I had the same issue one time (mars 2015) logging to Gmail
with imapsync.
The Gmail imap message error said "Please log in via your web browser"
so I logged for this account via a web browser. Then,
it asked me to receive a code via a mobile, I said yes,
I entered the code and everything went ok on next syncs
with imapsync.
R1 again. Since people rarely follow links in a txt file, I reproduce
Google documentation from the link
https://support.google.com/accounts/answer/185833
Sign in using App Passwords
An App Password is a 16-digit passcode that gives a non-Google app or
device permission to access your Google Account.
App Passwords can only be used with accounts that
have 2-Step Verification turned on.
When to use App Passwords
To help keep your account secure, use "Sign in with Google" to
connect apps to your Google Account.
If the app you’re using doesn’t offer this option, you can either:
Use App Passwords to connect to your Google Account
Switch to a more secure app
Create & use App Passwords
Note: If you use 2-Step-Verification
( https://support.google.com/accounts/answer/185839 )
and are seeing a "password incorrect" error when trying to access
your Google Account, an App Password may solve the problem.
* Go to your Google Account https://myaccount.google.com/
* On the left navigation panel, choose Security.
* On the "Signing in to Google" panel, choose App Passwords.
If you don’t see this option:
* 2-Step Verification is not set up for your account.
* 2-Step Verification is set up for security keys only.
* Your account is through work, school, or other organization.
* You’ve turned on Advanced Protection for your account.
* At the bottom, choose Select app and choose the app you’re using.
* Choose Select device and choose the device you’re using.
* Choose Generate.
* Follow the instructions to enter the App Password.
The App Password is the 16-character code in the yellow bar on your device.
* Choose Done.
Most of the time, you’ll only have to enter an App Password once per
app or device, so don’t worry about memorizing it.
=======================================================================
Q. Can not open imap connection on [imap.gmail.com]:
Unable to connect to imap.gmail.com
R0. It looks like this issue is related to ipv6. Both ipv4 and ipv6
protocols should work with gmail and imapsync, I test that regularly,
imapsync works fine for both ipv4 and ipv6.
If you disable ipv6 then disable also ipv6 resolution!
The default names resolution order is to present ipv6 name resolutions
first. If you know how to make ipv4 answers be taken before ipv6
then tell me.
R1. First solution, run imapsync with the option --inet4:
imapsync ... --inet4
R2. A second solution is to use directly gmail ipv4 ip address:
imapsync ... --host1 64.233.184.108
In case it changes, get it with any command showing the imap.gmail.com
name resolution, try one of those:
nslookup imap.gmail.com
host imap.gmail.com
ping imap.gmail.com
Or go to http://ping.eu/nslookup/ to get the resolution.
Thanks to Chris Nolan to report, understand and fix this issue!
=======================================================================
Q. I want to sync the Sent folder only, what is it on Gmail?
R1. Use:
imapsync ... --folder "[Gmail]/Sent Mail"
R2. In fact, by configuration from the user, it can be another folder
name presented in IMAP. Use --automap --dry to see this mapping.
See below
=======================================================================
Q. From Gmail to Gmail, I want to keep Categories. What can I do?
R. First, read a document to learn how to play with Gmail Categories:
https://support.google.com/mail/answer/3094499
Categories are not presented in IMAP in Gmail, so synchronizing them
with imapsync requires some work.
Let's say I want to synchronize the category named "Updates".
1. Create a label "Updates_L", or name it as you want. You can't
create a label named "Updates" like the category name because
Gmail will refuse to create it, saying it's a reserved name.
2. Select all messages in the category "Updates".
To do so, you can use the string "category:updates" in the
search input text located at the top of the Gmail page.
Notice the case is irrelevant to the search,
"category:updates" equals "Category:Updates".
Gmail then announces "All 100 messages on this page are selected."
It's not enough, I want all the "Updates" category messages.
So, on the right of this statement, click on the text
"Select all messages that match this search".
Now Gmail announces "All messages in this search are selected."
3. Apply label "Updates_L" created at step 1 to all the selected
messages. Gmail will present a warning and a
"Confirm bulk action" button. Press "Ok".
4. Repeat steps 1 to 3 for each category you want to synchronize.
5. Transfer your mailbox to the new account with imapsync using the
options --gmail1 --gmail2, the labels will be synchronized.
6. For each of the labels you made for categories, do the steps 7 to 9.
As an example, I take the label "Updates_L" previously used.
7. Select all messages with the label "Updates_L".
You can use the string "label:Updates_L" in the search input
text located at the top of the Gmail page.
Gmail then announces "All 100 messages on this page are selected."
It's not enough, I want all the "updates_l" labeled messages.
So, on the right of this statement, click on the text
"Select all messages that match this search".
Now Gmail announces "All messages in this search are selected."
(does it sound similar to a previous §?)
8. Apply the category "Updates" to all the selected
messages. Gmail will present a warning and a
"Confirm bulk action" button. Press "Ok".
If you search where are the categories: the categories are
located with the labels, as if they were labels.
9. Delete the temporary label. That part is optional.
Maybe one day Google will make categories optionally available
in IMAP, like any other label and this trick will be obsolete.
Labels can be "Show in IMAP" at
https://mail.google.com/mail/u/0/#settings/labels
Categories are listed there but not possibly "Show in IMAP", not yet :-)
Thanks to Michael Leggett https://twitter.com/leggett for this
trick. See the story at
https://twitter.com/leggett/status/1484632316697739267?s=20
=======================================================================
Q. The context is a Gmail account at the source, but not Gmail at the
destination. I want cross-duplicates, different Gmail folders/labels
generating duplicate messages across folders on the destination,
except for the "All Mail" folder.
R. This behavior, different Gmail labels generating duplicates across
folders except for the "All Mail" folder, is obtained with a
synchronization with no --gmail1 but instead --exclude "All Mail"
followed by another synchronization with --gmail1 but without
--exclude "All Mail".
imapsync ... --exclude "All Mail"
imapsync ... --gmail1
This second sync is done to get the messages with no labels at all
from the folder "All Mail", if they are some of them.
Technically speaking, the first synchronization has not the option
--skipcrossduplicates but the second one has the option
--skipcrossduplicates and it also performs the Gmail "All Mail"
folder/label at the end so only the messages that have no labels will
end up in the "All Mail" destination folder (which is not Gmail,
remember the context).
Thanks to Mike Alexander for this suggestion.
Mike also provided a patch to implement that directly in imapsync.
See https://github.com/imapsync/imapsync/pull/266
(I refused it because well maintaining it will be very hard for me
and I have other priorities).
=======================================================================
Q. I can't sync more than 1000 messages in each Gmail folder.
What can I do?
R. It's a Gmail configuration feature that someone set for your
Gmail account. Go to
https://mail.google.com/mail/u/0/?tab=wm#settings/fwdandpop
and select
"Do not limit the number of messages in an IMAP folder (default)"
=======================================================================
Q. What are the Gmail imap names of the folders
"Sent" "Trash" "Drafts" "Junk" "Flagged" and "All"?
R. By default, Gmail imap names of the folders
"Sent" "Trash" "Drafts" "Junk" "Flagged" and "All"
are like this:
Turned on automapping folders ( use --noautomap to turn off automapping )
Host1: special [Gmail]/All Mail = \All
Host1: special [Gmail]/Drafts = \Drafts
Host1: special [Gmail]/Sent Mail = \Sent
Host1: special [Gmail]/Spam = \Junk
Host1: special [Gmail]/Starred = \Flagged
Host1: special [Gmail]/Trash = \Trash
Host2: special [Gmail]/All Mail = \All
Host2: special [Gmail]/Bin = \Trash
Host2: special [Gmail]/Drafts = \Drafts
Host2: special [Gmail]/Sent Mail = \Sent
Host2: special [Gmail]/Spam = \Junk
Host2: special [Gmail]/Starred = \Flagged
This is an excerpt from the imapsync output log done with the
--automap option. This listing is located just before the
folders listings.
This mapping can be changed by configuration in the Gmail
web interface so the result may vary.
=======================================================================
Q. Can I transfer Gmail "Chat" messages with imapsync?
R. No.
Even if you think you can because the Chat folder is seen in IMAP,
with Chat messages in them, they won't be well transferred.
Chat messages are not well presented and not well transferred
using the IMAP protocol.
=======================================================================
Q. Can I safely use --useuid or --usecache for Gmail transfers?
R. Yes, but I suggest to not use --useuid nor --usecache
for Gmail transfers.
Using UIDs is useless with Gmail in the case of global duplicates
(duplicates across different folders). Gmail always accept a global
duplicate message as a new message, giving imapsync a new UID for this
message, and throw it away because it already has it. Gmail
will do this at each run so imapsync will always try to copy the
message, and Gmail will always accept and throw away the new copy. It
ends up with no duplicates on Gmail but a waste of bandwidth and time,
which is the opposite goal of --usecache implied by --useuid.
=======================================================================
Q. Can I use --syncduplicates Gmail transfers?
R. Yes, but Gmail will accept the new duplicate message and
throw away the new copy. It ends up with no duplicates on Gmail but
a waste of bandwidth and time.
=======================================================================
Q. Gmail does not really delete messages in folder [Gmail]/All Mail
What happens? What can I do?
R. It's true and explained in Gmail documentation at
https://support.google.com/mail/answer/7401?hl=en#zippy=%2Cempty-your-trash
To really remove messages in folder "[Gmail]/All Mail", they have to
be moved to the "Trash" folder and be deleted from "Trash".
Quoting the link above (4 february 2022)
...
Empty your Trash
If you don't want a message to stay in your trash for 30 days,
you can permanently delete it.
* On your computer, go to Gmail.
* On the left side of the page, click More.
* Scroll to Trash.
* Check the box next to messages you want to permanently delete.
Then click Delete forever.
* Tip: The "Delete forever" option is only available in the trash.
You won’t be able to delete items permanently from the Search result view.
To delete all messages in your trash, click Empty Trash now.
=======================================================================
Q. Can I use the Extension of the SEARCH command: X-GM-RAW described at
https://support.google.com/mail/answer/7190?hl=en
https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw
R. Sure. Example, to search only emails with attachment and in unread state:
On Unix:
imapsync ... --search 'X-GM-RAW "has:attachment in:unread"'
On Windows:
perl imapsync ... --search "X-GM-RAW ""has:attachment in:unread"""
=======================================================================
Q. How to avoid the [IMAP] prefix on Gmail side?
How to stop creating folder with this prefix?
Any switch we can use? e.g. [IMAP]/Archive
R. No switch in imapsync since [IMAP]/ prefix is done by Gmail,
it might be configurable within Gmail parameters configuration.
=======================================================================
Q. Does imapsync have the capability to do 2 stage authentication?
R. No, imapsync doesn't support 2 stage authentication.
Reading https://support.google.com/mail/answer/1173270?hl=en
it looks like it can't because imapsync uses imap protocol.
So you have to follow the Google recommendation and generate an
application-specific password or normal authentication
or use XOAUTH or XOAUTH2.
=======================================================================
Q. How to use XOAUTH2 to globally authenticate gmail users?
R. Yes, but really tested on Unix systems, not sure on Windows. See:
http://imapsync.lamiral.info/FAQ.d/FAQ.XOAUTH2.txt
=======================================================================
Q. How to use XOAUTH to globally authenticate gmail users?
R0. XOAUTH is considered obsolete and superseded by XOAUTH2
See http://imapsync.lamiral.info/FAQ.d/FAQ.XOAUTH2.txt
=======================================================================
Q. How to use a Gmail account to backup several different imap accounts?
R. For each account named xxx use:
imapsync ... --subfolder2 xxx/xxx
It syncs the account xxx under a sub-subfolder xxx/xxx. This way there
is no supplementary label created on the multi-archive Gmail
destination account. No labels all over the place and all original
xxx sub-folders show up nested within xxx/xxx.
=======================================================================
Q. How to migrate email from gmail to google apps?
R. Take a look at:
http://biasecurities.com/2009/02/migrate-email-from-gmail-to-google-apps/
http://www.thamtech.com/blog/2008/03/29/gmail-to-google-apps-email-migration/
=======================================================================
=======================================================================

23
FAQ.d/FAQ.GoDaddy.txt Normal file
View File

@ -0,0 +1,23 @@
$Id: FAQ.GoDaddy.txt,v 1.1 2021/05/21 18:19:29 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.GoDaddy.txt
=======================================================================
Imapsync tips for GoDaddy
=======================================================================
Questions answered in this FAQ are:
Q. What are the GoDaddy settings for imapsync?
Now the questions again with their answers.
=======================================================================
Q. What are the GoDaddy settings for imapsync?
R. The imap server is imap.secureserver.net

238
FAQ.d/FAQ.ISP.txt Normal file
View File

@ -0,0 +1,238 @@
#!/bin/cat
$Id: FAQ.ISP.txt,v 1.14 2022/05/22 08:04:05 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.ISP.txt
=======================================================================
Imapsync tips for ISP. Specific issues and solutions.
=======================================================================
* IMAPSync - usage scenario with ISP - by Flavio Zarur Lucarelli
(https://www.lucanet.com.br)
I thought I had to write a quick step by step on my attempts to learn
the imapsync features that matter the most, so it works as we expected
in the scenario in which we use it, which is to migrate customers from
their old ISP to our ISP/email hosting. Thanks to the master Gilles
Lamiral for all his help and hard work.
First of all, remember to use --dry to test things first always and
check the log file to see what would actually happen.
This type of copy keeps destination exactly like source (except, it
doesn't copy duplicates), so use it ONLY before changing your MX to
the new server, as it will delete whatever is in destination which
isn't in source.
imapsync --host1 imap.myisp.com --user1 user@domain.com --password1 pwd \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd \
--subscribeall --delete2 --delete2folders --addheader
Note: add header adds message ID when it doesn't exist.
This syntax can also be used to sync different source accounts to one
same destination account, simply execute it as many times as desired,
switching source user (user1).
If you have a list of accounts to sync, use this:
https://imapsync.lamiral.info/examples/sync_loop_unix.sh
Adjust the parameters according to the original syntax which I
explained above.
The Sync loop Unix script now outputs an error log, which can be
useful.
We also have shell scripts to check for errors, such as:
- Accounts over quota
grep -R "EXIT_OVERQUOTA" *.txt
- Login error
grep -R "Error login" *.txt
- Check just critical errors
egrep 'EXIT_AUTHENTICATION_FAILURE|EXIT_CONNECTION_FAILURE|EXIT_TLS_FAILURE|EXIT_OVERQUOTA' *.txt
- Check all sorts of errors (can be too much info)
grep 'Exiting with return value' *.txt | grep -v 'Exiting with return value 0'
Also, I like to check log file sizes and compare account sizes on
source and destination.
--
Back to it.
After you do a full sync, check that everything is consistant and
let's proceed to the final stage.
Lower the TTL (ex: 5 min) for the host associated with the MX record,
in the domain's DNS server. Let's say customer has a host mail which
his MX points to, with a high TTL (usually 1 hour). Lower it to 5 min
so that, when you change the MX, it propagates faster.
When comes time to switch over to the new host, do a final sync with
above syntax, before changing the MX. Then, change the MX and tell
your users to start using exclusively the new host.
A few hours after the MX change, we will run Imapsync again. We have
to start preserving emails users move or flags in the new host, which
they started using, so we can't do an exact sync anymore.
One simple solution is the same syntax as before, but without
--delete2 and --delete2folders.
So:
imapsync --host1 imap.myisp.com --user1 user@domain.com --password1 pwd \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd \
--subscribeall --addheader
The only issue with that syntax is, it would simply copy what is in
source that is not in destination. For example, user might have moved
emails to some sub-folder in host2. However, if you do this once, one
hour after MX change, for example, it's comprehensible that
destination is not exactly as the source anymore.
What we've been doing, especially in the case of huge migrations,
since our destination server auto-archives emails in sub-folders and
changes the hierarchy, is syncing just the last day or few days of
email and just of INBOX.
So we do the sync initially explained in this doc, before the MX
change. Then we auto-archive accounts and after auto-archiving, we
change MX and do a final sync using maxage.
Something like this:
imapsync --host1 imap.myisp.com --user1 user@domain.com --password1 pwd \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd \
--subscribeall --addheader --maxage 1
If your last full sync was 2 days ago, then use --maxage 2
Goal is to get all the new emails (2 last days) from source.
Another alternative for syncing after the MX change, which Lamiral
suggests, involves deleting emails from source from the last day. I
prefer the method explained above. Anyway, here is this other
scenario...
Something like this:
imapsync --host1 imap.gmail.com --user1 user@domain.com --password1 pwd --ssl1 \
--host2 imap.myisp.com --user2 user@domain.com --password2 pwd --ssl2 \
--folder INBOX --noexpungeaftereach --skipemptyfolders --maxage 1 --delete1
This way, any new email that arrived at source is copied to
destination and deleted from source. Without the --delete1 option,
messages moved on host2 by the user are coming back from host1, and
flags changed on host2 are also reset to what they are on host1.
However, as I wrote before, if you run it just once without --delete1,
perhaps not that bad.
*** Other tips for ISPs
1) Sync entire account into 1 folder of another account
imapsync --host1 xxx --user1 user1@domain.com --password1 secret1 --ssl1 \
--host2 yyy --user2 user2@domain.com --password2 secret2 --ssl2 \
--addheader --subfolder2 "otheraccountfolder" --delete2 --delete2foldersonly /otheraccountfolder/
Above, user must not be using such destination folder in destination
account until you finish syncing.
I've used it without --delete2 and --delete2foldersonly, as it caused
problems for us, of that folder being unsubscribed in destination...
Don't think it should be a problem as this copy is done usually
without using really using the folder in host2 until you instruct him
to.
Another nice scenario, sync just one folder hierarchy of an account to
another folder of another account.
--subfolder1 test --subfolder2 test2 --delete2 --delete2foldersonly /test2/ --addheader
Also, very useful is to be move emails older than one year, from an
account to another one, to offload the source account.
imapsync ... --subfolder2 "account_user1" --addheader --minage 365 --delete1
So everything older than 365 days is deleted from source. You can even
specify same account for source and recipient, if you want to "delete
older than x days" for the account itself, without moving old emails
to another account.
2) Translate folder names
I also ended up requiring a regex to translate folder names. On the
old server (Gmail), Sent items were in a folder called [Gmail]/E-mails
enviados and on the new server, its simply called SENT. Same with
lixeira (trash) and rascunhos (drafts).
So this was added:
--regextrans2 "s,\[Gmail\].,," \
--regextrans2 "s,E-mails enviados,Sent," \
--regextrans2 "s,Lixeira,Trash," \
--regextrans2 "s,Rascunhos,Drafts,"
Option --automap is also good to do most of the translate job.
3) Folders to exclude, to speed up syncs, etc.
imapsync ... --exclude "^Backups" --exclude "(?i)spam" --exclude "(?i)junk" --exclude "(?i)trash" --exclude "(?i)lixeira" --exclude "(?i)Deleted Items"
4) Delete older than 1 year from an account.
Use the account itself as source and destination and use:
imapsync ... --minage 365 --delete1
5) Check size of folders in an account. Use Imapsync with same account
as source and destination, with this parameter:
imapsync ... --justfoldersizes
6) If you have problems copying messages that have labels which are
not supported, there are a few ways out.
We use Dovecot and it doesn't support a series of labels, except the
standard ones. By default, imapsync won't copy messages that have
labels not supported by the recipient server. For that reason, I now
always use --filterbuggyflags and tell customers that not all labels
are copied.
There are other alternatives, such as replacing problematic flags with
other names or copying just standard flags, detailing which ones to
copy, however, --filterbuggyflags was the easiest solution, which
worked fine for us.
7) Definitely worth reading about dupes, here
https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt, if you have
problems. Always try to check, even if by sampling, if dupes
occur. In a big migration, when you do several passes, try to check a
few accounts.
The one issue we had with many duplicated emals in the destinatio, was
solved with the first solution in the above FAQ, using this syntax,
when copying from source to the destination, hence effectively
removing the dupes.
imapsync ... --useheader "Message-Id" --delete2duplicates
Note that --delete2duplicates is on when --delete2 is used, unless
--syncduplicates is used or --nodelete2duplicates is used.
=======================================================================
=======================================================================

23
FAQ.d/FAQ.IceWarp.txt Normal file
View File

@ -0,0 +1,23 @@
#!/bin/cat
$Id: FAQ.IceWarp.txt,v 1.1 2020/02/25 10:49:07 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.IceWarp.txt
=======================================================================
Imapsync tips for IceWarp accounts.
=======================================================================
=======================================================================
Q. How to migrate from or to IceWarp?
R. Use:
imapsync ... --noabletosearch
The option --noabletosearch is needed only if options --maxage or --minage
are used. I don't know which --search commands could work, maybe none.

34
FAQ.d/FAQ.James.txt Normal file
View File

@ -0,0 +1,34 @@
#!/bin/cat
$Id: FAQ.James.txt,v 1.2 2022/04/06 10:28:16 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.James.txt
=======================================================================
Imapsync tips for James. Specific issues and solutions.
=======================================================================
James: JAVA Apache Mail Enterprise Server
Questions answered in this FAQ are:
Q. How to migrate to Dovecot with an admin/MasterUser account?
=======================================================================
Q. How to migrate to James with an admin/MasterUser account?
R. James supports impersonation and is set up via the administratoId key
field stored in <install>/conf/usersrepository.conf
There are multiple user repository definitions, each using
a different key:value pair to define administratorId
https://james.apache.org/server/config-users.html
(Thanks to From Rob Davis for this Q/R)
=======================================================================
=======================================================================

46
FAQ.d/FAQ.Kerio.txt Normal file
View File

@ -0,0 +1,46 @@
#!/bin/cat
$Id: FAQ.Kerio.txt,v 1.3 2022/04/05 13:48:52 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Kerio.txt
=======================================================================
Imapsync tips for Kerio accounts.
=======================================================================
=======================================================================
Q. How to migrate from Kerio Connect to XXX
R. No special options required.
See also:
http://linux-france.tk/prj/imapsync_list/msg01756.html
http://web.archive.org/web/20140224041102/https://www.safetynet-it.com/it-support/mac-kerio-server-to-microsoft-exchange-2010-migration-1/
http://web.archive.org/web/20121125074756/http://www.safetynet-it.com/it-support/mac-kerio-server-to-microsoft-exchange-2010-migration-2/
=======================================================================
Q. How to use Kerio master authentication with imapsync?
R. Example:
Kerio at host1
imapsync ... --user1 USERNAME --password1 MASTERPASSWORD --authmech1 X-MASTERAUTH
Kerio at host2
imapsync ... --user2 USERNAME --password2 MASTERPASSWORD --authmech2 X-MASTERAUTH
See also:
https://github.com/imapsync/imapsync/pull/53
If you get the error
"Host failure: Error login on ... auth [X-MASTERAUTH]: 2 BAD AUTHENTICATE Unknown authentication method"
Then admin authentication may not be activated yet. See:
https://forums.gfi.com/index.php?t=msg&th=28458&goto=118447&#msg_118447
=======================================================================

41
FAQ.d/FAQ.Logfile.txt Normal file
View File

@ -0,0 +1,41 @@
$Id: FAQ.Logfile.txt,v 1.2 2021/07/23 12:27:31 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Logfile.txt
======================================================================
Imapsync log file
======================================================================
Questions answered in this FAQ are:
Q. I have these errors "could not be fetched", like this one:
Err 1/1: - msg INBOX/17219 {0} S[36592] F[] I[16-Jul-2021 12:04:15 +0000] could not be fetched
What can I do about it?
Now the questions again with their answers.
======================================================================
Q. I have these errors "could not be fetched", like this one:
Err 1/1: - msg INBOX/17219 {0} S[36592] F[] I[16-Jul-2021 12:04:15 +0000] could not be fetched
What can I do about it?
R. Explanation of a log line:
- msg INBOX/17219 {0} S[36592] F[] I[16-Jul-2021 12:04:15 +0000]
INBOX = folder
17219 = UID
{0} = got 0 bytes
S[36592] = should have got 36592 bytes (RFC size announced)
F[] = Flags, none here
I[16-Jul-2021 12:04:15 +0000] = Internal date
Usually this error doesn't appear on next runs, at least not for the
same messages. So rerun imapsync until all messages are synced.
======================================================================
======================================================================

37
FAQ.d/FAQ.MailEnable.txt Normal file
View File

@ -0,0 +1,37 @@
#!/bin/cat
$Id: FAQ.MailEnable.txt,v 1.2 2020/07/07 10:45:56 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.MailEnable.txt
=======================================================================
Imapsync tips for MailEnable imap server
=======================================================================
======================================================================
Q: Syncing to MailEnable 10.31 and after
R: No special imapsync option is required, default is ok.
======================================================================
Q: Syncing from MailEnable 1.75
R: Use options:
imapsync ... --sep1 "/" --prefix1 ""
======================================================================
Q: Syncing from MailEnable 2.2
R: Use options:
imapsync ... --sep1 "." --prefix1 ""
======================================================================
Q: Syncing to MailEnable until MailEnable release 10.31 not included
R: Use options:
imapsync ... --sep2 / --prefix2 "" --addheader --messageidnodomain --syncflagsaftercopy
======================================================================

358
FAQ.d/FAQ.Massive.txt Normal file
View File

@ -0,0 +1,358 @@
#!/bin/cat
$Id: FAQ.Massive.txt,v 1.34 2022/07/14 16:00:23 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Massive.txt
=======================================================================
Imapsync tips for massive/bulk migrations.
=======================================================================
Questions answered here are:
Q. How long will take the whole migration?
Q. I need to migrate hundreds of accounts, how can I do that?
Q. I have to migrate 500k users using 400 TB of disk space.
How do I proceed? How about speed?
Q. How to determine where is the bottleneck in an imapsync process?
Q. Can I run several instances of imapsync in parallel on a Windows host?
Q. I run multiple imapsync applications at the same time and then get a
warning "imapsync.pid already exists, overwriting it".
Is this a potential problem when trying to sync multiple
IMAP account in parallel?
=======================================================================
Q. How long will take the whole migration?
R1. First, you have to consider several periods. There is the global
period, from when the migration process is decided to the end,
where all mailboxes are migrated. This global period can be divided
into three smaller periods.
The first period is the analysis period: you play with the tools
available, you estimate the volume to be transferred, the number of
accounts, you measure how long it takes for one account under
your context.
The second period is what I call the presyncing period.
The users are still using
the old accounts but nothing prevents you from starting to sync
the old accounts, as they are, to the new accounts.
With tons of gigabytes to transfer, this period may be the longest one.
There is nothing more than launching the presyncs and monitoring them
until the round is finished.
The last period is the final sync period where only the last
mailboxes' changes need to be synced before switching
the users to their new mailboxes.
R2. To estimate the presyncing period, consider the mean imapsync transfer
rate to be around 340 Kbytes/s, ie, 2.8 Mbps, no matter the local
link bandwidth.
It's a mean, measured upon various syncs, coming from
the online service /X where the network card flow rate is 200 Mbps
(Rx 100 Mbps, Tx 100 Mbps) and the provider bandwidth is also 200 Mbps.
The maximum global rate seen is 22 MiB/s (176 Mbps).
At 340 Kbytes/s,
1 TB to transfer and one sync at a time will end in 35 days (1024^3/340/3600/24).
At 10 transfers at a time, 1 TB will take 3.5 days.
At 100 transfers at a time, 1 TB will take 8 hours.
Double the time because the best scenario never happens.
Triple the time because, well, the real world is like that.
R3. Another way to better evaluate the end of the presyncing period can
be based on your actual data. Just apply a simple rule of three
on the mailboxes already migrated to estimate the global end.
If it took X hours to finish Y% of the mailboxes, then it
will take 100*X/Y hours to finish 100% of the mailboxes.
Following the same idea but using mathematical garbage,
the ETA can be estimated like this:
t_0 = time of global start (the start of the first presync)
t_now = time of now.
Nb_total = total number of mailboxes to be migrated.
Nb_now = number of mailboxes already migrated.
then
ETA = t_end = (t_now - t_0) * (Nb_total / Nb_now) + t_0
R4. To estimate the last period, the final sync, just rerun a complete
presync, ie, resync all the mailboxes, the final sync should take
the same amount of time.
=======================================================================
Q. I need to migrate hundreds of accounts, how can I do that?
R. If you have many mailboxes to migrate, think about a little
script program. Write a file called file.txt (for example)
containing hosts, users, and passwords on both sides.
The separator used in this example is ";"
The file.txt file contains for example:
host001_1;user001_1;password001_1;host001_2;user001_2;password001_2;
host002_1;user002_1;password002_1;host002_2;user002_2;password002_2;
host003_1;user003_1;password003_1;host003_2;user003_2;password003_2;
host004_1;user004_1;password004_1;host004_2;user004_2;password004_2;
etc.
Most of the time, the first column (host001_1, host002_1 ...) will
contain the same value, the value of --host1 parameter. Same
thing for the third column (host001_2, host002_2).
On Unix the shell script can be:
#!/bin/sh
{ while IFS=';' read h1 u1 p1 h2 u2 p2 fake
do
imapsync --host1 "$h1" --user1 "$u1" --password1 "$p1" \
--host2 "$h2" --user2 "$u2" --password2 "$p2" "$@"
done
} < file.txt
You can add extra options inside this script, just after the variable "$@".
You can also pass extra options via the parameters of this script
since they will go in "$@"
Here is a complete Unix example ready to use:
http://imapsync.lamiral.info/examples/sync_loop_unix.sh
On Windows the batch script can be:
CD /D %~dp0
SET csvfile=file.txt
FOR /F "tokens=1,2,3,4,5,6,7 delims=; eol=#" %%G IN (%csvfile%) DO (
imapsync ^
--host1 %%G --user1 %%H --password1 %%I ^
--host2 %%J --user2 %%K --password2 %%L %%M ...
)
You can add extra options inside this script, just after the variable %%M.
You can add extra options inside the file.txt, in the last column. Add
an extra semicolon at the end (optional)
Example:
host001_1;user001_1;password001_1;host001_2;user001_2;password001_2;
host002_1;user002_1;password002_1;host002_2;user002_2;password002_2;
becomes
host001_1;user001_1;password001_1;host001_2;user001_2;password001_2; --automap --addheader
host002_1;user002_1;password002_1;host002_2;user002_2;password002_2; --automap --addheader
With this solution, options can be added, changed, or removed per account.
Technically those options will go in %%M in the loop body
Here is a complete Windows example ready to use:
http://imapsync.lamiral.info/examples/sync_loop_windows.bat
Another solution to add extra arguments is to write another .bat that
calls sync_loop_windows.bat with the extra arguments, like this
for example:
sync_loop_windows.bat --automap --addheader --maxmessagespersecond 4
Technically those options will go in %arguments% in the loop body
of sync_loop_windows.bat
=======================================================================
Q. I have to migrate 500k users using 400 TB of disk space.
How do I proceed? How about speed?
R. A good solution to this issue is two words: parallelism and measurements.
Since all mailboxes are functionally independent, they can be processed
independently, here comes the parallelism, ie, launching several imapsync
processes in parallel.
Meanwhile, mailboxes usually belong to the same server, and the syncs
share the same imapsync host via the same bandwidth, here come
some limitations and bottlenecks.
How many syncs can we run in parallel in your context?
Here comes some measurements.
1) Measure the total transfer rate by adding each one printed in each run.
Since adding this way is not so easy, just look at the overall
network rate of the imapsync host.
On Linux and FreeBSD, the command "nload" is a good candidate to measure this overall
network rate. For example, to measure the rate every 6 seconds (-t 6000),
on eth0 or em0 interface, with values in Kbytes (-u K), use:
nload -t 6000 eth0 -u K # Linux
nload -t 6000 em0 -u K # FreeBSD
On Linux only, another very good network tool is dstat:
dstat -n -N eth0 6 # Linux only (in 2018)
Another excellent tool to measure the network traffic is iftop.
The following command will monitor imap and imaps connections
on interface eth0, only them, and sum them up:
iftop -i eth0 -f 'port imap or port imaps' -B # Linux
iftop -i em0 -f 'port imap or port imaps' -B # FreeBSD
During iftop, press the h to see the display commands available,
every single feature is useful! Press h again and try each one.
My preferred display combination is by typing the keys
t p >
t means "one line per connection"
p means "show port numbers"
> means "sort by destination"
On Windows 8.1 Windows 10 Windows 2012 R2 Windows 2016,
get the overall network rate with the classical
task manager (Ctrl-Shift-Esc), there is a Performance tab
in it, where resides a Network monitor.
On Windows 7, get the overall network rate with the classical
task manager (Ctrl-Shift-Esc), there is a Network tab in it.
I'm looking for a free and simple tool on Windows that could
sum up only the imap traffic.
2) Launch new parallel runs, one by one, as long as the total
transfer rate increase.
3) When the total transfer rate starts to diminish, stop new launches.
Note N as the number of runs in parallel you got until then.
4) Only keep N-2 parallel runs for the future.
=======================================================================
Q. How to determine where is the bottleneck in an imapsync process?
R1. Divide and conquer.
To detect whether host1/link1 is the bottleneck or
host2/link2, we have several tests to explore:
1) run a sync from host1 to host1, with a host1 test account as the destination.
This way, only host1 and link1 are tested, host2 is not directly concerned.
If performances increase a lot then host2/link2 is the bottleneck.
2) run a sync from host2 to host2, with a host2 test account as the destination.
This way, only host2 and link2 are tested, host1 is not concerned.
If performances increase a lot then host1/link1 is the bottleneck.
If performances increase on both tests 1) and 2), I have no clue to explain that.
Same thing if they both decrease!
R2. Isolating and overcoming bottlenecks
In any process involving several mechanisms, among all elements taking
part in the process, there is always a bottleneck. No one knows in
advance what is the first bottleneck. The first bottleneck has to be
determined, by measurements, not by guesses. Once this first
bottleneck is known and overcome then the next bottleneck has to be
determined and overcome too, if needed. Repeat the process of looking
for the next bottleneck and its elimination until you estimate the
transfer rates, money costs, time spent on this, and final dates
are good enough to proceed with the whole huge migration.
Possible bottlenecks:
- Throttles.
IMAP servers have artificial limits.
For example, Gmail, Office365, and Exchange have throttle limits.
- Bandwidth.
Usually, the available bandwidth is not a bottleneck.
Meanwhile, it can be a bottleneck on small Internet connections.
Imapsync downloads messages from host1 and upload messages to host2,
consider this in case the connection is asymmetric.
- I/O, aka "Input/Output" on the disks of the imap servers.
The I/O on disks are a classical bottleneck, almost always forgotten.
Unlike CPU and RAM, Input/Output performances don't improve
very much as time goes on so it's often a bottleneck.
To measure and overcome an I/O disk bottleneck, you need
usually direct access to host1 and host2.
An I/O bottleneck where imapsync runs is possible if
--usecache or --useuid is used or with very big messages.
- RAM.
On all sides, monitor that your systems don't swap its
running processes on disk, because swapping running processes
on disks decreases performance by a factor of 20, at least.
It's not because the swap memory is used that your
system swaps processes on disk.
- CPU.
100% CPU during a whole transfer means the system is busy.
CPU can be a problem with imapsync but it can also be a problem
with one or both of the imap servers.
Other possible bottlenecks:
- Number of hosts available to run imapsync processes.
- Imapsync itself.
- Management of errors.
- MX domains, DNS.
- Incompetence.
- Money.
- Time.
- Bad luck.
- ...
=======================================================================
Q. Can I run several instances of imapsync in parallel on a Windows host?
R. Yes!
Q. Any performance issues?
You have to try and check the transfer rates, sum them up to
have a unique numeric criterion.
There is always a limit, depending on remote imap servers
and the one running imapsync.
CPU, memory, Inputs/Outputs are the classical bottlenecks,
the worst bottleneck is the winner that sets the limit.
examples/sync_loop_windows.bat says
...
REM ==== Parallel executions ====
REM If you want to do parallel runs of imapsync then this current script is a good start.
REM Just copy it several times and replace, on each copy, the csvfile variable value.
REM Instead of SET csvfile=file.txt write for example
REM SET csvfile=file01.txt in the first copy
REM then also
REM SET csvfile=file02.txt in the second copy etc.
REM Of course you also have to split the data contained in file.txt
REM into file01.txt file02.txt etc.
REM After that, just double-click on each batch file to launch each process
=======================================================================
Q. I run multiple imapsync applications at the same time and then get a
warning "imapsync.pid already exists, overwriting it".
Is this a potential problem when trying to sync multiple
IMAP account in parallel?
R1. No issue with the file imapsync.pid if you don't use its content
by yourself.
This file can help you to manage multiple runs by sending signals
to the processes (sigterm or sigkill) using their PID.
Each run can have its pid file with --pidfile option.
The file imapsync.pid contains the PID of the current imapsync process.
This file is removed at the end of a normal run.
You can safely ignore the warning if you don't use imapsync.pid file
to manage imapsync processes.
=======================================================================
=======================================================================

69
FAQ.d/FAQ.Memory.txt Normal file
View File

@ -0,0 +1,69 @@
#!/bin/cat
$Id: FAQ.Memory.txt,v 1.8 2018/08/31 13:29:53 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Memory.txt
=======================================================================
Imapsync tips about memory issues.
=======================================================================
=======================================================================
Q. I got an "Out of memory" error message then imapsync crashes.
How to fix that?
R0. Add option --debugmemory to debug memory issues.
On Windows this option uses a DOS command similar to:
tasklist /NH /FO CSV | findstr imapsync
On Unix it uses:
ps -o vsz -p PID_of_imapsync # value is in KB
Similar to:
ps -ax -o vsz,comm | grep imapsync
Use those commands in a DOS window or a Unix shell in order
to monitor a imapsync run you know it will crash.
The goal is to detect the memory limit, when and how it happens.
* Is the "final" amount near the total RAM available or
just a small part of it?
* Does the rise of memory consumption happens suddenly
or along the sync?
R1. It's probably a very big message on the host1 account.
Before imapsync release 1.688 memory usage is at least 5 times
the biggest message size. then 2 times with release 1.688.
So a 700 MB message will crunch 3.5 GB of memory.
Starting with 1.688 it will crunch 1.4 GB of memory.
R2. To check if the problem is a big message crunching to much
memory, add --maxsize 50_000_000, this will limit the message
size treated to 50 MB. If the sync succeed while limiting
messages sizes it's probably is a big message crunching to much
memory issue. Possible solutions:
* Try on another host
* Buy memory
* Build imapsync.exe binary 64bits
R3. If R2 shows that the memory issue is not a big message crunching
too much memory, then it can be imapsync using too much memory
to identify messages (a very big folder served by a
very talkative imap server). A possible solution is to use
the option --useuid. If it still fails, I have no clue.
If you have time, drop me a note.
R4. Usually "Out of memory" errors are related to old days,
to old Mail::IMAPClient module releases, before 3.26.
At that time memory usage was around 17 times the biggest message.
Look at imapsync output first lines to get the Mail::IMAPClient
release used. Then upgrade Mail::IMAPClient Perl module if needed.
=======================================================================
=======================================================================

View File

@ -0,0 +1,476 @@
#!/bin/cat
$Id: FAQ.Messages_Selection.txt,v 1.18 2021/07/04 23:05:24 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt
=======================================================================
Imapsync tips to select messages.
=======================================================================
Questions answered in this FAQ are:
Q. What messages imapsync syncs by default?
Q. Is there a way we can specify a date range to sync emails?
If yes, can you please share an example?
Q. Is there a way we can specify an age to sync emails?
If yes, can you please share some examples?
Q. I want to sync messages based on their UID.
Q. Can I migrate only mails with attachments?
Q. How can I move messages marked \Deleted from all folders to
a dedicated folder?
Q. What are the selection criteria available with --search option?
=======================================================================
Q. What messages imapsync syncs by default?
R1. By default, Imapsync syncs all messages, except duplicates.
If you want to sync duplicates, add the option --syncduplicates
R2. You can alter the default "all messages selected" by using one or
more of the following options:
Extract from the README https://imapsync.lamiral.info/README
OPTIONS/message selection
--maxsize int : Skip messages larger (or equal) than int bytes
--minsize int : Skip messages smaller (or equal) than int bytes
--maxage int : Skip messages older than int days.
final stats (skipped) don't count older messages
see also --minage
--minage int : Skip messages newer than int days.
final stats (skipped) don't count newer messages
You can do (+ zone are the messages selected):
past|----maxage+++++++++++++++>now
past|+++++++++++++++minage---->now
past|----maxage+++++minage---->now (intersection)
past|++++minage-----maxage++++>now (union)
--search str : Selects only messages returned by this IMAP SEARCH
command. Applied on both sides.
For a complete set of what can be search see
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt
--search1 str : Same as --search but for selecting host1 messages only.
--search2 str : Same as --search but for selecting host2 messages only.
So --search CRIT equals --search1 CRIT --search2 CRIT
--maxlinelength int : skip messages with a line length longer than int bytes.
RFC 2822 says it must be no more than 1000 bytes but
real life servers and email clients do more.
--useheader str : Use this header to compare messages on both sides.
Ex: Message-ID or Subject or Date.
--useheader str and this one, etc.
--usecache : Use cache to speed up next syncs. Not set by default.
--nousecache : Do not use cache. Caveat: --useuid --nousecache creates
duplicates on multiple runs.
--useuid : Use UIDs instead of headers as a criterion to recognize
messages. Option --usecache is then implied unless
--nousecache is used.
=======================================================================
Q. Is there a way we can specify a date range to sync emails?
If yes, can you please share an example?
R. Yes, a date range is possible with the --search option.
imapsync ... --search "SENTSINCE 1-Jan-2010"
or
imapsync ... --search "SENTBEFORE 31-Dec-2010"
or
imapsync ... --search "SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010"
Months are specified like this:
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
=======================================================================
Q. Is there a way we can specify an age to sync emails?
If yes, can you please share some examples?
R. Yes, with the --maxage or the --minage option.
Have in mind that the options --maxage and --minage can NOT be
combined successfully with the --search command. If the --search
option is present on the command line then --maxage and --minage
are ignored.
Example.1 Sync only messages less than 2 days old:
imapsync ... --maxage 2
Example.2 Sync only messages more than 2 days old:
imapsync ... --minage 2
Example.3 Sync only messages more than 30 days old and less than 365 days old:
imapsync ... --minage 30 --maxage 365
Example.4 Sync only messages less than 30 days old or more than 365 days old:
imapsync ... --maxage 30 --minage 365
Full explanation:
--maxage <int> : Skip messages older than <int> days.
final stats (skipped) don't count older messages
see also --minage
--minage <int> : Skip messages newer than <int> days.
final stats (skipped) don't count newer messages
You can do (+ are the messages selected):
past|----maxage+++++++++++++++>now
past|+++++++++++++++minage---->now
past|----maxage+++++minage---->now (intersection)
past|++++minage-----maxage++++>now (union)
Caveat.1 By default,
Option --maxage is implemented as a --search SENTSINCE
RFC 3501 says: SENTSINCE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within or later than the specified date.
Option --minage is implemented as a --search SENTBEFORE
RFC 3501 says: SENTBEFORE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is earlier than the specified date.
If --noabletosearch is on then --minage and --maxage deal
with the internal dates given by a FETCH imap command but
not the Date: header. Internal date is the arrival date
in the mailbox. Same remark for --noabletosearch1 and
--noabletosearch2 but only for one side then.
=======================================================================
Q. I want to sync messages based on their UID.
R. First have in mind that UIDs are unique only per folder, so work this
way only with one folder at a time, with --folder option.
To show UIDs, there is the --debugLIST parameter.
imapsync ... --debugLIST
To sync only a part of all messages, selected by UIDs
from 10000 to 11000:
imapsync ... --search1 "UID 10000:11000"
To sync from INBOX only 3 messages UIDs 20000 20002 20004:
imapsync ... --search1 'OR OR UID 20000 UID 20002 UID 20004' --folder INBOX
To sync all messages from INBOX except 3 messages
UIDs 20000 20002 20004:
imapsync ... --search1 'NOT OR OR UID 20000 UID 20002 UID 20004' --folder INBOX
If you search n UIDs then you have to put n-1 OR in the search line,
that's IMAP.
=======================================================================
Q. Can I migrate only mails with attachments?
R. Use:
imapsync ... --search "HEADER Content-Disposition attachment"
or
imapsync ... --search "HEADER Content-Type multipart/mixed"
or more generally:
imapsync ... --search "OR HEADER Content-Disposition attachment HEADER Content-Type multipart/mixed"
The last one doesn't work on all imap servers, not on Gmail at least (tested 2020/06/16)
=======================================================================
Q. How can I move messages marked \Deleted from all folders to
a dedicated folder?
R. To move \Deleted messages from all folders to a specific folder,
let's call it Trash, use:
imapsync ... --search DELETED --regextrans2 "s/.*/Trash/"
=======================================================================
Q. What are the selection criteria available with --search option?
R. The list of search criteria are listed below, an excerpt from RFC3501.
https://tools.ietf.org/html/rfc3501#section-6.4.4
...
The SEARCH command searches the mailbox for messages that match
the given searching criteria. Searching criteria consist of one
or more search keys. The untagged SEARCH response from the server
contains a listing of message sequence numbers corresponding to
those messages that match the searching criteria.
When multiple keys are specified, the result is the intersection
(AND function) of all the messages that match those keys. For
example, the criteria DELETED FROM "SMITH" SINCE 1-Feb-1994 refers
to all deleted messages from Smith that were placed in the mailbox
since February 1, 1994. A search key can also be a parenthesized
list of one or more search keys (e.g., for use with the OR and NOT
keys).
In all search keys that use strings, a message matches the key if
the string is a substring of the field. The matching is
case-insensitive.
The defined search keys are as follows. Refer to the Formal
Syntax section for the precise syntactic definitions of the
arguments.
<sequence set>
Messages with message sequence numbers corresponding to the
specified message sequence number set.
ALL
All messages in the mailbox; the default initial key for
ANDing.
ANSWERED
Messages with the \Answered flag set.
BCC <string>
Messages that contain the specified string in the envelope
structure's BCC field.
BEFORE <date>
Messages whose internal date (disregarding time and timezone)
is earlier than the specified date.
BODY <string>
Messages that contain the specified string in the body of the
message.
CC <string>
Messages that contain the specified string in the envelope
structure's CC field.
DELETED
Messages with the \Deleted flag set.
DRAFT
Messages with the \Draft flag set.
FLAGGED
Messages with the \Flagged flag set.
FROM <string>
Messages that contain the specified string in the envelope
structure's FROM field.
HEADER <field-name> <string>
Messages that have a header with the specified field-name (as
defined in [RFC-2822]) and that contains the specified string
in the text of the header (what comes after the colon). If the
string to search is zero-length, this matches all messages that
have a header line with the specified field-name regardless of
the contents.
KEYWORD <flag>
Messages with the specified keyword flag set.
LARGER <n>
Messages with an [RFC-2822] size larger than the specified
number of octets.
NEW
Messages that have the \Recent flag set but not the \Seen flag.
This is functionally equivalent to "(RECENT UNSEEN)".
NOT <search-key>
Messages that do not match the specified search key.
OLD
Messages that do not have the \Recent flag set. This is
functionally equivalent to "NOT RECENT" (as opposed to "NOT
NEW").
ON <date>
Messages whose internal date (disregarding time and timezone)
is within the specified date.
OR <search-key1> <search-key2>
Messages that match either search key.
RECENT
Messages that have the \Recent flag set.
SEEN
Messages that have the \Seen flag set.
SENTBEFORE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is earlier than the specified date.
SENTON <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within the specified date.
SENTSINCE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within or later than the specified date.
SINCE <date>
Messages whose internal date (disregarding time and timezone)
is within or later than the specified date.
SMALLER <n>
Messages with an [RFC-2822] size smaller than the specified
number of octets.
SUBJECT <string>
Messages that contain the specified string in the envelope
structure's SUBJECT field.
TEXT <string>
Messages that contain the specified string in the header or
body of the message.
TO <string>
Messages that contain the specified string in the envelope
structure's TO field.
UID <sequence set>
Messages with unique identifiers corresponding to the specified
unique identifier set. Sequence set ranges are permitted.
UNANSWERED
Messages that do not have the \Answered flag set.
UNDELETED
Messages that do not have the \Deleted flag set.
UNDRAFT
Messages that do not have the \Draft flag set.
UNFLAGGED
Messages that do not have the \Flagged flag set.
UNKEYWORD <flag>
Messages that do not have the specified keyword flag set.
UNSEEN
Messages that do not have the \Seen flag set.
...
search = "SEARCH" [SP "CHARSET" SP astring] 1*(SP search-key)
; CHARSET argument to MUST be registered with IANA
...
search-key = "ALL" / "ANSWERED" / "BCC" SP astring /
"BEFORE" SP date / "BODY" SP astring /
"CC" SP astring / "DELETED" / "FLAGGED" /
"FROM" SP astring / "KEYWORD" SP flag-keyword /
"NEW" / "OLD" / "ON" SP date / "RECENT" / "SEEN" /
"SINCE" SP date / "SUBJECT" SP astring /
"TEXT" SP astring / "TO" SP astring /
"UNANSWERED" / "UNDELETED" / "UNFLAGGED" /
"UNKEYWORD" SP flag-keyword / "UNSEEN" /
; Above this line were in [IMAP2]
"DRAFT" / "HEADER" SP header-fld-name SP astring /
"LARGER" SP number / "NOT" SP search-key /
"OR" SP search-key SP search-key /
"SENTBEFORE" SP date / "SENTON" SP date /
"SENTSINCE" SP date / "SMALLER" SP number /
"UID" SP sequence-set / "UNDRAFT" / sequence-set /
"(" search-key *(SP search-key) ")"
seq-number = nz-number / "*"
; message sequence number (COPY, FETCH, STORE
; commands) or unique identifier (UID COPY,
; UID FETCH, UID STORE commands).
; * represents the largest number in use. In
; the case of message sequence numbers, it is
; the number of messages in a non-empty mailbox.
; In the case of unique identifiers, it is the
; unique identifier of the last message in the
; mailbox or, if the mailbox is empty, the
; mailbox's current UIDNEXT value.
; The server should respond with a tagged BAD
; response to a command that uses a message
; sequence number greater than the number of
; messages in the selected mailbox. This
; includes "*" if the selected mailbox is empty.
seq-range = seq-number ":" seq-number
; two seq-number values and all values between
; these two regardless of order.
; Example: 2:4 and 4:2 are equivalent and indicate
; values 2, 3, and 4.
; Example: a unique identifier sequence range of
; 3291:* includes the UID of the last message in
; the mailbox, even if that value is less than 3291.
sequence-set = (seq-number / seq-range) *("," sequence-set)
; set of seq-number values, regardless of order.
; Servers MAY coalesce overlaps and/or execute the
; sequence in any order.
; Example: a message sequence number set of
; 2,4:7,9,12:* for a mailbox with 15 messages is
; equivalent to 2,4,5,6,7,9,12,13,14,15
; Example: a message sequence number set of *:4,5:7
; for a mailbox with 10 messages is equivalent to
; 10,9,8,7,6,5,4,5,6,7 and MAY be reordered and
; overlap coalesced to be 4,5,6,7,8,9,10.
See also the excellent "Introduction to IMAP" by Nick Babcock
https://nbsoftsolutions.com/blog/introduction-to-imap
See also "WITHIN Search Extension to the IMAP Protocol"
https://tools.ietf.org/html/rfc5032
...
search-key =/ ( "OLDER" / "YOUNGER" ) SP nz-number
; search-key defined in RFC 3501
4. Example
C: a1 SEARCH UNSEEN YOUNGER 259200
S: a1 * SEARCH 4 8 15 16 23 42
Search for all unseen messages within the past 3 days, or 259200
seconds, according to the server's current time.
=======================================================================
=======================================================================

View File

@ -0,0 +1,139 @@
#!/bin/cat
$Id: FAQ.Messages_Too_Big.txt,v 1.8 2021/12/16 11:44:26 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Too_Big.txt
=======================================================================
Dealing with too big messages with Imapsync.
=======================================================================
Questions answered in this FAQ are:
Q. Imapsync reports errors like these ones:
"could not append...Maximum size of appendable message has been exceeded"
"could not append...maximum message size exceeded"
"The most frequent error is ERR_APPEND_SIZE"
What does it mean?
Q. Does imapsync limits big messages?
Q. What are the well known servers messages size limits?
Q. What can I do to transfer messages bigger than what allows the
imap destination server?
Q. The solution --truncmess xxxx doesn't work with Gmail.
What can I do?
Now the questions again with their answers.
=======================================================================
Q. Imapsync reports errors like these ones:
"could not append...Maximum size of appendable message has been exceeded"
"could not append...maximum message size exceeded"
"The most frequent error is ERR_APPEND_SIZE"
What does it mean?
R. It means the message couldn't be transferred to the destination
account because it is too big, imapsync tries to APPEND it to the
server (that's IMAP jargon) but the server refuses it by saying
either "NO. Maximum size of appendable message has been exceeded"
or "NO. maximum message size exceeded".
=======================================================================
Q. Does imapsync limits big messages?
R. No. Imapsync synchronizes messages no matter their sizes.
When a message is copied to the destination server, the destination
server accepts it or not. It does it as a whole, it takes the full
message or nothing. The destination server can refuse a too big
message, it then triggers an APPEND error message in imapsync.
So far, December 2021, the biggest message transferred by imapsync on
the online /X service is 1.51 GiB. Who can top that?
=======================================================================
Q. What are the well known servers messages size limits?
* Gmail message size limit is 35 MB.
Gmail announces APPENDLIMIT=35651584 (June 2021).
* Office365 message size limit is 45 MB by default
and it can be upped by configuration. See
https://imapsync.lamiral.info/FAQ.d/FAQ.Office365.txt
Q. I see "NO Maximum size of appendable message has been exceeded"
* Exchange message size limit is 10 MB by default
and it can be upped by configuration. See
https://imapsync.lamiral.info/FAQ.d/FAQ.Exchange.txt
Q. I see "NO Maximum size of appendable message has been exceeded"
* Dovecot message size limit is variable. It is defined
by the sysadmin. For example, quota_max_mail_size = 100M
https://doc.dovecot.org/configuration_manual/quota/
* Usually, any IMAP server has a message size limit, defined
by the sysadmin. You discover it sooner or later
=======================================================================
Q. What can I do to transfer messages bigger than what allows the
imap destination server?
R1. A solution is ask imapsync to truncate the message to the maximum
size allowed before copying it to host2 by using the option
--truncmess.
That's a little dirty but it's ok if you prefer having a truncated
message than no message at all. The attachments after the truncation
won't be available but everything before should be ok. It may also
depend on the software tool reading the email message.
To truncate messages to 10 MB, use:
imapsync ... --truncmess 10_000_000
R2. A not-so-dirty solution would be to transform the big messages,
replace the attachments with links to the attachments stored
elsewhere. Well, it's not done and it's too much complicated compared
to a simpler solution which is to allow bigger messages on the
destination account.
As time goes on, disk space is cheaper and cheaper. Don't make the
new user account, the destination account, worse than their previous
account: buy disks.
Monitor and rule your hardware resources, serve your users,
not the contrary.
=======================================================================
Q. The solution --truncmess xxxx doesn't work with Gmail.
What can I do?
R. Gmail currently limits messages size up to 35651584 bytes, aka 34
MiB (June 2021) but imapsync knows it so it avoid to transfer these
big messages. Solution, use:
imapsync ... --truncmess 35_651_584 --appendlimit 1_000_000_000_000
Explanation. By default, imapsync checks if the host2 imap server
announces a message size limit with "APPENDLIMIT=xxxx" in its
CAPABILITY response, like Gmail does. Imapsync takes this APPENDLIMIT
value to set automatically the --maxsize option, unless --maxsize is
already set on the command line and has a smaller value than the
APPENDLIMIT value.
1_000_000_000_000 is one TB, I guess you don't have any message of
that size yet.
=======================================================================
=======================================================================

View File

@ -0,0 +1,163 @@
#!/bin/cat
$Id: FAQ.Migration_Plan.txt,v 1.9 2021/12/19 10:00:32 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Migration_Plan.txt
=====================================================================
Imapsync. Suggestions for a good, low impact on users,
well-executed email migration plan.
=====================================================================
There are two main different scenarios. Choosing which one fits your
context depends on the response to the following question:
Will the imap software tools used by the users use the same
credentials triplet for both imap servers, the old server host1 and
the new server host2?
The credentials triplet is hostname/username/password.
If the answer is yes, ie, clients' email tools use the same triplet
credentials, then it is possible to perform a migration without
changing anything on the users' side. This may be a very time-saving
option. But it's a rare condition so I'll describe this scenario later
in this document.
=====================================================================
Classical scenario, credentials triplets are different on both sides
=====================================================================
* Decrease the TTL of the MX, to 5 minutes (or even less). See
FAQ.TTL.txt to understand why it's an advantage. If you can't
decrease the TTL, the migration will span a little more but that's
ok, the situation is not that bad.
* Create the new mailboxes on the destination server host2. If the
users are already playing with the new mailboxes on host2, don't
follow this scenario.
* Pre-synchronize all the mailboxes from the old server host1 to the
new server host2. If an imap server name is going to change its IP
address, then don't use this name, use a name that will always
match the same imap servers, or use their IP addresses.
Pre-synchronizations can usefully be done with --delete2 to get an
exact synchronization. But never use the option --delete2 once the
users have started to play with their new account on host2, their
play will be lost on the next synchronization. Don't use --delete2
either when the MX is changed since INBOX will start to receive new
messages that are not on host1 and then removing them is not a good
idea.
* Decide a migration day/hour.
* Repeat the pre-synchronizations (with the --delete2 options) daily
until the migration hour. This repeated process will show how long
should take the last synchronization.
* At the migration hour, cut access to the users to the old server
host1, if you can. Or tell them to not use it anymore.
* Do the last pre-synchronization exactly like the previous ones.
* Change the MX, the new messages should start to arrive in the new
imap server host2.
* Wait for the TTL value, aka 5 minutes. Now, new messages should
not arrive at the old server host1.
* Tell the users that the old imap server host1 is down and no
longer available.
* Do a post-synchronization. A post-synchronization is a run with the
following options: --folder INBOX --delete1 --maxage 1
This post-synchronization will copy the messages arrived in the
last day (--maxage 1) in the folder INBOX (--folder INBOX) on the
source account, to the destination account. It will also delete
them on host1 (--delete1). It's --delete1, it's not --delete2.
Remember, do not use the option --delete2 in a post
synchronization, as users won't appreciate seeing their newly
arrived messages disappear because of you.
* Give access to new accounts to the users with their new credential
triplet hostname/username/password. If the way to contact users is
by email then you should give them the new credentials long before
shutting down the old server.
* Migration done.
* In case there are still messages arriving at the old imap server
host1, you can perform more post-synchronizations, ie, runs every
day with the options: --maxage 1 --delete1 --folder INBOX
* Increase the TTL of the MX back to its previous value, usually
24 hours, 86400 seconds. You don't want all your email system
to break down completely when your DNS are not available
temporarily, keeping dns values in cache for a 24h is a savvy
practice.
=====================================================================
Lucky scenario, credentials triplets are the same on both sides
=====================================================================
* Decrease the TTL of the MX, as well as the imap hostname resolution,
to 5 minutes (or even less). The document FAQ.TTL.txt explains why.
* Create the new mailboxes on the destination server host2.
* Pre-synchronize all the mailboxes from the old host1 to the new
server host2, using different names than the ones used by the imap
software clients (use their IP for example). Presyncs have to be
done with --delete2 but never use --delete2 once users have started
playing with their new account on host2.
* Decide a migration day/hour.
* Repeat the pre-synchronizations (the runs with the --delete2
options) daily until the migration hour. This repeated process will
show how long should take the last sync.
* At the migration hour, cut access to the users to the old server.
You can do this by changing the imap host1 hostname to a non-imap
server for example, or by changing their password on host1.
* Do the last run exactly like the pre-synchronizations.
* Change also the MX resolution, the new messages should start
to arrive in the new imap server very soon.
* Wait for the TTL value, aka 5 minutes. Now, new messages should
not arrive at the old server host1.
* Do a post-synchronization. A post-synchronization is a run with the
following options: --folder INBOX --delete1 --maxage 1
This post-synchronization will copy the messages arrived in the
last day (--maxage 1) in the folder INBOX (--folder INBOX) on the
source account, to the destination account. It will also delete
them on host1 (--delete1). It's --delete1, it's not --delete2.
Remember, do not use the option --delete2 in a post
synchronization, as users won't appreciate seeing their newly
arrived messages disappear because of you.
* Shut down the old imap server.
* Change the user imap hostname resolution from the old IP of host1
to the IP of the new imap server host2.
* Migration done.
* Increase the TTL of the MX back to its previous value, usually
24 hours, 86400 seconds. You don't want all your email system
to break down completely when your DNS are not available
temporarily, keeping dns values in cache for a 24h is a savvy
practice.
=======================================================================
=======================================================================

686
FAQ.d/FAQ.Office365.txt Normal file
View File

@ -0,0 +1,686 @@
$Id: FAQ.Office365.txt,v 1.38 2022/06/22 17:07:35 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Office365.txt
======================================================================
Imapsync tips for Office365.
======================================================================
Questions answered in this FAQ are:
Q. Can I use imapsync to transfer from or to Office365 accounts?
Q. Can imapsync work for users that are administrators for
an Office 365 domain? (quick answer: no)
Q. Does imapsync support OAUTH2 authentication for Office365 accounts?
Q. How to sync from Office365 to XXX?
Q. How to sync from XXX to Office365
Q. For Office365 I have double and triple-checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
Q. How can I access an Office365 shared mailbox?
Q. How can I use a shared account as a backup account for several
mailboxes and so avoid spending too many dollars in backup accounts?
Q. Office365 fails with "User is authenticated but not connected".
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
Q. Every single mail synced to exchange online owns the
category $MDNSent after migration. How can I avoid this?
Q. The imap connection to Office365 is not working very efficiently,
is there a solution to fix that?
Q. Office365 throttles the sync and says:
"Request is throttled. Suggested Backoff Time: 299961 milliseconds".
What can I do with that?
Q. What are the receive and sending limits of Office365?
Q. The sync fails with many "Trying command when NOT connected!".
What can I do?
Q. How to see or migrate public folders on Office365?
R. https://docs.microsoft.com/en-us/exchange/collaboration/public-folders/migrate-to-exchange-online?view=exchserver-2019
https://www.exchangesavvy.com/moving-your-public-folders-to-office-365-what-you-need-to-know/
Q. Office365 refuses to create the folder named "Files" with the error
"NO Folder name is reserved". What happens?
Q. Office365 users complain that a folder named "Files" contains
messages with no sender.
Q. From XXX to Office365, read receipts are all resent again after sync.
Even for old messages. How can I fix that?
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
Q. From XXX Office365 I get this error message sometimes:
"BAD Command Argument Error 11". What does it mean?
Q. From XXX to Office365 the flag Flagged does not seem to be well synced.
What can I do?
Q. How to migrate from or to Office 365 with an admin/authuser account?
Q. Couldn't create folder [trash] "Mailbox already exists".
Now the questions again with their answers.
======================================================================
Q. Can I use imapsync to transfer from or to Office365 accounts?
R0. Yes. But IMAP access to Office365 account is not always allowed
by default so it has to be allowed in the server configuration part.
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365
R1. Imapsync doesn't support OAUTH2 authentication for Office365
accounts yet. That's not totally true, if you have an access token
to access your mailbox then you can authenticate with imapsync this
way:
imapsync ... --oauthaccesstoken1 tokenfile
where "tokenfile" is a file containing the access token.
R2. Enable double-step authentication and configure it but after use
an "app password" with imapsync. Details:
a) Go to
https://account.microsoft.com/security
b) Click on "Advanced Security Options"
Turn "Two-step verification" on. Follow the steps and finish".
c) Then now "App passwords" is available.
Click on "Create a new app password".
Use this password to authenticate with imapsync.
d) Delete this app password when the job with imapsync is finished.
R3. Also, check a license is assigned to that account in Office365.
R4. From Dave Pusey
https://github.com/imapsync/imapsync/issues/317#issuecomment-1027776418
I quote Dave nearly verbatim:
"I had created an app password, and security defaults are already off.
I have now figured out the issue.
Despite the MS365 and EXO admin centers showing that IMAP and Basic
Auth were all enabled, it turns out that in Oct 2021, Microsoft began
disabling basic auth for all tenants that had have never used it by
that point. There was an item in my Message Center from that date
saying my tenant was being done.
You can re-enable it for specific protocols (IMAP in this case) using
the diagnostic process detailed at
https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-september-2021-update/bc-p/2782230
(see the section "Limited Opt Out")
R5. Microsoft introduced something called "security defaults" which is enabled
by default for new tenants. One of the rules blocks IMAP access as of imapsync.
The funny thing is that you can't disable a single rule of this
security package without buying additional licenses.
Switching the whole thing off allows the IMAP login.
Also, disable double-step authentication on the Azure/Active Directory portal.
See here:
https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults
Thanks to Stephan Buhre for this R5 answer.
R6. Are there special characters in the password?
https://imapsync.lamiral.info/FAQ.d/FAQ.Authentication_failure.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Mac.txt
R7. Triple check the hostname then. Try all of these:
* outlook.office365.com
* imap-mail.outlook.com
* imap.outlook.com
======================================================================
Q. Can imapsync work for users that are administrators for
an Office 365 domain? (quick answer: no)
R. It doesn't seem possible to use imap for administrators, so imapsync
won't be able to work for an administrator's mailbox.
See:
https://github.com/imapsync/imapsync/issues/310#issuecomment-1002396218
https://exhaust.lewiscollard.com/post/146866104/office365-to-migadu-migration/
Solution: as explained in the article above, use Davmail as a proxy to
access this mailbox. http://davmail.sourceforge.net/
Thanks to Lewis Collard for this report and solution.
======================================================================
Q. Does imapsync support OAUTH2 authentication for Office365 accounts?
R. Yes but partially. Imapsync won't help you getting an access token
but if you have one then you can use it with imapsync this way:
Office365 as source:
imapsync ... --oauthaccesstoken1 tokenfile1
Office365 as destination:
imapsync ... --oauthaccesstoken2 tokenfile2
======================================================================
Q. How to sync from Office365 to XXX?
R0. IMAP is not enabled by default on Office365, see how to enable it:
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365
If IMAP cannot be enabled or if it doesn't work well, you can try DavMail
(http://davmail.sourceforge.net/) which acts as an IMAP gateway
through Outlook Web Access.
R. On Windows, use:
imapsync.exe ... --office1
On Unix, use:
imapsync ... --office1
Option --office1 is like (release 1.970 or higher):
imapsync ... --host1 outlook.office365.com \
--ssl1 \
--exclude "^Files$"
======================================================================
Q. How to sync from XXX to Office365
R0. IMAP is not enabled by default on Office365, see how to enable it:
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365
If IMAP cannot be enabled or if it doesn't work well, you can try DavMail
(http://davmail.sourceforge.net/) which acts as an IMAP gateway
through Outlook Web Access.
R. Here is a command-line summary that solves most encountered issues when
migrating to Office365.
imapsync ... --office2
which is equivalent to (in imapsync release 1.870 or higher):
imapsync ... \
--host2 outlook.office365.com \
--ssl2 \
--maxsize 45000000 \
--maxmessagespersecond 4 \
--disarmreadreceipts \
--regexmess "s,(.{10239}),$1\r\n,g" \
--f1f2 "Files=Files_renamed_by_imapsync"
On Linux, you can also try the "reformime" command
that can be used like:
imapsync ... --maxlinelengthcmd "reformime -r7"
To get reformime, install the "maildrop" package.
On Linux again, there is a good Python script in the tarball that can
fix several things that Exchange or O365 have issues with.
Use it like this:
./imapsync ... --pipemess W/tools/fix_email_for_exchange.py
It often does some miracles on messages.
======================================================================
Q. For Office365 I have double and triple-checked the username and
password spelling but I still get a "LOGIN failed". Any clue?
R. Go to "Q. Can I use imapsync to transfer from or to Office365 accounts?"
======================================================================
Q. How can I access an Office365 shared mailbox?
R. First, create a shared mailbox, for example shared@example.com.
Then give full permissions to a licensed account user@example.com.
Now with imapsync, use the licensed user login with the syntax:
user@example.com\shared@example.com
and the password for user@example.com
Caveat: Character \ is a special character, so
use double-quotes around the user login, like:
imapsync ... --user1 "user@example.com\shared@example.com"
Sources:
https://adam-hand.com/2017/07/25/connect-a-shared-mailbox-from-o365-to-outlook-via-imap/
https://www.arclab.com/en/kb/email/imap-settings-shared-mailbox-office-exchange-online.html
https://social.technet.microsoft.com/Forums/en-US/336e02ee-6767-4810-90a0-1352bd7cc9e9/office-365-how-to-access-a-shared-mailbox-using-imap-client?forum=onlineservicesexchange
======================================================================
Q. How can I use a shared account as a backup account for several
mailboxes and so avoid spending too many dollars in backup accounts?
R. Use the option --subfolder2 backup_foo
imapsync ... --user2 sharedloginsyntax --subfolder2 backup_foo
To restore, do the reverse with:
imapsync ... --user1 sharedloginsyntax --subfolder1 backup_foo
======================================================================
Q. Office365 fails with "User is authenticated but not connected".
R1. "The message User is authenticated but not connected is due to a
bug in the Office365 server's IMAP implementation. If the client
presents a valid user name but an invalid password, the server
accepts the login, but subsequent commands fail with the
aforementioned error message." Source:
https://unix.stackexchange.com/questions/164823/user-is-authenticated-but-not-connected-after-changing-my-exchange-password
Thanks to James Abbottsmith for this link and explanation at
https://github.com/imapsync/imapsync/issues/32#issuecomment-153561647
R2. Miguel Alameda reported understanding and solving this issue
like this, the context was admin/authuser:
"The admin user had not permission in the target mailbox."
======================================================================
Q. I see "NO Maximum size of appendable message has been exceeded"
What can I do with that?
R. Office365 supports send/receive max message sizes of up to 150MB
but you need to make changes in your tenant(s) to support it.
The following PowerShell command will increase the message sizes that
can be sent/received. The trick in getting IMAPSync to work is to
apply these settings to the accounts performing the migration,
NOT the accounts associated with the target mailbox (assuming you're
using service accounts to perform transfers on behalf of users).
Set-mailbox -Identity $UPN -MaxReceiveSize 150mb -MaxSendSize 150mb
e.g.
Set-mailbox -Identity "migrationaccount@testtenant.onmicrosoft.com" -MaxReceiveSize 150mb -MaxSendSize 150mb
We're transferring data between Office 365 tenants so we set these
values on the migration accounts in the source and target tenants.
Thanks to Sean McDougall, Ian Thomas & Matt Wilks from Toronto
for this FAQ item.
======================================================================
Q. Every single mail synced to exchange online owns the
category $MDNSent after migration. How can I avoid this?
R. To remove the flag $MDNSent from all messages, use:
imapsync ... --regexflag "s/\$MDNSent//g"
See also the document
https://imapsync.lamiral.info/FAQ.d/FAQ.Flags.txt
======================================================================
Q. The imap connection to Office365 is not working very efficiently,
is there a solution to fix that?
R. Yes. Try DavMail
http://davmail.sourceforge.net/
I don't use it myself but a user, Yannick Palanque, reported
great results using it.
======================================================================
Q. The sync fails with many "Trying command when NOT connected!".
What can I do?
R. The --debugimap option can show you more details, especially
messages like this one:
"BAD Request is throttled. Suggested Backoff Time: 178755 milliseconds"
In that case, see the next faq item and its fixes.
======================================================================
Q. Office365 throttles the sync and says:
"Request is throttled. Suggested Backoff Time: 299961 milliseconds".
What can I do with that?
R. Office365 has throttle mechanisms to limit any huge usage.
Sometimes imapsync transfers are too stressful for servers.
The following message
"Request is throttled. Suggested Backoff Time: 299961 milliseconds"
comes from the imap Office365 server, imapsync just reports it
before being disconnected from it.
To solve the throttles issues from 0365, there are two solutions
at least:
R1. Call Microsoft Office365 and ask them to remove the limits on your
mailboxes. That's not a joke, they do it for 90 days usually,
sometimes only after you reach the second technician you call,
the first one usually be not enough competent to understand
what you're talking about (I would be glad to remove this bad
fact).
R2. Play with options --maxbytespersecond or --maxmessagespersecond
or --exitwhenover
imapsync ... --maxbytespersecond 100_000
imapsync ... --maxmessagespersecond 2
imapsync ... --exitwhenover 1_000_000_000
I don't know the upper value that avoids the default throttling from 0365
and I guess it changes over time.
======================================================================
Q. What are the receive and sending limits of Office365?
R1. 3600 messages per hour (22 July 2021). Maybe 1200 as imapsync can
be viewed as a single sender. Tell me what you experienced,
I'll update this point.
https://docs.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#receiving-and-sending-limits
In that case, imapsync can adapt to this with (1 message/second = 3600 messages per hour)
imapsync ... --maxmessagespersecond 1
or
imapsync ... --maxmessagespersecond 0.33
R2. I also found "Microsoft theoretically allows for about 300MB of
throughput per user per hour." at
https://www.systools.in/blog/microsoft-office-365-throttling-policy/
In that case, imapsync can adapt to this with (83333 bytes/second = 300 MBytes/hour)
imapsync ... --maxbytespersecond 83333
======================================================================
Q. Office365 refuses to create the folder named "Files" with the error
"NO Folder name is reserved". What happens?
R. The folder Files is a standard folder in Office365.
It should not be synced in IMAP. See the next question.
======================================================================
Q. Office365 users complain that a folder named "Files" contains
messages with no sender.
R0. To fix this, add --exclude Files
imapsync ... --exclude Files
If you use --office1 then imapsync will add this exclusion
automatically like using the option:
imapsync ... --exclude "^Files$"
If you use --office2 then imapsync will add a renaming of any
"Files" folder on host1, like using the option:
imapsync ... --f1f2 "Files=Files_renamed_by_imapsync"
The host2 account ends up with a folder named
"Files_renamed_by_imapsync", but no complaining.
R1. This folder "Files" seems to be a standard folder in Exchange Online,
but it is not. This folder contains all attachments in every email
that is in the mailbox but without any headers.
This causes some confusion for users as these appear in their search
results as duplicate lines but without the sender details or even
the message body.
This folder seems to be usually hidden so IMAP clients can’t see it,
but for some reason sometimes it becomes visible.
R2. It looks to be a common problem with Exchange Online.
I’m not sure what causes the folder to appear.
More info here:
https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_exchon-mso_o365b/exclude-the-exchange-online-system-folder-called/2adbdf84-db4a-4c7f-ac29-738757980a0d
https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_dep365-mso_o365b/no-sender-emails-in-files-folder/534bae8f-a7d7-4f5f-8ed7-5bad0d5fa23f
(This question/answer is taken quasi verbatim from Perttu Aaltonen)
======================================================================
Q. From XXX to Office365, read receipts are all resent again after sync.
Even for old messages. How can I fix that?
R. Imapsync can remove the header containing this read-receipt request.
On Unix or Windows use:
imapsync ... --disarmreadreceipts
Since read receipts should be sent for unseen messages that will go
to a seen state after the migration, you could be strict and apply
the regex only to seen messages.
Selecting seen message can be done with:
imapsync ... --search1 "SEEN" --disarmreadreceipts
A second run has to be run without the --disarmreadreceipts
for unseen messages:
imapsync ... --search1 "UNSEEN"
If fact --disarmreadreceipts is just an option equivalent to:
--regexmess 's{\A(.*?(?! ^$))^Disposition-Notification-To:(.*?)$}{$1X-Disposition-Notification-To:$2}igxms'
That regex changes the header Disposition-Notification-To. It prefixes
it with an X- so that it becomes inactive.
Disposition-Notification-To: blabla
becomes
X-Disposition-Notification-To: blabla
Thanks to David Karnowski for pointing and solving this issue.
======================================================================
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
R1. "SSL read or write error" happens sometimes, it isn't related to
imapsync directly but to the ssl underlying library when communicating
with Exchange in TLS/SSL encrypted mode.
Next runs should put the sync further, so rerun the syncs
until it is well completed.
R2. Another solution is to remove --tls or --ssl options for Exchange
and accept clear text syncs.
R3. See also the FAQ FAQ.SSL_errors.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.SSL_errors.txt
======================================================================
Q. From XXX Office365 I get this error message sometimes:
"BAD Command Argument Error 11". What does it mean?
R. This error message comes from the Office365 IMAP server when it
encounters any problem. Most of the time it is one of the following:
* Some messages are bigger than the size limit.
45 MB by default on Office365.
I don't know if it can be upped by configuration for Office365.
If you can't fix this limit on Office365 then use the option
--maxsize 45000000 for 45 MB to tell imapsync to skip those messages.
imapsync ... --maxsize 45000000 # 45 MB for Office365
* Quota reached. The whole account is full.
It can be upped by configuration.
* You use --synclabels --resynclabels from a previous command line
related to syncing from Gmail to Gmail. Remove them.
* Some messages have some lines too long. Use option --maxlinelength
to skip messages whose max line length is over some bytes.
--maxlinelength 1000 is an RFC2822 must but most servers support
higher values. Office365 supports 10500 characters line length:
imapsync ... --maxlinelength 10500
In case you prefer fixing messages with long lines the hard way,
instead of skipping them with --maxlinelength 10500, just use:
On Windows
imapsync ... --regexmess "s,(.{10500}),$1\r\n,g"
On Unix
imapsync ... --regexmess 's,(.{10500}),$1\r\n,g'
Have also in mind that Office365 closes the connection after 10 errors
encountered so you might also see "BYE Connection closed" errors from
Office365, which means Office365 leaves the session and says goodbye,
come back later. Redo some sync then.
On Linux, there is a good Python script in the tarball that can
fix several things that Office365 has issues with.
Use it like this:
./imapsync ... --pipemess W/tools/fix_email_for_exchange.py
It often does some miracles on messages.
It's called fix_email_for_exchange because Office365 is Exchange,
or at least started to be an Exchange server with the same issues.
======================================================================
Q. From XXX to Office365 the flag Flagged does not seem to be well synced.
What can I do?
R. Use the following trick. Run imapsync twice, one with --regexflag
and one without, like this:
1) imapsync ... --regexflag "s/\\Flagged//g"
2) imapsync ...
You can add --debugflags if you want to see what imapsync gets and
does in detail with flags.
The magic of this trick is on ignoring the \Flagged flag on the first
sync and setting it on the second sync, with STORE instead of APPEND.
This Office365 bug seems that Office365 gets and sets well the Flagged
flag with APPEND in IMAP but then it forgets it with other protocols;
With STORE it sets and gets the "\Flagged" flag everywhere.
Thanks to Dave Murray and Simon Savva for reporting and solving
this issue.
======================================================================
Q. How to migrate from or to Office 365 with an admin/authuser account?
Note from Yago Torres Fernandez:
(a working command using admin/authuser on host2 Office 365)
imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated@domain.com ^
--password2 XXXX --ssl2
but previously in Office365, you must do something like the following, using Powershell:
Add-MailboxPermission -identity user_to_be_migrated@domain.com -user user_admin@domain.com -accessrights fullaccess -inheritancetype all
Note from Betsy Lawlor: You can use global modern authentication with two factor
on Exchange Online (M365) but you must have "AllowBasicAuthImap" on
the admin account you are using to migrate the mail.
Note from Guido (5 April 2022):The way I fixed it was by turning off security defaults
https://docs.microsoft.com/nl-nl/azure/active-directory/fundamentals/concept-fundamentals-security-defaults.
You still need to check IMAP access on an account-basis though.
Remark: PLAIN authentication is the only way to go with --authuser1 for now.
So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.
Note from Rafael Alvarez Ballesteros:
When you get an Office365 license you will receive an admin user to handle
your licenses and products like admin@yourcompanyname.onmicrosoft.com.
This account is the administrator account; some weeks or months ago Microsoft
has decided you need to use two-factor authentication by default,
so if two factors authentication is enabled you will no be able to sync
the mail (it will not connect to host2).
OWA (I think this is the two-factor authentication) needs to be disabled
globally and enabled individually on the users you want to.
One account can have the right to access other mailboxes no matter if admin or
any other account but needs to have OWA disabled to be able to connect
to the office365 server.
Note from Martin Paulucci:
I had to remove the domain part for the user
but not for the admin. Example:
imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated
See also:
http://linux-france.tk/prj/imapsync_list/msg02203.html
Subject: RE: [imapsync] Office 365 - 'Master User'?
Date: Mon, 1 Jun 2015 17:53:54 +0000
======================================================================
Q. Couldn't create folder [trash] "Mailbox already exists".
R. Some servers take care of character cases in folder names,
some servers do not, like Exchange. Since not respecting the case
can merge two different folders into one then imapsync respects the case.
For example, if a host1 server has a folder name called "trash"
and the host2 server already has a folder "Trash" or "TRASH"
then imapsync will try to create the folder "trash" on host2
because trash and Trash are different strings. But if host2
does not respect character case it will consider folder "trash"
already exists and will say it. That's the error message reported
by imapsync: "Mailbox already exists". This message comes from the server.
The folder creation fails but messages are well transferred in
so take a look at this warning, understand why it happens
and it should be fine most of the time.
To avoid this warning use --regextrans2 to map the folder names
imapsync ... --regextrans2 "s/^trash$/Trash/"
If there are two folders Trash and trash on host1 then both
will be merged into only one Trash folder on host2.
In case of the option --delete2 is used the regextans2 above becomes
mandatory, otherwise imapsync will sync messages from the
first Trash and then delete them when syncing trash.
If you want to avoid merging folders that are considered different
on host1 but are considered the same on destination host2 because of
case sensitivities and insensitivities, use --nomixfolders
======================================================================
======================================================================

View File

@ -0,0 +1,66 @@
#!/bin/cat
$Id: FAQ.Old_Style_Web_Design.txt,v 1.9 2020/12/02 21:38:13 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Old_Style_Web_Design.txt
=======================================================================
Imapsync and its ugly website style
=======================================================================
=======================================================================
Q. Why does the imapsync website look so old fashioned?
R. Once in a while I receive messages from people complaining about
the website style, not that much about the content by the way,
as people don't read.
I know the imapsync website has a grandmother style,
I'm not offended by complaints about it.
Here are several reasons it stays like that for now:
* I like to do things myself
* I am lazy
* I like static web.
Although you could argue that I do use some dynamic things, like SSI.
That's true, I use Server Side Include, aka SSI, denoted by the ".shtml"
extension of some files, instead of the classical ".html" extension.
It's because I like dynamic web when the dynamic brings something that
can't be done easily with static files.
* I'm proud to be w3c compliant
https://validator.w3.org/check?uri=https%3A%2F%2Fimapsync.lamiral.info
Very very few sites are w3c compliant, it's because nearly all CMS frameworks aren't.
Only true crazy people are w3c compliant seekers. I'm one of them.
If you know a CMS w3c compliant, tell me, it will be such a crazy news.
If you know any web site w3c compliant, tell me, it will be crazy news as well.
Update: https://contao.org/ is a CMS and it is w3c compliant! This is astonishing!
(Thanks to Markus Rupprecht for this crazy input!)
* I prefer to spend more time on the product itself than the showcase
* The product is a command line, a fancy website to sell a command line
makes more disappointed buyers
I agree I could do some sort of A/B experiment and see whether during
a month I sell more imapsync on a brand new style B site than
the current A one.
Besides, don't misunderstand buyers. The surprising thing about complainers
is that all are web designers, none of them is a user not in the webdesign branch.
I even received some phone call from a buyer saying: "I saw your website
and I told to myself: this can't be a fake one, the product must be a good one".
Another folk like me:
http://keyhut.com/pos.htm
(well, not w3c compliant, he is not that mad)
So maybe one day...

300
FAQ.d/FAQ.OnlineUI.txt Normal file
View File

@ -0,0 +1,300 @@
#!/bin/cat
$Id: FAQ.OnlineUI.txt,v 1.36 2022/09/14 11:18:05 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.OnlineUI.txt
=====================================================================
Imapsync tips about the online visual user interfaces
https://imapsync.lamiral.info/X/
=====================================================================
Questions answered in this FAQ are:
Q. Can I launch several imap synchronizations on the visual
user interface /X?
Q. How secure is the online visual user interface /X?
Q. Does the online service store any sensitive information
like my passwords?
Q. I want to switch from the visual interface /X to the
imapsync command line or to the script examples
https://imapsync.lamiral.info/#DOC_BASIC
What should I know?
Q. Shall I have issues when the browser times out?
What happens if the browser connection is closed for whatever reason?
Q. Shall I have issues when the webserver times out? What happens
if the web server closes the connection for whatever reason?
Q. The sync stalls at the beginning, just after a line like:
"Host1: xxx says it has CAPABILITY for AUTHENTICATE LOGIN"
What is the problem?
Q. The synchronization fails with the error message like:
Err 1/1: Host1 failure: Error login on [10.1.161.155] with user [webmaster@truc.com] auth [LOGIN]: 2 NO [ALERT] LOGIN DENIED -- COUNTRY IS BLACKLISTED
Now the questions again with their answers.
=====================================================================
Q. Can I launch several imap synchronizations on the visual
user interface /X?
A. Yes. Open several tabs/windows on your browser and fill each one
with different credentials.
=====================================================================
Q. How secure is the online visual user interface /X?
A0. Well, I don't know if asking the provider whether his online
service is secure or not would be of any interest.
Let's do it anyway, you'll be the judge.
A1. Some figures
Date of this report: 24 December 2021. Happy Christmas!
The online imapsync service /X started 9 January 2017,
5 years ago (1810 days of service).
On average, /X has 51 users per day, each user launches on average
6 account migrations, from just 1 launch to many (hundreds).
The total volume /X transferred so far is around 240 TiB coming
from nearly 520 000 imap account migrations and 810 million email messages.
A2. Pros & Cons
The online imapsync service /X runs on HTTPS only, with a letsencrypt
certificate, an up to date certificate overall rated "A+" at
https://www.ssllabs.com/ssltest/analyze.html?d=imapsync.lamiral.info
Because of the HTTPS usage, what the users enter in their browser, the
imap logins and passwords, can't be eavesdropped on the network.
Imapsync itself takes care of encryption for the imap sessions, if
possible. First, imapsync tries to use SSL on port 993, then TLS on
port 143 if the servers announce TLS, then no encryption at all.
Concerning encryption, what is done with the source imap server host1
is independent of what is done with the destination imap server host2.
On the date of 24 December 2021, there is no security problem detected
or reported to me (Gilles LAMIRAL), so far.
Concerning the Log4j threats, the imapsync server doesn't use Log4j at
all, so it should be safe about this issue.
Feel free to attack the service and feel free to report any hole
encountered. Have in mind that I can watch what you do from the
server-side, and then take measures if the service suffers from your
acts. Drop me a note before will be fair play and I will let you act
as harsh as you can.
As the owner of the service, it could have been 520 000 pairs of
credentials collected and nearly 240 terabytes of email messages.
That's massive. I haven't kept them but I can't prove I haven't. It's
just trust, like nearly every online service in the universe.
Concerning imapsync transfers, the imap server certificates are not
checked for authenticity by default because too many imap servers are
crappy configured regarding their certified certificates.
This default behavior is chosen like this because users of /X
want their emails transferred, instead of being not transferred
because of an incompetent imap server sysadmin.
I admit that this part, checking imap ssl/tls certificates,
could be improved from my side by including well known
certificates directly in imapsync.
If an imap server doesn't honor ssl nor tls, then logins, passwords
and everything will go clear text during the imap transfers. That's
not good at all! But what "comforts" me is that if an imap server does
only clear text transfers, then it's also true for all the imap
sessions the account owner encounters, imapsync is just one of them;
the imap servers are then always unsecured for any imap software
client.
Last point, who could be sure that no cracker cracked the online hosts
and that he isn't currently sniffing the credentials?
No one! I'm not sure myself, even if I do take care of that
possibility. So changing the imap accounts passwords after
the sync is a safe and recommended practice! You can even
change the password just after imapsync has started its job,
just after a successful login. This way you know the previously
given password is no longer working just after imapsync has
started its job.
=====================================================================
Q. Does the online service store any sensitive information
like my passwords?
No.
The online passwords are kept by your browser. The goal is that the
next time you run a sync from your browser, all the parameters are
already there without having to re-enter them. The save is done when
you click on the "Sync or resync" green button. So, if you don't want
your browser to keep your password, enter a wrong one or empty the
password field and click on the "Sync or resync" green button, the
wrong password will be saved in your browser and no sync will be done,
since a correct login is mandatory to do anything on your mailboxes.
=====================================================================
Q. I want to switch from the visual interface /X to the
imapsync command line or to the script examples
https://imapsync.lamiral.info/#DOC_BASIC
What should I know?
A. Let's do some ascii art.
The visual interface looks roughly like this, for the textfields input part:
+------------------------------------------+------------------------------------------+
| IMAP source Mailbox | IMAP destination Mailbox |
| | |
| Login (usually an email address) | Login (usually an email address) |
| | |
| test1 | test2 |
| | |
| Password | Password |
| | |
| secret1 | secret2 |
| | |
| IMAP Server hostname (or its IP address) | IMAP Server hostname (or its IP address) |
| | |
| test1.lamiral.info | test2.lamiral.info |
| | |
+------------------------------------------+------------------------------------------+
Notice the 6 examples values I put in the previous "picture":
* test1
* secret1
* test1.lamiral.info
* test2
* secret2
* test2.lamiral.info
The corresponding imapsync command line on Windows is:
imapsync.exe --host1 "test1.lamiral.info" --user1 "test1" --password1 "secret1" ^
--host2 "test2.lamiral.info" --user2 "test2" --password2 "secret2"
The order of the parameters is whatever you want as long as you respect each pair
like: --optionname "value"
There is no need to type this in a command prompt window, just use the example script
https://imapsync.lamiral.info/examples/imapsync_example.bat
The corresponding imapsync command line on Linux is:
imapsync --host1 "test1.lamiral.info" --user1 "test1" --password1 "secret1" \
--host2 "test2.lamiral.info" --user2 "test2" --password2 "secret2"
There is no need to type this in a command prompt window, just use the example script
https://imapsync.lamiral.info/examples/imapsync_example.sh
=====================================================================
Q. Shall I have issues with the browser timing out?
What happens if the browser connection is closed for whatever reason?
A. A browser connection closed closes also the imapsync process,
ie, the sync is ended right away.
Further comments on this behavior.
When using the /X interface there are three connections.
One connection is the Browser/WebServer connection,
the two other connections are the WebServer/ImapServers
connections (imapsync stuff).
If the Browser/WebServer connection timeouts or ends,
the imapsync sync is also ended immediately by the remote
Apache HTTPS server. Technically, Apache sends a TERM signal
to the imapsync process, then wait some seconds before
sending a KILL signal if the imapsync process is still alive.
You can relaunch the sync with the "Sync!" button, at any time.
If the "Sync!" button is gray/inactive then just reload
the page (F5 or similar), and reenter the credentials.
If the interface tells you that the sync is already going on,
it may be that the sync is running from another browser or place.
You can stop this sync with the "Abort!" button from any /X
tab/window, even from another browser or place. To be able
to abort with success, you have to give the same account
parameters, same credentials, or imapsync will ignore the demand.
In other words, you can try safely to launch several parallel
runs between the same mailboxes. Open a new tab/windows with /X,
and start the same sync. It's safe, the /X page will say that
there is already one sync running and it will present
the logfile running the sync like a "tail -f" command (isn't that magic?).
=====================================================================
Q. Shall I have issues when the webserver times out? What happens
if the web server closes the connection for whatever reason?
A. If the webserver closes the connection then usually it also
kills the imapsync process and the imap connections as well.
The current webserver timeout at
https://imapsync.lamiral.info/X/
is 3600 secondes, one hour.
=====================================================================
Q. The sync stalls at the beginning, just after a line like:
"Host1: xxx says it has CAPABILITY for AUTHENTICATE LOGIN"
What is the problem?
A. I've seen this issue on /X with the imap server
CommuniGate Pro IMAP Server 6.0.11
The issue looks related to special characters in the password.
Solution for now: change the password, keep only standard (ASCII)
alphanumeric characters ABC-YZ abc-yz 012-89.
=====================================================================
Q. The synchronization fails with the error message like:
Err 1/1: Host1 failure: Error login on [10.1.161.155] with user [webmaster@truc.com] auth [LOGIN]: 2 NO [ALERT] LOGIN DENIED -- COUNTRY IS BLACKLISTED
A. The message "[ALERT] LOGIN DENIED -- COUNTRY IS BLACKLISTED" comes directly
from the IMAP server at 10.1.161.155
It looks like that the 10.1.161.155 imap server filters incoming
connections based on their IPS. In that case, here are the current
IPs of the imapsync online service (September 2022):
ks5.lamiral.info has address 91.121.221.224
ks5ipv6.lamiral.info has IPv6 address 2001:41d0:2:84e0::1
ks6.lamiral.info has address 5.39.87.81
ks6ipv6.lamiral.info has IPv6 address 2001:41d0:8:9951::1
ks7.lamiral.info has address 5.135.177.225
ks7ipv6.lamiral.info has IPv6 address 2001:41d0:8:b8e1::1
i050.lamiral.info has address 213.32.72.139
i050ipv6.lamiral.info has IPv6 address 2001:41d0:302:1000::155d
vp3.lamiral.info has address 51.178.81.27
vp3ipv6.lamiral.info has IPv6 address 2001:41d0:404:200::4d81
vp4.lamiral.info has address 51.38.34.201
vp4ipv6.lamiral.info has IPv6 address 2001:41d0:305:2100::4c46
The imapsync online service may contact your imap server with one
or more of those IP addresses.
CPhulk of Cpannel has a buildin country block that can cause this behavior.
Later on you may encountered issues with maximum sessions per ip.
You can change that within cPannel.
https://support.cpanel.net/hc/en-us/articles/4406663082519-What-is-cPHulk-
https://docs.cpanel.net/whm/security-center/cphulk-brute-force-protection/
=====================================================================
=====================================================================

118
FAQ.d/FAQ.POP3.txt Normal file
View File

@ -0,0 +1,118 @@
#!/bin/cat
$Id: FAQ.POP3.txt,v 1.6 2022/04/05 13:48:52 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.POP3.txt
=======================================================================
Imapsync tips about POP3.
=======================================================================
Questions answered here are:
Q. Can I use imapsync to migrate emails from pop3 server to imap server?
Q. How can I handle those terrible POP-leaving-a-copy users?
POP-leaving-a-copy clients will download a new copy of all
their emails after a migration, which is a big pain and
causes hundreds of support calls.
=======================================================================
Q. Can I use imapsync to migrate emails from pop3 server to imap server?
R1. No, but you can migrate emails from a pop3 server to an imap server
with the command line tool pop2imap:
http://linux-france.tk/prj/pop2imap/
http://linux-france.tk/prj/pop2imap/README
R2. Yes, sometimes, because many pop3 servers runs in parallel
with an imap server on exactly the same mailboxes. They serve
the same INBOX, ie, imap serves INBOX and several other folders,
pop3 serves only INBOX.
So have a try with imapsync on the same host1.
=======================================================================
Q. How can I handle those terrible POP-leaving-a-copy users?
POP-leaving-a-copy clients will download a new copy of all
their emails after a migration, which is a big pain and
causes hundreds of support calls.
R1. Context and condition: you have access to the POP email client tool
and the user WILL STAY using POP.
POP email client tools can leave or delete the messages after pulling
them. It's very often a configurable behavior. Configure the email client
tool to delete the messages already pulled and then they won't never
be pulled again from the server to the client.
If you can't access to the POP email client tool, see R2 below.
Problem in the future: INBOX will be seen as empty when the email client
go to IMAP (unless the email client tool allows then an upload
synchronization, but don't count on that...)
R2. Context and condition: you know which users use POP.
If the pop mailboxes are known then all the INBOX messages can be moved
in an subfolder called INBOX_backup for example. To help you do this,
the tool pop2imap transfers messages from POP to IMAP, without duplicates,
without deleting the messages transferred, by default, and can put them
in any subfolder you want to. It can also delete them and, that way,
do what the email client tool don't or can't do, for any reason.
We're here to do a good job with what we have, do we?
In case of Dovecot for example, the magic can be there since already
fetched messages via POP are seen as \Seen in IMAP but the new messages are
seen as new messages, thanks to the Dovecot developer Timo's cleverness!
So it is possible to sync back only the new messages to INBOX,
with "imapsync ... --search UNSEEN ...", and then POP users won't see the old
messages come back but they will see the new messages and fetch them via POP.
R3. Context and condition: you know nothing
Do R2 without the sync back of UNSEEN messages. It can't hurt to have a backup
of INBOX in a subfolder for IMAP or POP users.
R4. Here are some points, an analysis that made me propose the not
too bad but not very good R2 answer.
There was a discussion about this issue on the imapsync mailing-list:
http://linux-france.tk/prj/imapsync_list/msg02622.html
http://linux-france.tk/prj/imapsync_list/msg02623.html
My reply was not very good
http://linux-france.tk/prj/imapsync_list/msg02624.html
It's because I wrote pop2imap and it uses a stateless
mechanism to avoid duplicates, I wasn't aware of UIDL when I wrote pop2imap,
and later when I entered the discussion.
See also Rick Sanders discussion about this issue:
https://www.emailquestions.com/threads/how-to-prevent-duplicate-emails-after-server-migration-for-pop3-users-with-leave-mail-on-server.8109/
Quoting Timo Sirainen on
https://wiki2.dovecot.org/Migration
"If a client already saw changed UIDLs and decided to start
re-downloading mails, it's unlikely there is anything you can do to
stop it. Even going back to your old server is unlikely to help at
that point."
"Some (many?) POP3 clients also require that the message ordering is
preserved."
"Some clients re-download all mails if you change the hostname in the
client configuration. Be aware of this when testing."
Quoting
https://wiki.dovecot.org/POP3Server
"Flag changes
By default when a message is RETRed, \Seen flag is added to it.
POP3 itself doesn't support flags, but if the mailbox is opened
with IMAP (eg. from webmail) it's shown as seen."
=======================================================================
=======================================================================

View File

@ -0,0 +1,38 @@
#!/bin/cat
$Id: FAQ.Passwords_on_Mac.txt,v 1.1 2020/02/12 11:25:48 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Mac.txt
=======================================================================
Imapsync issues with passwords on Mac OS X.
=======================================================================
=======================================================================
Q. On Mac, some passwords contain some weird *()",;&~ characters.
Login fails.
R1. Enclose the password within single-quotes in the imapsync
command line:
imapsync ... --password1 'passw*()",;&rd~'
CAVEAT: Be careful to use quotes and double-quotes characters encoded
from the ASCII table. How to be sure of that?
Mac OS X has a "use smart quotes" option. This is particularly handy
when typesetting and the system automatically substitutes opening and
closing quotes. It isn't handy when editing the imapsync command line.
Disable "use smart quotes and dashes" in the Keyboard configuration,
located in the "Text" tab. It will fix this unfortunate character
transformation. You can put it back later.
R2. Change the password in order to keep only f...ing normal characters,
ie, alphanumerics like aAzZ-_09, and minus and underscore.
Make it long and random if strong security is your concern.
=======================================================================
=======================================================================

View File

@ -0,0 +1,40 @@
#!/bin/cat
$Id: FAQ.Passwords_on_Unix.txt,v 1.7 2020/12/04 11:46:54 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
=======================================================================
Imapsync issues with passwords on Unix.
=======================================================================
=======================================================================
Q. On Unix, some passwords contain some weird *()$,";&~ characters.
Login fails.
R1. Enclose the password within single-quotes in the imapsync
command line:
imapsync ... --password1 'passw*()`"$,;&rd~'
R2. Change the password to keep only f...ing normal characters.
Make it long and random if strong security is your concern.
R3. If R1 fails, with very old imapsync or old Mail::IMAPClient Perl module,
try also using double-quotes within single-quotes. It will enclose
the password within double-quotes in the imap LOGIN command:
imapsync ... --password1 '"passw*()`$,;&rd~"'
Old Mail::IMAPClient 3.37 is a bad horse having this issue solved
by this double-quotes trick.
fixed in 3.38 https://metacpan.org/changes/distribution/Mail-IMAPClient
version 3.38: Tue Feb 9 02:48:21 UTC 2016
- rt.cpan.org#110273: failure to quote password values
(regression introduced in 3.36 via fix for rt.cpan.org#100601)
=======================================================================
=======================================================================

View File

@ -0,0 +1,65 @@
#!/bin/cat
$Id: FAQ.Passwords_on_Windows.txt,v 1.5 2018/06/11 22:35:10 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
=======================================================================
Imapsync issues with passwords on Windows.
=======================================================================
Related documentation:
http://www.robvanderwoude.com/escapechars.php
http://stackoverflow.com/questions/3288552/how-can-i-escape-an-exclamation-mark-in-cmd-scripts
In case you're brave and relentless, understand and try this:
http://www.dostips.com/forum/viewtopic.php?f=3&t=1733
=======================================================================
Q. On Windows, some passwords contain $ or other special characters
like one more of the nine characters $%&<>|^"!
Login fails.
R0. Change the password so that only normal characters remain.
It's often a quick and good solution. If you care about
strong password policy, just make it longer.
If you can't avoid special characters use no problematic
characters such as underscore _ or minus - or comma , or dot.
If you can't change passwords then read on and good luck!
R1. Enclose passwords between ""
imapsync ... --password1 "zzz$zz$$z"
R2. Prefix each $ character with a ^ since ^ is the escape character
on Windows
imapsync ... --password1 zzz^$zz^$^$z
For a password that is exactly the 8 characters string $%&<>|^"
you have to enter
imapsync ... --password1 "$%%&<>|^"^"
The exclamation mark ! does not have to be escaped sometimes,
so have a try.
=======================================================================
Q. On Windows, some passwords begin with an equal = character.
Login fails. What can I do?
R. Use twice equals == characters instead; For example, if =secret
is the password then use:
imapsync ... --password1 ==secret
or even
imapsync ... --password1 "==secret"
=======================================================================
=======================================================================

49
FAQ.d/FAQ.Principles.txt Normal file
View File

@ -0,0 +1,49 @@
$Id: FAQ.Principles.txt,v 1.10 2021/04/18 13:00:00 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Principles.txt
=======================================================================
Imapsync principles and design ideas
=======================================================================
Questions answered in this FAQ are:
Q. Give me some design ideas of imapsync
Now the questions again with their answers.
=======================================================================
Q. Give me some design ideas of imapsync
R.
* No configuration file
* Stateless
* Rsync in mind
* Reliable
* Robust
* The user is not an expert, help him with:
* Good behavior by default.
* No harm by default.
* Prefer fixing an issue in the code than documenting the issue
* Inline help suggestions
Consequences of those design ideas:
* Stateless:
* A sync can be done by another tool before or after imapsync, it is
safe as long as the other tools don't modify the messages' headers
"Message-Id:" and "Received:", or at least "Message-Id:".
* Ready to run in a container.
=======================================================================
=======================================================================

View File

@ -0,0 +1,80 @@
#!/bin/cat
$Id: FAQ.Release_Checklist.txt,v 1.27 2022/09/16 05:35:27 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Release_Checklist.txt
=======================================================================
Imapsync developer notes
=======================================================================
Checklist before making a new release:
- Make "imapsync --tests" pass on Linux
- Make "imapsync --tests" pass on Mac
- Make "imapsync --tests" pass on Windows 32 bits
- Make "imapsync --tests" pass on Windows 64 bits
make linuxtests
make win32tests
make win64tests
make mactests
or
make -j linuxtests win32tests win64tests mactests
- Make "imapsync --testslive" pass on Linux
- Make "imapsync --testslive" pass on Mac
make linuxtestslive mactestslive mactestslive6
- Add a new section in S/news.shtml reading "rlog imapsync"
- In case of a new dependency, add it to all the INSTALL/* files.
- Generate the README
make doc
- Run a spell checker on the README
- Read the README again slowly. Fix all issues, all.
- Read the OPTIONS section of README and read it very slowly
- Read slowly README_Windows.txt
- Read slowly the TUTORIAL_Unix file in html
- Make "rcsdiff imapsync" give nothing relevant and exit 0
- make ci pass.
- Review the newsletter by running:
m4 -P W/ml_announce.in.txt
- Update the file W/rsync_exclude_dist.txt
- Update the file .gitignore
- Review the TODO file and mark done what is done.
- Review the general FAQ.d/FAQ.General.txt
- Report values of --gmail1 --gmail2 to FAQ.Gmail.txt
- Report values of --exchange* to FAQ.Exchange.txt
- Report values of --office* to FAQ.Office365.txt
- /X verify direct cgi
- /X verify under noscript with firefox
- /X verify under private with chrome
- Check the binaries imapsync.exe imapsync_32bit.exe with
https://www.virustotal.com/
https://www.metadefender.com/
- make dist
- verify on Windows the zip extraction and simple execution of imapsync_example.bat
- verify binary version matches imapsync script version
- make publish
- verify VERSION is uploaded
=======================================================================
=======================================================================

View File

@ -0,0 +1,33 @@
#!/bin/cat
$Id: FAQ.Reporting_Bugs.txt,v 1.4 2018/05/24 11:34:30 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Reporting_Bugs.txt
=======================================================================
Imapsync bugs reports
=======================================================================
=======================================================================
Q. How can I report bugs or problems I encountered with Imapsync?
R. Help me to help you: follow the following guidelines.
Report any bug or feature request directly to the author by
email at <gilles@lamiral.info>
Put a useful title with word "imapsync" in it: my spam filters
won't filter it.
Provide me any useful information. The simplest way is to attach
the complete log file, in case it is not too big, let say less
than 1MB. Don't zip it, it will slow my response.
Thanks!
=======================================================================
=======================================================================

256
FAQ.d/FAQ.SSL_errors.txt Normal file
View File

@ -0,0 +1,256 @@
#!/bin/cat
$Id: FAQ.SSL_errors.txt,v 1.18 2022/01/14 21:20:37 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.SSL_errors.txt
======================================================================
Imapsync SSL errors
======================================================================
Questions answered in this FAQ are:
Q. What is the error
DEBUG: .../IO/Socket/SSL.pm:1177: global error: Undefined SSL object
Q. What are the errors
DEBUG: .../IO/Socket/SSL.pm:1165: local error: SSL write error
or
DEBUG: .../IO/Socket/SSL.pm:1088: local error: SSL read error
Q. What can I do to avoid those "SSL read/write errors"?
Q. What are the errors
SSL connect attempt failed SSL
routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
or
SSL connect attempt failed SSL
SSL routines:ssl_choose_client_version:unsupported protocol
Q. What is the error
fatal SSL error: SSL connect attempt failed with unknown error
SSL wants a read first
Q. How to see the certificate and identify problems in it?
Now the questions again with their answers.
======================================================================
Q. What is the error
DEBUG: .../IO/Socket/SSL.pm:1177: global error: Undefined SSL object
R. It's a fake error from the Perl Module IO::Socket::SSL
Imapsync works well despite this fake warning but it's disturbing
when you encounter errors due to something else, you believe it's
the issue but no, it's something else to deal with.
This fake error is fixed in IO::Socket::SSL release 2.073
https://metacpan.org/dist/IO-Socket-SSL/changes
"fix #110 - prevent internal error warning in some cases"
https://github.com/noxxi/p5-io-socket-ssl/issues/110
imapsync.exe release 2.178 uses this fixed 2.073 IO::Socket::SSL
======================================================================
Q. What are the errors
DEBUG: .../IO/Socket/SSL.pm:1165: local error: SSL write error
or
DEBUG: .../IO/Socket/SSL.pm:1088: local error: SSL read error
R1. As they claim, those errors are SSL errors. SSL is not directly
done by imapsync but by an underlying Perl module called
IO::Socket::SSL. Those errors arise sometimes and sometimes
they form a series that ends with imapsync auto-abortion.
Those errors happen with some hosts but not with others,
it's often Exchange or Office365. I don't know what exactly happens.
Those errors happen more often on Windows than on Linux.
======================================================================
Q. What can I do to avoid those "SSL read/write errors"?
R0. Windows users: upgrade to imapsync.exe release 1.836 (or next ones)
Those errors appear less often with imapsync releases post 1.836
R1. Remove all ssl/tls encryption
imapsync ... --nossl1 --notls1 --nossl2 --notls2
R2. If you don't want to quit encryption, rerun imapsync until the
complete sync is over. Those errors are not at the same place
each time, so imapsync will sync the remaining messages at each run
until none remains.
R3. Run imapsync on a Linux machine, a VM is ok, there are less
SSL errors on Unix.
R4. Use https://imapsync.lamiral.info/X/
It's a Linux host so response R3 applies there.
R5. Set up a ssltunnel proxy to the host.
Read the file FAQ.Security.txt for an example to set up
a ssltunnel proxy.
======================================================================
Q. What are the errors
SSL connect attempt failed SSL
routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
or
SSL connect attempt failed SSL
SSL routines:ssl_choose_client_version:unsupported protocol
R1. Use:
imapsync ... --sslargs1 SSL_cipher_list=DEFAULT
or
imapsync ... --sslargs2 SSL_cipher_list=DEFAULT
depending on where the error occurs, host1 or host2 or both.
R2. If it doesn't work, I let you try other things,
I quote the "SSL_version" section of
https://metacpan.org/pod/IO::Socket::SSL (Module version: 2.066)
imapsync ... --sslargs1 SSL_cipher_list=DEFAULT
imapsync ... --sslargs1 SSL_version=SSLv2
imapsync ... --sslargs1 SSL_version=SSLv23
imapsync ... --sslargs1 SSL_version=SSLv3
imapsync ... --sslargs1 SSL_version=TLSv1
imapsync ... --sslargs1 SSL_version=TLSv1_1
imapsync ... --sslargs1 SSL_version=TLSv1_2
imapsync ... --sslargs1 SSL_version=TLSv1_3
Those examples are for host1. For host2, use --sslargs2 instead.
Feedback on what worked for you is welcome!
A loop to check every version and print the good ones:
for v in SSLv2 SSLv23 SSLv3 TLSv1 TLSv1_1 TLSv1_2 TLSv1_3; do
imapsync ... --sslargs1 SSL_version=$v && GOOD="$GOOD $v"
done
echo "$GOOD"
I reproduce below the documentation of the underlying Perl
module IO::Socket::SSL used by imapsync:
https://metacpan.org/pod/IO::Socket::SSL
...
SSL_version
Sets the version of the SSL protocol used to transmit data.
'SSLv23' uses a handshake compatible with SSL2.0, SSL3.0 and TLS1.x,
while 'SSLv2', 'SSLv3', 'TLSv1', 'TLSv1_1', 'TLSv1_2', or 'TLSv1_3'
restrict handshake and protocol to the specified version.
All values are case-insensitive. Instead of 'TLSv1_1', 'TLSv1_2', and 'TLSv1_3'
one can also use 'TLSv11', 'TLSv12', and 'TLSv13'.
Support for 'TLSv1_1', 'TLSv1_2', and 'TLSv1_3'
requires recent versions of Net::SSLeay and openssl.
Independent from the handshake format you can limit to set of
accepted SSL versions by adding !version separated by ':'.
The default SSL_version is 'SSLv23:!SSLv3:!SSLv2' which means,
that the handshake format is compatible to SSL2.0 and higher,
but that the successful handshake is limited to TLS1.0 and higher,
that is no SSL2.0 or SSL3.0 because both of these versions have
serious security issues and should not be used anymore.
You can also use !TLSv1_1 and !TLSv1_2 to
disable TLS versions 1.1 and 1.2 while still allowing TLS version 1.0.
Setting the version instead to 'TLSv1' might break interaction
with older clients, which need and SSL2.0 compatible handshake.
On the other side some clients just close the connection
when they receive a TLS version 1.1 request.
In this case setting the version
to 'SSLv23:!SSLv2:!SSLv3:!TLSv1_1:!TLSv1_2' might help.
======================================================================
Q. What is the error
fatal SSL error: SSL connect attempt failed with unknown error
SSL wants a read first
R. If you're using --ssl1 or --ssl2, try instead --tls1 or --tls2
======================================================================
Q. How to see the certificate and identify problems in it?
R. Use the command openssl like this:
echo | openssl s_client -crlf -connect imap.gmail.com:993
echo | openssl s_client -crlf -connect test1.lamiral.info:993
and examine carefully the content, the "verify return:" lines,
the chain. Sometimes, the server certificate is ok but not the whole
chain of certificates so the certification fails.
Here is an example.
One of the certificate is expired:
echo | openssl s_client -crlf -connect test1.lamiral.info:993
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = test1.lamiral.info
verify error:num=10:certificate has expired
notAfter=Apr 11 10:14:05 2021 GMT
verify return:1
depth=0 CN = test1.lamiral.info
notAfter=Apr 11 10:14:05 2021 GMT
verify return:1
---
Certificate chain
0 s:/CN=test1.lamiral.info
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKjCCBBKgAwIBAgISBHYZCE3qSTIlvq97HI5TpBeAMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTAxMTExMDE0MDVaFw0yMTA0MTExMDE0MDVaMB0xGzAZBgNVBAMT
EnRlc3QxLmxhbWlyYWwuaW5mbzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAMUTJVdrTl86nDI2yO6Vz5l1qxMMPqJylQcgi9vDHpwsnUq5HGPv+qZNhM69
...
After an complete server update ("apt update && apt upgrade && /etc/init.d/dovecot restart"):
echo | openssl s_client -crlf -connect test1.lamiral.info:993
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = test1.lamiral.info
verify return:1
---
Certificate chain
0 s:/CN=test1.lamiral.info
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISBD4QN3cfB1JpTm75oVrkkAElMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTAzMTIxODQxMTJaFw0yMTA2MTAxODQxMTJaMB0xGzAZBgNVBAMT
EnRlc3QxLmxhbWlyYWwuaW5mbzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBANuPNbYLPMZ4vPa9NBoHAUdIXqpi0eqdXMXd2sT+qRmqxS5ihr999BHOROcr
...
Champagne!
======================================================================
======================================================================

313
FAQ.d/FAQ.Security.txt Normal file
View File

@ -0,0 +1,313 @@
#!/bin/cat
# $Id: FAQ.Security.txt,v 1.26 2021/07/11 12:11:38 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Security.txt
=======================================================================
Imapsync tips about security. Issues and solutions.
=======================================================================
Questions answered in this FAQ are:
Q. Is running this program a secure method of transferring emails?
Are there any security concerns?
Q. Does imapsync refer to SSL security protocols with --ssl1 and --ssl2
and does it refer to TLS security protocols with the --tls1 and --tls2
options?
Short answer: No.
Q. I noticed that the online UI has no option for TLS/SSL.
Is this secure?
Is this more secure than using the .bat file on my computer?
Q. Are transferred emails/attachments stored on any other
server/location aside from my originating/destination server(s)?
Q. Other than changing passwords on the originating/destination email
accounts once the relevant emails have been moved,
are there any other security tips I should know?
Q. I need to transfer mail from an imap server to another imap server.
Which ports need to be open on the firewall to make this possible?
Q. Does imapsync support the IMAP command STARTTLS?
Q. Does imapsync support IMAP over SSL/TLS (IMAPS)?
Q. How can I test an ssl/tls imap connection without imapsync?
Q. How can I manually test login using --ssl?
Q. Imapsync used to use SSL_VERIFY_PEER now it uses SSL_VERIFY_NONE.
How can I change this back to the more secure SSL_VERIFY_PEER?
Q: How can I have an imaps server?
Now the questions again with their answers.
=======================================================================
Q. Is running this program a secure method of transferring emails?
Are there any security concerns?
R. Well, it depends. Use encryption and secure access to the host running
imapsync then everything shall be safe.
=======================================================================
Q. Does imapsync refer to SSL security protocols with --ssl1 and --ssl2
and does it refer to TLS security protocols with the --tls1 and --tls2
options?
Short answer: No.
R. No.
Imapsync behaviour:
--ssl: Goes to encryption before the imap session start.
The connection is on port 993.
Can use the TLS or SSL security protocols.
Fails if encryption cannot be established.
--tls: Goes to encryption after the imap session start but before the
credential are sent.
The connection is on port 143.
Can use the TLS or SSL securityprotocols.
Fails if encryption cannot be established.
Explanation:
To force transferring emails over an encrypted connection, you can use
the imapsync parameters --ssl1 and --ssl2 or the parameters --tls1 and
--tls2, they all force the connection to be encrypted by the security
protocols series SSL/TLS.
In a security context, SSL refers to all deprecated Secure Sockets
Layer protocols. TLS refers to the SSL successors, Transport Layer
Security protocols. But TLS 1.0 and TLS 1.1 are also deprecated.
Current endorsed versions of TLS are only TLS 1.2 and TLS 1.3 (July 2021).
The whole story is detailed here:
https://en.wikipedia.org/wiki/Transport_Layer_Security
Options --ssl1 and --ssl2 are a little more paranoid than --tls1 and
--tls2 because they verify that the hostname of the certificate is the
same as the one used by imapsync. Other than that, the security
behavior is the same.
=======================================================================
Q. I noticed that the online UI has no option for TLS/SSL.
Is this secure?
Is this more secure than using the .bat file on my computer?
R1. The online UI does TLS/SSL imap connections if the imap servers
support TLS/SSL.
If you are concerned about security then using the .bat file or .sh on
your computer should be more secure since you can examine and secure
it by yourself, no matter high is your paranoid spirit compared to
mine.
The online UI security is mine, I am concerned by security, not to the
utmost high level possible but I won't give you direct access to the
host to discover my level. With a good guy spirit, feel free to try
to break the online UI security and report me any security issue you
encounter, I'll do my best to fix them as soon as possible. Drop me a
note before starting because I may detect a sort of abuse and ban your
IPs.
=======================================================================
Q. Are transferred emails/attachments stored on any other
server/location aside from my originating/destination server(s)?
R. No!
=======================================================================
Q. Other than changing passwords on the originating/destination email
accounts once the relevant emails have been moved,
are there any other security tips I should know?
R. Yes. Secure the host where imapsync is running since credentials
are on it.
=======================================================================
Q. I need to transfer mail from an imap server to another imap server.
Which ports need to be open on the firewall to make this possible?
R. It depends. Open either:
* port 143 in basic (no special option) or tls mode (--tls1 or --tls2)
* port 993 in ssl mode (--ssl1 or --ssl2)
=======================================================================
Q. Does imapsync support the IMAP command STARTTLS?
R1. Yes.
Use --tls1 and --tls2 options:
--tls1 tells imapsync to use STARTTLS on host1.
--tls2 tells imapsync to use STARTTLS on host2.
R2. Since imapsync release 1.755 STARTTLS mode is activated
automatically when the server announces that it supports it by
listing STARTTLS inside the response to the CAPABILITY command.
If either --notls or --ssl are explicitly mentioned on the
command-line options then STARTTLS won't be done.
=======================================================================
Q. Does imapsync support IMAP over SSL/TLS (IMAPS)?
R. Yes natively since release 1.161.
Still, there are 2 ways, at least, to use ssl:
a) Use native --ssl1 and/or --ssl2 options
--ssl1 tells imapsync to use ssl/tls on host1.
--ssl2 tells imapsync to use ssl/tls on host2.
b) Use stunnel
http://www.stunnel.org/
Use stunnel3 command since stunnel now usually calls
stunnel4 or stunnel5 and the command line options syntax
has changed (option "-c" not recognized for example).
Assuming there is an imaps (993) server on imap.foo.org,
on your localhost machine (or bar machine), run:
stunnel3 -c -d imap -r imap.foo.org:imaps -f
or using numbers instead of names:
stunnel3 -c -d 143 -r imap.foo.org:993 -f
then use imapsync on localhost (or bar machine) imap (143) port.
If the local port 143 is already taken then use a free one,
like 10143 for example.
c) Another example for accessing Gmail with no local root access
to open port 143:
stunnel3 -P '' -c -d 9993 -r imap.gmail.com:993 -f
Then, to access Gmail as host2 use:
imapsync ... --host2 localhost --port2 9993 --nossl2
=======================================================================
Q.How can I test an ssl/tls imap connection without imapsync?
R1. Use either ncat or telnet-ssl or openssl commands like in the
following examples with imap.gmail.com server:
ncat --ssl -C imap.gmail.com 993
telnet-ssl -z ssl imap.gmail.com 993
openssl s_client -crlf -connect imap.gmail.com:993
The previous commands are interactive, hit ctrl-c
to finish them. If you want to finish automatically, then use:
{ sleep 2; echo "A LOGOUT"; sleep 1; } | ncat --ssl -C imap.gmail.com 993
=======================================================================
Q. How can I manually test login using --ssl?
R. Use either ncat or telnet-ssl or openssl commands like in the
following examples with imap.gmail.com server:
ncat --ssl -C imap.gmail.com 993
telnet-ssl -z ssl imap.gmail.com 993
openssl s_client -crlf -connect imap.gmail.com:993
Typical dialog for an imap LOGIN command:
* OK Gimap ready for requests from 78.196.254.58 q1mb175739668wix
A1 LOGIN "gilles.lamiral@gmail.com" "secret"
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE ... ESEARCH
A1 OK gilles.lamiral@gmail.com Gilles Lamiral authenticated (Success)
A2 LOGOUT
* BYE LOGOUT Requested
A2 OK 73 good day (Success)
The client part you have to type is
A1 LOGIN ...
A2 LOGOUT
while replacing ... by your credentials values,
other lines are the server responses.
=======================================================================
Q. How can I test a STARTTLS imap connection without imapsync?
R1. Use openssl command like the following example with
an outlook.office365.com server:
openssl s_client -crlf -starttls imap -connect outlook.office365.com:143
The previous commands are interactive, hit ctrl-c
to finish them. If you want to finish automatically, then use:
{ sleep 2; echo "a logout"; sleep 1; } | openssl s_client -crlf -starttls imap -connect outlook.office365.com:143
Replace outlook.office365.com with your imap server name.
======================================================================
Q. Imapsync used to use SSL_VERIFY_PEER now it uses SSL_VERIFY_NONE.
How can I change this back to the more secure SSL_VERIFY_PEER?
R. After imapsync 1.673,
to set SSL_verify_mode to SSL_VERIFY_PEER on host1
and SSL_verify_mode to SSL_VERIFY_NONE on host2
imapsync ... --ssl1 --ssl2 \
--sslargs1 SSL_verify_mode=1 \
--sslargs2 SSL_verify_mode=0
See "perldoc IO::Socket::SSL" for all possibilities, also at
http://search.cpan.org/perldoc?IO%3A%3ASocket%3A%3ASSL
It might be possible you need an extra option
--sslargs1 SSL_ca_file=/etc/ssl/certs/ca-certificates.crt
to help the ssl software verifying the server certificate.
The file ca-certificates.crt may be elsewhere on your system, even
named differently.
The imap server certificates are not checked for authenticity
by imapsync by default because too many imap servers are crappy
configured regarding certified certificates.
This default behavior is chosen like this because users
want their emails transferred, instead of being not transferred
because of an incompetent imap server sysadmin.
I admit that this part, checking imap ssl/tls certificates,
could be improved from my side by including well known
certificates directly in imapsync.
Drop me a note to encourage me, I'm lazy.
=======================================================================
Q: How can I have an imaps server?
R. Three solutions.
a) Install one
b) or use stunnel :
Assuming there is an imap (143) server on localhost
stunnel -d 993 -r 143 -f
c) or use stunnel on inetd
imaps stream tcp nowait cyrus /usr/sbin/stunnel -s cyrus -p /etc/ssl/certs/imapd.pem -r localhost:imap2
=======================================================================
=======================================================================

77
FAQ.d/FAQ.SmarterMail.txt Normal file
View File

@ -0,0 +1,77 @@
#!/bin/cat
$Id: FAQ.SmarterMail.txt,v 1.15 2021/04/30 13:02:17 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.SmarterMail.txt
=======================================================================
Imapsync tips for SmarterMail. Specific issues and solutions.
=======================================================================
https://imapsync.lamiral.info/README
...
--prefix1 str : Remove prefix str to all destination folders,
usually "INBOX." or "INBOX/" or an empty string "".
imapsync guesses the prefix if host1 imap server
does not have NAMESPACE capability. So this option
should not be used most of the time.
--prefix2 str : Add prefix to all host2 folders. See --prefix1
--sep1 str : Host1 separator. This option should not be used
most of the time.
Imapsync gets the separator from the server itself,
by using NAMESPACE, or it tries to guess it
from the folders listing (it counts
characters / . \\ \ in folder names and choose the
more frequent, or finally / if nothing is found.
--sep2 str : Host2 separator. See --sep1
--noabletosearch : Makes --minage and --maxage options use the internal
dates given by a FETCH imap command instead of the
"Date:" header. Internal date is the arrival date
in the mailbox.
--noabletosearch equals --noabletosearch1 --noabletosearch2
--noabletosearch1 : Like --noabletosearch but for host1.
--noabletosearch2 : Like --noabletosearch but for host2.
--useheader str : Use this header to compare messages on both sides.
Example: Message-ID or Subject or Date.
--useheader str and this one, etc.
=======================================================================
Q. Synchronizing from SmarterMail to XXX
On Unix:
imapsync --host1 imap.d1.org --user1 joe --password1 secret1 \
--host2 imap.d2.org --user2 joe --password2 secret2 \
--prefix1 "" --sep1 "/" --useheader Message-Id --noabletosearch
On Windows:
imapsync.exe --host1 imap.d1.org --user1 joe --password1 secret1 ^
--host2 imap.d2.org --user2 joe --password2 secret2 ^
--prefix1 "" --sep1 "/" --useheader Message-Id --noabletosearch
=======================================================================
Q. Synchronizing from XXX to SmarterMail
On Unix:
imapsync --host1 imap.d1.org --user1 joe --password1 secret1 \
--host2 imap.d2.org --user2 joe --password2 secret2 \
--prefix2 "" --sep2 "/" --useheader Message-Id --noabletosearch
On Windows:
imapsync.exe --host1 imap.d1.org --user1 joe --password1 secret1 ^
--host2 imap.d2.org --user2 joe --password2 secret2 ^
--prefix2 "" --sep2 "/" --useheader Message-Id --noabletosearch
=======================================================================
=======================================================================

37
FAQ.d/FAQ.TTL.txt Normal file
View File

@ -0,0 +1,37 @@
#!/bin/cat
$Id: FAQ.TTL.txt,v 1.2 2018/05/24 11:34:30 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.TTL.txt
=====================================================================
Imapsync tips about TTL when changing name resolution of hosts
=====================================================================
Why decrease the TTL (Time To Live) delay in DNS configuration, down
to 5 minutes?
A small TTL is not mandatory, it's safer and more easy to work with when
migrating.
It's about how long it takes to be sure the users are using the new
imap host, and how long it takes to be sure all new incoming messages
are going right now at the right place. Will you shut down the old
server just after the resolution change? I guess you won't and you'll
be right.
The TTL is just a value, very well supported by machines, with a
little name resolution supplementary work for them when it is set to a
small value like 5 minutes, but it's a tremendous comfort for migrator
people like us.
Be sure to wait 24h after this TTL change before changing any
resolution since the TTL change has to be propagated as well. After
the migration done, no problem to set back the TTL to 24h or more. If
you can't decrease TTL under 4h or even 24h, it's ok anyway, imapsync
can sync the new messages dropped in the old server.
=======================================================================
=======================================================================

104
FAQ.d/FAQ.Two_Ways_Sync.txt Normal file
View File

@ -0,0 +1,104 @@
$Id: FAQ.Two_Ways_Sync.txt,v 1.8 2022/07/25 14:03:49 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Two_Ways_Sync.txt
=======================================================================
================ Two ways sync with Imapsync =========================
=======================================================================
=======================================================================
Q. Can Imapsync do a good "two ways" sync?
Short answer: no, not a good one. Why?
R. Imapsync can't do good two ways syncs.
A good "two ways" sync is impossible with imapsync because imapsync is
stateless.
Each time imapsync runs, it considers messages and folders as if it
were the first time it encounters them. Imapsync looks at messages,
flags, and folders as they are now, not considering what they were
before the current. Imapsync has no memory outside the current running
sync.
So now, why a stateless behavior cannot handle well a two ways sync
between an account A and an account B?
The problem arises with deletions, messages deletions, folders
deletions, or with moves, messages moves across folders, folders
moves, and also folders renamings. Deletions and moves are ambiguous
changes when combined with creations on the opposite side.
For example, if a message is deleted from A by a user, then imapsync
cannot know whether it is a message deleted from A that has to be
deleted in B (what the user actually did) or a missing message on A
that has to be copied from B.
But if you know the answer yourself, that missing messages on one side
A are deleted messages that have to be deleted on the other side then
run a sync with the --delete2 option from A to B. This assumes there
is no new messages on B because --delete2 will delete them.
If you know that the missing messages on A are messages from B
that has to be copied to A, then run a sync from B to A.
If you know it's a mixed scenario, that there are some deletions/moves
on A, and there are some deletions/moves on B, but not the same, then
you are in trouble and it ends up with a not very good "two ways"
sync. In that case, two runs of imapsync, one from A to B, one from B
to A, put the two accounts in a synchronized state. The deletions of
messages on either side get cancelled, deleted messages come back. A
folder renaming on either side brings back the old folder. It's not a
good two ways synchronization.
My suggestion is:
a) Use a single active imap account that fits all your needs,
available from anywhere, at any time. Access it from both your
laptop and your iphone.
b) Feed this active account with foreign and passive accounts with any
tool that can handle the transfer, imapsync being one of them.
c) Have a slave imap backup account of the active one, available also
from anywhere, at any time. Use it only as a failsafe account, when
the active one is broken or to bring back misguidely deleted
messages to the active account. Use any tool that can handle the
synchronization, imapsync being a good one of them.
=======================================================================
Q. What can I do instead of having two ways synchronizations?
R. My suggestion is:
a) Use only one single active imap account that fits all your needs,
available from anywhere, at any time. Access it from all your
devices, laptops, phones, tablets.
b) Feed this active account with foreign and passive accounts with any
tool that can handle the transfer, imapsync being one of them.
c) Have a slave imap backup account of the active one, available also
from anywhere, at any time. Use it only as a failsafe account, when
the active one is broken or to bring back misguidely deleted
messages to the active account. Use any tool that can handle the
synchronization, imapsync being a good one of them.
=======================================================================
Q. What tools can I use to handle two ways synchronizations?
R. There are offlineimap and mbsync/isync tools:
offlineimap is located at http://www.offlineimap.org/
mbsync/isync is located at https://isync.sourceforge.io/
=======================================================================
=======================================================================

37
FAQ.d/FAQ.UCS.txt Normal file
View File

@ -0,0 +1,37 @@
#!/bin/cat
$Id: FAQ.UCS.txt,v 1.8 2021/05/29 08:06:51 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.UCS.txt
=======================================================================
Imapsync tips for UCS. Specific issues.
=======================================================================
UCS: Univention Corporate Server
https://www.univention.com/
UCS was previously Oracle-UCS, Sun JES, IPlanet, etc.
The underlying code was the free open source imap software server Cyrus.
Now UCS uses Dovecot since UCS 4.0-2 (July 2015)
Quoting the page
https://docs.software-univention.de/cyrus-4.2.html
"Since Univention Corporate Server version 4.0-2 Dovecot is used as
the default IMAP and POP3 server."
So, when you deal with UCS issues, follow
https://imapsync.lamiral.info/FAQ.d/FAQ.Cyrus.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Dovecot.txt
Also:
https://imapsync.lamiral.info/FAQ.d/FAQ.APPEND_errors.txt
=======================================================================
=======================================================================

View File

@ -0,0 +1,39 @@
#!/bin/cat
$Id: FAQ.Use_addheader.txt,v 1.6 2018/12/10 20:12:05 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Use_addheader.txt
=======================================================================
Imapsync --addheader suggestion.
=======================================================================
=======================================================================
Q. What means this log message:
"Host1 Sent/1234 ignored (no wanted headers so we ignore this message.
To solve this: use --addheader)"
R. In order to sync messages from one account to another, Imapsync has
to identify messages on both sides, and compare them.
Imapsync can use two completely different methods to identify
messages on each account.
The first one, the default one, uses some headers of messages.
Real world syncs with various imap servers have bring me to use
two headers "Message-Id:" and "Received:". It works for most
messages except for special folders like Sent or Draft where
messages in those folders don't have "Message-Id:" nor
"Received:" headers.
Here comes --addheader option. When a message has no "Message-Id:"
nor "Received:" headers, option --addheader adds a "Message-Id" header
consisting of the imap UID of the message on the host1 folder,
like "Message-Id: 12345@imapsync".
This way, messages are well identified on both sides,
transferred, and only once.
=======================================================================
=======================================================================

174
FAQ.d/FAQ.Use_cache.txt Normal file
View File

@ -0,0 +1,174 @@
$Id: FAQ.Use_cache.txt,v 1.11 2021/04/22 13:20:18 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Use_cache.txt
=======================================================================
Imapsync --usecache or --useuid options
=======================================================================
Questions answered in this FAQ are:
Q. What is --usecache good for?
Q. What is the technical principle of --usecache?
Q. On Windows, with --useuid and after a reboot or a new login, a new
run generates duplicates. What can I do to avoid and fix that?
Q. On Windows, with --useuid or --usecache a problem occurs with long
nested folder names. The error message is:
"No such file or directory; The filename or extension is too long"
Q. Inode problem with --usecache or --useuid on Linux
Now the questions again and their answers.
=======================================================================
Q. What is --usecache good for?
R. The option --usecache is useful to speed up next syncs with --usecache.
The first sync with --usecache between two imap accounts is as low
as the first sync without --usecache, the next syncs are faster.
=======================================================================
Q. What is the technical principle of --usecache?
R. Option --usecache creates a empty file per email message
in order to keep the UIDs mapping between account1 and account2.
The cache is maintened at each run. Messages UIDs enregistred
in the cache are not fetched again to be identified. It really
speeds up the process of syncing.
The cache directory is
<tmpdir>/imapsync_cache/
where <tmpdir> is the temporary directory given by the --tmpdir option
or the one by default, /tmp on Unix, variable on Windows.
=======================================================================
Q. On Windows, with --useuid and after a reboot or a new login, a new
run generates duplicates. What can I do to avoid and fix that?
R. The cache is build in a temporary directory. Imapsync uses a system
call to get this temporary directory, it is not hardcoded and it
is system dependent, the call garanties a write access. Sometimes on
Windows this directory name changes upon reboot or for any other reason.
Anyway, the temporary directory used by imapsync can be set by the
option --tmpdir and that's the way to avoid this issue.
The new problem is that the temporary directory must be writable, which
is not a big deal to found.
Use for example either a) b) or c)
a) The current directory:
imapsync ... --tmpdir .
b) The root directory:
imapsync ... --tmpdir /
c) Any directory you want, like:
imapsync ... --tmpdir "C:\TEMP"
Now, to fix the duplicates on host2, have a run with the option
--delete2duplicates
imapsync ... --delete2duplicates
=======================================================================
Q. On Windows, with --useuid or --usecache a problem occurs with long
nested folder names. The error message is:
"No such file or directory; The filename or extension is too long"
R. This comes from a Windows limitation on pathnames.
No more than 260 characters are allowed for pathnames.
See more details on page
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx#maxpath
The workaround solution given at the previous link,
ie using \\?\D:", does not work for imapsync.
So this imapsync Windows bug is still pending and needs a fix using
a different technique to cache, like a database file for example.
A solution to fix the issue is to use a Linux virtual host on a
Windows box, with VirtualBox or VmWare etc. There is no bug this way,
pathnames can be several thousands characters long.
Better said that done but not so difficult nor expensive these days,
VirtualBox is free and VmWare Player is free for personal or test use.
If you have to stick on Windows, there are two good workarounds
to reduce the cache directory name:
1) Use --tmpdir "D:\\temp" or simply --tmpdir "D:" and imapsync
will build and use the cache in the sub directory
D:\imapsync_cache\
2) add two equivalent entries in the etc/hosts for host1 imap.truc.org
and host2 imap.trac.org.
If you map the ip of imap.truc.org just with the letter a
and same thing for imap.trac.org then you gain characters
etc/hosts
192.168.12.1 a
192.168.55.3 b
Then use:
imapsync --host1 a --host2 b ...
You can get the ip of a host with the ping command line
C:\> ping imap.truc.org
3) A third solution is to not use options --useuid nor --usecache
Fixing this long path problem directly in imapsync is in the TODO file
for a very long time.
=======================================================================
Q. Inode problem with --usecache or --useuid on Linux
R. You may run out of inodes using --usecache, especially with large
migration. Option --usecache creates a empty file per email message
in order to keep the UIDs mapping between account1 and account2.
So, if you plan to sync regularly 10 millions messages over a period
of migration then the filesystem of --tmpdir needs 10 millions of free
inodes. If it hasn't those free inodes then create a new special
filesystem devoted to the imapsync cache.
# Create a file of 10 GB (10 millions*1024):
dd if=/dev/zero of=/var/tmp/fscache bs=1M count=10000
# Create a filesystem where each file is only 1024 bytes per inode:
mkfs.ext2 -F -i 1024 /var/tmp/fscache
# Mount this brand new filesystem
mkdir -p /var/tmp/cachedir
mount -o loop /var/tmp/fscache /var/tmp/cachedir
df -i /var/tmp/cachedir # count inodes available
# Tell imapsync to use it
imapsync ... --tmpdir /var/tmp/cachedir/
# When the migration is over and the cache becomes really useless
umount /var/tmp/cachedir/
rm /var/tmp/fscache
=======================================================================
=======================================================================

View File

@ -0,0 +1,65 @@
#!/bin/cat
$Id: FAQ.User_Concurrent_Access.txt,v 1.6 2022/05/05 13:18:29 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.User_Concurrent_Access.txt
=======================================================================
Imapsync and user concurrent access.
=======================================================================
=======================================================================
Q. What happens if a user access the mailbox during the transfer process?
R. Think all potential problems with a sync between
a master/source/host1 and a slave/destination/host2
What happens when a user access syncing accounts depends
on what he does, where he does and when.
As a summary, reading is safe, writing/deleting is not.
Details:
* Reading: no problem anywhere, at source/host1 or at destination/host2,
anytime. Reading is safe.
* Writing at source/host1:
* adding folder => will need a resync after
* deleting folder => folders deletions are not synced unless --delete2folders
* deleting message => messages deletions are not synced unless --delete2
* adding message => will need a resync
* moving message => will need a resync. Moving is like adding + deleting
so --delete2 is needed to sync the deletion.
* message flag change => will need a resync
Let's more precise about timing.
Imapsync works per folder, one folder at a time.
If imapsync has already synced a folder during a run,
what happens to this folder now won't be synced by the current run.
If imapsync has not already synced a folder during a run,
what happens to this folder now will be synced by the current run.
If imapsync is syncing a folder during a run,
what happens to this folder now depends on when things happen.
Imapsync starts by listing messages and getting meta data about them,
Message-Id, Date, Size.
A message deleted just after the listing, won't be synced.
A new message arriving just after the listing, won't be synced.
* Writing at destination/host2:
* adding folder => no problem, unless --delete2folders is used
* deleting folder => will reappear by a new sync
* adding message => no problem, unless --delete2 is used on next sync.
* deleting message => might reappear by a new sync.
* message flag change => will be reseted by a new sync.
=======================================================================
=======================================================================

View File

@ -0,0 +1,137 @@
#!/bin/cat
$Id: FAQ.Various_Software_Servers.txt,v 1.19 2021/01/26 14:05:23 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Various_Server_Softwares.txt
=======================================================================
Imapsync tips for various imap server softwares.
=======================================================================
=======================================================================
Q. From or to Zimbra
R. Read the FAQ.Zimbra.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Zimbra.txt
=======================================================================
Q. From Emailchemy (https://weirdkid.com/emailchemy/)
R. Use option --noabletosearch :
imapsync ... --noabletosearch
=======================================================================
Q. From or to HMailServer version 4.4.1.
R. You have to add prefix and separator manually because 4.4.1 doesn't
honor the NAMESPACE imap command.
Example for host1:
imapsync ... \
--prefix1 "" --sep1 .
No specific option for HMailServer 5.3.3 since NAMESPACE is supported.
Maybe --subscribe_all will help you to see all migrated folders.
=======================================================================
Q. from Microsoft's Exchange 2007 to Google Apps for your Domain
(GAFYD)
R. Take a look at:
http://mark.ossdl.de/2009/02/migrating-from-exchange-2007-to-google-apps-mail/
=======================================================================
Q. Migrating from or to Parallels Plex Server
R. It depends on the OS
Parallells Plesk Panel for Windows requires --sep2 / --prefix2 ""
Parallells Plesk Panel for Linux works with default parameters.
=======================================================================
Q. Migrating from or to the Cyrus imap server
R. Read the dedicated
https://imapsync.lamiral.info/FAQ.d/FAQ.Cyrus.txt
=======================================================================
Q. Migrating from iPlanet Messaging Server
5.2 Patch 2 (built Jul 14 2004)) to Groupwise 7.0
I encounter many errors like this:
"Error trying to append string: 17847 BAD APPEND"
R. GroupWise 7 seems buggy. Apply GroupWise 7 support pack 1
=======================================================================
Q. Migrating from or to David Tobit
R. See FAQ.David_Tobit.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.David_Tobit.txt
=======================================================================
Q. I need to migrate 1250 mailboxes, passwords are in a MySQL Database.
Can you tell me if your script suits my needs?
R. Mailboxes must exist before running imapsync.
You have to extract users logins and passwords in a csv file.
See the "HUGE MIGRATION" section in the README file.
======================================================================
Q. From GMX IMAP4 StreamProxy
R. Use:
--prefix1 INBOX and --sep1 .
======================================================================
Q. From Courier to Archiveopteryx
R. You can read http://www.archiveopteryx.org/migration/imapsync
Default values might be fine now with latest imapsync.
======================================================================
Q. To Sun Java(tm) System Messaging Server 6.2-7.05
Q. To Communigate Pro - Solaris version
R. See and run patches/imapsync_1.267_jari
======================================================================
Q. From Softalk Workgroup Mail 7.6.4
R. Old Softalk releases don't support the IMAP SEARCH command.
Here are the options to get it working.
imapsync ... --sep1 '.' --prefix1 '' \
--noabletosearch1 --nocheckmessageexists --addheader
(Thanks to Andrew Tucker)
======================================================================
Q. From or to QQMail IMAP4Server
R. imapsync ... --noabletosearch1
======================================================================
Q. From XXX to FTGate
R. Do NOT use --usecache since new UIDs are not given by FTGate and also
badly guessed by imapsync. UIDEXPUNGE does not work so use also
--expunge2 when using --delete2
imapsync ... \
--sep2 / --prefix2 "" \
--useheader Message-Id \
=======================================================================
=======================================================================

74
FAQ.d/FAQ.Virus.txt Normal file
View File

@ -0,0 +1,74 @@
$Id: FAQ.Virus.txt,v 1.9 2020/01/21 21:35:53 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Virus.txt
=======================================================================
Imapsync and virus scanners
=======================================================================
=======================================================================
Q. My virus scanner claims imapsync.exe is a virus/malware/trojan etc.
What the hell?
R1. Yes, I find the same sometimes with few antivirus, some report that
imapsync might have a trojan in it.
Most virus scanners say imapsync.exe and is ok.
I've done this test on imapsync.exe releases
1.977, 1.945, 1.836 on the two following
meta-virus-scanners:
https://www.virustotal.com/
https://www.metadefender.com/
R2. Explanation:
It may come from the fact that imapsync checks whether there is a
new release available at http://imapsync.lamiral.info/VERSION
It's explained here:
https://imapsync.lamiral.info/#NUMBERS
Any other explanation is welcome!
R3. There is no virus alerts reported for the Perl script imapsync
itself. The binary imapsync.exe is just a compiled version of
the script imapsync, with perl interpreter itself and all modules
needed by imapsync in order to make imapsync.exe a standalone software.
R4. Using online meta-virus-scanners
https://www.virustotal.com/
https://www.metadefender.com/
Release 1.977
imapsync.exe (64bit) release 1.977 is reported as not clean:
* https://www.virustotal.com/gui/file/8fd16a5eb2aa4d9a037a528654cc0ecdc63936955d289811dee31c6af474f4b0/detection
SecureAge APEX Malicious Jiangmin Backdoor.Kasidet.qt
imapsync_32bit.exe release 1.977 is reported as totally clean:
https://www.virustotal.com/gui/url/94c89ef31190050bebc522f64e32d3f825ef0418cfc424c5847495b9857d57bf/detection
imapsync_bin_Darwin release 1.977 is reported as totally clean:
* https://www.virustotal.com/gui/file/41f107f4f7c86b15c392aef3c69a7f285f3b2a5f02328a1342903eddc5a5a06f/details
* https://metadefender.opswat.com/results#!/file/bzIwMDEwNUJKb2dXa1N3eWVJSDFoZ1d5clBrZ1U/regular/overview
Release 1.945
imapsync.exe release 1.945 is reported as totally clean:
* https://www.virustotal.com/gui/file/c122fd3bc82c9c784e8b27504e1843f95b1dfd9beba05f50a2893853ad4a6c59/details
* https://metadefender.opswat.com/results#!/file/bzE5MTIwNVN5ZXNDa2NMVEhCeWJpQTFjVVRT/regular/overview
imapsync_64bit.exe release 1.945 is reported as totally clean:
* https://www.virustotal.com/gui/file/f561c6e457c3173f7b0379ee8cc267347feff1bc1991cf47fea7c513c81ccb0a/detection
* https://metadefender.opswat.com/results#!/file/bzE5MDkzMHJ5dk9qb2gxZEhIMXVkaWloeV9C/regular/multiscan
Release 1.836
imapsync.exe release 1.836 is reported as totally clean:
https://www.virustotal.com/#/file/2a33eb93198da77677a4013a5dfb4868cb6aa8f4513e2102a55884b258d19043/detection
https://metadefender.opswat.com/results#!/file/ZTE4MDMxN1MxZTk4aGhxdEdCSlpxVTJoOVlN/regular/information

257
FAQ.d/FAQ.XOAUTH2.txt Normal file
View File

@ -0,0 +1,257 @@
#!/bin/cat
$Id: FAQ.XOAUTH2.txt,v 1.18 2022/04/05 13:48:52 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.XOAUTH2.txt
=======================================================================
Imapsync tips to use XOAUTH2 authentication (Gmail) and old XOAUTH
=======================================================================
Questions answered in this FAQ are:
Q. Is XOAUTH2 authentication available with imapsync to authenticate
my personnal gmail account? (Quick answer: not yet)
Q. Is XOAUTH2 authentication available with imapsync to authenticate
my personnal Office365 account? (Quick answer: not yet)
Q. Is XOAUTH2 authentication available with imapsync to globally
authenticate gmail users, ie as an admin?
Q. Imapsync XOAUTH2 fails with the following message, how to fix that?
Q. How to use XOAUTH2 via a json file to globally authenticate gmail users?
Q. How to use XOAUTH2 via pk12 file to globally authenticate gmail users?
Q. How to use a proxy with XOAUTH2 authentication?
Q. How to use old XOAUTH to globally authenticate gmail users?
Now the questions again with their answers.
=======================================================================
Q. Is XOAUTH2 authentication available with imapsync to authenticate
my personnal gmail account? (Quick answer: not yet)
R. No. But I'm working on that.
XOAUTH2 is a mess or at least, each time I look at it, it leaves
my brain like a mess.
=======================================================================
Q. Is XOAUTH2 authentication available with imapsync to authenticate
my personnal Office365 account? (Quick answer: not yet)
R. No. But I'm working on that too.
=======================================================================
Q. Is XOAUTH2 authentication available with imapsync to globally
authenticate gmail users, ie as an admin?
R. Yes, but XOAUTH2 has been really tested on Unix systems,
less profund on Windows but it should work.
Two file formats are available from Gmail: json and pk12.
json is easier to manage than pk12.
=======================================================================
Q. Imapsync XOAUTH2 fails with the following message, how to fix that?
{
"error": "unauthorized_client",
"error_description": "Unauthorized client or scope in request."
}
R. In order to work you also have to allow the service https://mail.google.com/
in the Google client API manager for OAUTH2.
"Select OAuth 2.0 scopes:"
=======================================================================
Q. How to use XOAUTH2 via a json file to globally authenticate gmail users?
R. Unless you use an imapsync binary like imapsync.exe or imapsync_bin_Darwin,
Perl modules needed for xoauth2 are:
Crypt::OpenSSL::RSA
JSON
JSON::WebToken
LWP
HTML::Entities
Encode::Byte
A easy way to install or upgrade Perl modules is to use cpanm command,
also called cpanminus. On Linux it is something like
sudo cpanm JSON::WebToken JSON Crypt::OpenSSL::RSA LWP HTML::Entities Encode::Byte
The json file patch code and explanation comes from Secretion at
https://github.com/imapsync/imapsync/pull/68
Here is a complete example for Gmail. It is a little stupid
since it is the same account as source and destination but
it's just to get the picture for xoauth2 authentication.
All xoauth2 config is given via the --password1 parameter.
It has the form:
--password1 secret.xoauth2.json
where secret.xoauth2.json is the json file given by Gmail.
imapsync \
--host1 imap.gmail.com --ssl1 --user1 gilles.lamiral@gmail.com \
--password1 secret.xoauth2.json --authmech1 XOAUTH2 \
--host2 imap.gmail.com --ssl2 --user2 gilles.lamiral@gmail.com \
--password2 secret.xoauth2.json --authmech2 XOAUTH2 \
--justlogin --debug
Use your own xoauth2 values.
The secret.xoauth2.json looks like:
{
"type": "service_account",
"project_id": "your-project-name",
"private_key_id": "1cfb..............................bd7fbe",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiGziM...ZV5ACKPHuOfp8A46I=\n-----END PRIVATE KEY-----\n",
"client_email": "jsonfile@your-project-name.iam.gserviceaccount.com",
"client_id": "105................689",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/jsonfile%40your-project-name.iam.gserviceaccount.com"
}
You get this json file by a link like:
https://console.developers.google.com/apis/credentials?project=your-project-name
See also:
https://developers.google.com/gmail/imap/xoauth2-protocol
https://developers.google.com/identity/protocols/OAuth2
=======================================================================
Q. How to use XOAUTH2 via pk12 file to globally authenticate gmail users?
R. First, consider the XOAUTH2 feature at a prototype level.
Perl modules needed for xoauth2 are:
Crypt::OpenSSL::RSA
JSON
JSON::WebToken
LWP
HTML::Entities
Encode::Byte
A easy way to install or upgrade Perl modules is to use cpanm command,
also called cpanminus.
sudo cpanm JSON::WebToken JSON Crypt::OpenSSL::RSA LWP HTML::Entities Encode::Byte
The code and first explanation comes from Joaquin Lopez at
https://github.com/imapsync/imapsync/pull/25
http://linux-france.tk/prj/imapsync_list/msg02129.html
Also, the binary command "openssl" is needed since it is used to
convert the pk12 file.
On Windows I've tried xoauth2 with openssl from
https://slproweb.com/download/Win32OpenSSL-1_0_2d.exe at
https://slproweb.com/products/Win32OpenSSL.html
It works.
Here is a complete example for Gmail. It is a little stupid
since it is the same account as source and destination but
it's just to get the picture for xoauth2 authentication.
All xoauth2 config is given via the --password1 parameter.
It has the form:
--password1 "A;B;C"
where A = 108687549524-gj68fg5ho5icoicv3v79dq2rcuf5c85e@developer.gserviceaccount.com
is the name of the Google Developer API service account.
where B = /g/var/pass/imapsync-xoauth2-15f8456ad5b7_notasecret.p12
is the location of the keyfile associated with it.
where C = notasecret
is the password to access the keyfile.
imapsync \
--host1 imap.gmail.com --ssl1 --user1 gilles.lamiral@gmail.com \
--password1 "108687549524-gj68fg5ho5icoicv3v79dq2rcuf5c85e@developer.gserviceaccount.com;/g/var/pass/imapsync-xoauth2-15f8456ad5b7_notasecret.p12;notasecret" \
--host2 imap.gmail.com --ssl2 --user2 gilles.lamiral@gmail.com \
--password2 "108687549524-gj68fg5ho5icoicv3v79dq2rcuf5c85e@developer.gserviceaccount.com;/g/var/pass/imapsync-xoauth2-15f8456ad5b7_notasecret.p12" \
--justfoldersizes --nofoldersizes \
--authmech1 XOAUTH2 --authmech2 XOAUTH2 --debug
Use your own xoauth2 values.
See also
http://www.notearthday.org/nedtech/2016/05/creating-creating-oauth2-credentials-with-google-apps/
=======================================================================
Q. How to use a proxy with XOAUTH2 authentication?
With imapsync 1.670, you have to set two environment variables
PERL_LWP_ENV_PROXY and https_proxy. Example:
PERL_LWP_ENV_PROXY=1 https_proxy=http://myproxy:8080/ imapsync --host1 ...
With later release than 1.670, you have to set only the https_proxy
environment variable, if it isn't already set. Example:
https_proxy=http://myproxy:8080/ imapsync --host1 ...
=======================================================================
Q. How to use old XOAUTH to globally authenticate gmail users?
R0. XOAUTH is considered obsolete and superseded by XOAUTH2
Anyway the manage part might be the same (I don't know).
R1. The XOAUTH code and this FAQ item come from Eduardo Bortoluzzi
Thanks Eduardo!
R2. In case you still have to use XOAUTH, here is the method:
The goal of OAUTH is to migrate all users from/to Google Apps
Premier Edition without knowing their passwords.
The global password is available at the Google Apps control panel,
at Advanced Tools -> Manage OAuth domain key.
./imapsync \
--host1 imap.gmail.com --ssl1 \
--user1 foo@lab3.dedal.br \
--password1 secret1 \
--authmech1 XOAUTH \
--host2 imap.gmail.com --ssl2 \
--user2 bar@lab3.dedal.br \
--password2 secret2 \
--authmech2 XOAUTH
Google Apps is a paid service, but you can try it for 30 days without any cost,
or you could try, time goes on on free trial offers.
Some notes about configuring the Google Apps XOAUTH:
On "Advanced Tools > Manage OAuth domain key > Two-legged OAuth access control"
the "Allow access to all APIs" must be checked
(https://support.google.com/a/bin/answer.py?answer=162105)
OR
On "Advanced Tools > Manage third party OAuth client access",
the configured costumer key must have the scope
"https://mail.google.com/" configured
(https://support.google.com/a/bin/answer.py?answer=162106).
=======================================================================
=======================================================================

105
FAQ.d/FAQ.Yahoo.txt Normal file
View File

@ -0,0 +1,105 @@
$Id: FAQ.Yahoo.txt,v 1.15 2022/05/05 14:17:22 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Yahoo.txt
=======================================================================
Imapsync tips for Yahoo.
=======================================================================
Questions answered in this FAQ are:
Q. Synchronizing from Yahoo to XXX
Q. Synchronizing from XXX to Yahoo
Now the questions again with their answers.
=======================================================================
Q. Synchronizing from Yahoo to XXX
R. Use --host1 imap.mail.yahoo.com
Example:
./imapsync \
--host1 imap.mail.yahoo.com \
--user1 Billy@yahoo.com \
--password1 billysecreta \
--host2 XXX \
--user2 Billy@other.com \
--password2 billysecretb
You also need to go to Yahoo, security and enable
"Allow less secure apps to login",
otherwise the login will not work.
To enable less secure apps to login:
* Login to the Yahoo mail account,
* click on the account name or the avatar and select "Account Info",
* click on "Account security",
* turn off "Two steps verification"
* turn on "Allow apps that use less secure authentication".​
Thanks to Eugen Mayer for this last point.
Another solution:
* Login to the Yahoo mail account,
* click on the account name or the avatar and select "Account Info",
* click on "Account security",
* Turn on "Two-step verification"
* Click on "Manage app passwords" and
generate a specific password for imapsync,
choose "Other app" at the bottom and type imapsync
since it is not in the predefined apps.
* Use this password with imapsync.
=======================================================================
Q. Synchronizing from XXX to Yahoo
R. Example:
On Linux:
./imapsync \
--host1 XXX \
--user1 Billy@other.com \
--password1 billysecreta \
--host2 imap.mail.yahoo.com \
--user2 Billy@yahoo.com \
--password2 billysecretb \
--regexmess 's{(?<![\n])\z}{\r\n}gxms'
On Windows:
imapsync.exe \
--host1 XXX \
--user1 Billy@other.com \
--password1 billysecreta \
--host2 imap.mail.yahoo.com \
--user2 Billy@yahoo.com \
--password2 billysecretb \
--regexmess "s{(?<![\n])\z}{\r\n}gxms"
You wonder why there is this ugly regex, what it does and where does it
come from?
It's there to get rid of the errors
"BAD [CLIENTBUG] Additional arguments found after last expected argument"
when copying some messages to Yahoo.
It adds a final \r\n on a message if a final \r\n is missing.
See https://github.com/imapsync/imapsync/issues/209
https://imapsync.lamiral.info/README
...
--regexmess reg : Apply the whole regex to each message before transfer.
Example: 's/\000/ /g' # to replace null by space.
--regexmess reg : and this one, etc.
=======================================================================
=======================================================================

26
FAQ.d/FAQ.Yandex.txt Normal file
View File

@ -0,0 +1,26 @@
#!/bin/cat
$Id: FAQ.Yandex.txt,v 1.1 2019/09/12 10:08:05 gilles Exp gilles $
This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Kerio.txt
=======================================================================
Imapsync tips for Yandex accounts.
=======================================================================
=======================================================================
Q. How to migrate from or to Yandex
R. Use:
imapsync ... --useheader Message-Id --noabletosearch
The option --noabletosearch is needed only if options --maxage or --minage
are used. I don't know which --search commands could work, maybe none.
The imap server for --host1 or --host2 is usually imap.yandex.ru

83
FAQ.d/FAQ.Zimbra.txt Normal file
View File

@ -0,0 +1,83 @@
#!/bin/cat
$Id: FAQ.Zimbra.txt,v 1.11 2021/04/18 09:15:45 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Zimbra.txt
=======================================================================
Imapsync tips for Zimbra. Specific issues and solutions.
=======================================================================
Please, don't follow
https://wiki.zimbra.com/wiki/Guide_to_imapsync
It's obsolete and it will give you, and me, more difficulties than
doing a standard sync without additional options.
Questions answered in this FAQ are:
Q. How to migrate from Zimbra to XXX?
Q. How to migrate from Zimbra with an admin account?
Q. I got this "NO APPEND failed" error:
could not append ( Subject:... Flags:[\Flagged \Seen EmailswithStickyNotes:FLAG51135] )
to folder XXX: 316 NO APPEND failed
What can I do?
Q. Zimbra disconnects without reason
Could not select: * BYE ... Zimbra IMAP4rev1 server closing connection
Now the questions again with their answers.
=======================================================================
Q. How to migrate from Zimbra to XXX?
The basic default (only credendials) should work fine with Zimbra.
=======================================================================
Q. How to migrate from Zimbra with an admin account?
R. Use:
imapsync ... --user1 "normal_user" --authuser1 "admin_user" --password1 "admin_user_password"
To setup or use a Zimbra admin user see:
https://zimbra.github.io/adminguide/8.8.9/index.html#_administrator_accounts
Thanks to Richard Street from thinkround for this tip.
=======================================================================
Q. I got this "NO APPEND failed" error:
could not append ( Subject:... Flags:[\Flagged \Seen EmailswithStickyNotes:FLAG51135] )
to folder XXX: 316 NO APPEND failed
What can I do?
R1.
https://zimbra.github.io/adminguide/8.8.10/index.html
Flagging
Users can create flags and assign them to messages, contacts, and files in Briefcase folders.
In the Admin Console, see COS -> Features -> Mail Features container
Is Flagging on on host2?
If yes, see https://imapsync.lamiral.info/FAQ.d/FAQ.Flags.txt
=======================================================================
Q. Zimbra disconnects without reason
Could not select: * BYE ... Zimbra IMAP4rev1 server closing connection
R. The cause of this problem may be that some users have a subscription
of a shared folder from a mailbox locked and so the shared folder.
The access problems may happen in a lot of folders,
not only the shared/locked one.
It should work after removing the locked/shared folder.
Thanks to Danny Herrlitz for this input!
=======================================================================
=======================================================================

24
FAQ.d/FAQ.iCloud.txt Normal file
View File

@ -0,0 +1,24 @@
#!/bin/cat
# $Id: FAQ.iCloud.txt,v 1.1 2022/04/13 11:44:19 gilles Exp gilles $
This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.iCloud.txt
=======================================================================
Imapsync tips for iCloud accounts.
=======================================================================
Q. How to migrate from or to an iCloud account?
R. From https://support.apple.com/en-us/HT202304
Use "imap.mail.me.com" as the imap server and use an app password.
How to generate an app password is described at
https://support.apple.com/kb/HT204397
It's not necessary to precise the port and the ssl thing, imapsync
auto-detect this.

82
FAQ.d/htaccess.txt Normal file
View File

@ -0,0 +1,82 @@
# $Id: htaccess.txt,v 1.33 2022/04/06 10:27:52 gilles Exp gilles $
AddDescription "<b>Back</b> to Imapsync main page." ..
AddDescription "<b>Where this Description column comes from</b>." htaccess.txt
AddDescription "<b>Private</b>." memo
AddDescription "<b>APPEND errors</b>." FAQ.Admin_Authentication.txt
AddDescription "<b>Authenticate via an admin account</b>." FAQ.APPEND_errors.txt
AddDescription "<b>Archiving</b> tips." FAQ.Archiving.txt
AddDescription "<b>Authentication failures</b>." FAQ.Authentication_failure.txt
AddDescription "<b>Bandwidth questions</b> tips." FAQ.Bandwidth.txt
AddDescription "<b>Big mailboxes</b> tips." FAQ.Big_Mailbox.txt
AddDescription "<b>Changing folders names</b>." FAQ.Folders_Mapping.txt
AddDescription "<b>Connection issues</b>." FAQ.Connection.txt
AddDescription "<b>Contacts & Calendars & Chat issues</b>." FAQ.Contacts_Calendars.txt
AddDescription "<b>Cyrus</b> accounts." FAQ.Cyrus.txt
AddDescription "<b>Dates issues</b>." FAQ.Dates.txt
AddDescription "<b>David Tobit</b> accounts." FAQ.David_Tobit.txt
AddDescription "<b>DBmail</b> accounts." FAQ.DBmail.txt
AddDescription "<b>Docker</b> usage." FAQ.Docker.txt
AddDescription "<b>Domino</b> accounts." FAQ.Domino.txt
AddDescription "<b>Dovecot</b> accounts." FAQ.Dovecot.txt
AddDescription "<b>Duplicated</b> messages issues." FAQ.Duplicates.txt
AddDescription "<b>Emptying</b> an account." FAQ.Emptying.txt
AddDescription "<b>Exchange 20xx</b> and <b>Office365</b> accounts." FAQ.Exchange.txt
AddDescription "<b>Selecting folders</b>." FAQ.Folders_Selection.txt
AddDescription "<b>Folders sizes with Imapsync</b>." FAQ.Folders_Sizes.txt
AddDescription "<b>FirstClass</b> accounts." FAQ.FirstClass.txt
AddDescription "<b>Flags tips and issues</b>." FAQ.Flags.txt
AddDescription "<b>General Data Protection Regulation</b> (GDPR)." FAQ.GDPR.txt
AddDescription "<b>General</b> and <b>pot-pourri</b> issues" FAQ.General.txt
AddDescription "<b>Gmail</b> accounts." FAQ.Gmail.txt
AddDescription "<b>GoDaddy</b> accounts." FAQ.GoDaddy.txt
AddDescription "<b>IceWarp</b> accounts." FAQ.IceWarp.txt
AddDescription "<b>ISP tips</b>." FAQ.ISP.txt
AddDescription "<b>James</b> accounts." FAQ.James.txt
AddDescription "<b>Kerio</b> accounts." FAQ.Kerio.txt
AddDescription "<b>MailEnable</b> accounts." FAQ.MailEnable.txt
AddDescription "<b>Massive and bulk</b> migrations." FAQ.Massive.txt
AddDescription "<b>Log file, content, errors and tips</b>." FAQ.Logfile.txt
AddDescription "<b>Memory</b> issues." FAQ.Memory.txt
AddDescription "Migration <b>Plan</b>." FAQ.Migration_Plan.txt
AddDescription "<b>Office365</b> accounts." FAQ.Office365.txt
AddDescription "<b>The Online UI service Q&R</b>." FAQ.OnlineUI.txt
AddDescription "<b>Password & special characters on Windows</b>." FAQ.Passwords_on_Windows.txt
AddDescription "<b>Password & special characters on Unix</b>." FAQ.Passwords_on_Unix.txt
AddDescription "<b>Password & special characters on Mac OS X</b>." FAQ.Passwords_on_Mac.txt
AddDescription "<b>POP3 issues</b>." FAQ.POP3.txt
AddDescription "<b>Principles & design decisions</b>." FAQ.Principles.txt
AddDescription "<b>Selecting messages</b>." FAQ.Messages_Selection.txt
AddDescription "<b>How to transfer too big messages</b>." FAQ.Messages_Too_Big.txt
AddDescription "Why the imapsync web site <b>looks so old?</b>" FAQ.Old_Style_Web_Design.txt
AddDescription "<b>UCS</b> accounts." FAQ.UCS.txt
AddDescription "<b>Checklist before release a new release</b>." FAQ.Release_Checklist.txt
AddDescription "<b>Guidelines to report bugs</b>." FAQ.Reporting_Bugs.txt
AddDescription "<b>Security</b>." FAQ.Security.txt
AddDescription "<b>SSL errors</b>." FAQ.SSL_errors.txt
AddDescription "<b>SmarterMail</b> accounts." FAQ.SmarterMail.txt
AddDescription "<b>TTL (Time To Live) delay in DNS configuration</b>." FAQ.TTL.txt
AddDescription "<b>Why Imapsync can't do two ways syncs</b>." FAQ.Two_Ways_Sync.txt
AddDescription "<b>Option --usecache and inodes</b>." FAQ.Use_cache.txt
AddDescription "<b>User concurrent access</b>." FAQ.User_Concurrent_Access.txt
AddDescription "<b>Virus scanners on Imapsync</b>." FAQ.Virus.txt
AddDescription "<b>Why use --addheader?</b>." FAQ.Use_addheader.txt
AddDescription "<b>Various imap software servers</b>." FAQ.Various_Software_Servers.txt
AddDescription "<b>XOAUTH2</b> (<b>Gmail</b>)." FAQ.XOAUTH2.txt
AddDescription "<b>Yahoo</b> accounts." FAQ.Yahoo.txt
AddDescription "<b>Yandex</b> accounts." FAQ.Yandex.txt
AddDescription "<b>Zimbra</b> accounts." FAQ.Zimbra.txt

4
FAQ.d/memo Normal file
View File

@ -0,0 +1,4 @@
( cd .. && make upload_FAQ )

218
INSTALL
View File

@ -1,218 +0,0 @@
# $Id: INSTALL,v 1.30 2013/07/03 04:13:52 gilles Exp gilles $
#
# INSTALL file for imapsync
# imapsync : IMAP sync or copy tool.
INTRODUCTION
------------
imapsync works fine under any operating system with Perl and Perl modules (listed below).
imapsync.exe works fine standalone under Windows XP, Vista, Seven, 20XX, either 32 or 64bit.
UNIX
----
Purchase imapsync at
http://imapsync.lamiral.info/
You'll have access to a compressed tarball called imapsync-x.xx.tgz
where x.xx is the version number. Untar the tarball where
you want (on Unix):
tar xzvf imapsync-x.xx.tgz
Go into the directory imapsync-x.xx and read the INSTALL file.
You're already reading the INSTALL file.
WINDOWS
-------
a) Simplest way:
- Buy imapsync.exe at http://imapsync.lamiral.info/
- Run imapsync.exe in a command prompt (execute cmd.exe).
- Or simpler, look at the batch file at
http://imapsync.lamiral.info/examples/imapsync_example.bat
and replace the parameters with your values
then run the batch file with a double-click.
It is simpler to edit a batch file with notepad than
the command line in dos mode.
b) Hard way:
- Get imapsync-x.xx.tgz
- Install Perl if it isn't already installed.
Strawberry Perl is a very good candidate
http://strawberryperl.com/
- Use the command CPAN to install modules listed in the PREREQUISITES section.
There is also a batch file that does this install for you
It is called install_modules.bat available at
http://imapsync.lamiral.info/examples/install_modules.bat
c) How to build imapsync.exe?
- Do the hard stuff in b)
- Run build_exe.bat (found in the tarball)
PREREQUISITES
-------------
This section doesn't concern Windows imapsync.exe users.
You need :
- Perl
try : perl -v
Get Perl here:
http://www.perl.org/
- Some perl modules. To verify you have
all modules in one command, just run:
perl -c imapsync
The good answer is "imapsync syntax OK".
Here is some individual module help:
- Perl Mail::IMAPClient module
try :
perl -mMail::IMAPClient -e ''
Get the latest Mail::IMAPClient module here:
http://search.cpan.org/dist/Mail-IMAPClient/
I use always the latest Mail-IMAPClient-3.xx
To know the version you have on your system try :
perl -mMail::IMAPClient -e 'print $Mail::IMAPClient::VERSION, "\n"'
New Mail-IMAPClient-3.xx works very well with imapsync,
Use at least Mail-IMAPClient-3.25 (previous may bug).
Don't hesitate to use latest Mail-IMAPClient-3.xx
(3.xx >= 3.33 at the time of this writing)
Look at the script named "i3" in the tarball, it can be used to
run imapsync with included Mail-IMAPClient-3.33/ wherever you
unpacked the imapsync tarball.
- Perl Digest::MD5 module.
try:
perl -mDigest::MD5 -e ""
http://search.cpan.org/
http://search.cpan.org/~gaas/Digest-MD5-2.52/
To know the version you have on your system try :
perl -mDigest::MD5 -e 'print $Digest::MD5::VERSION, "\n"'
I use 2.39 (Ubuntu package)
- Term::ReadKey
perl -mTerm::ReadKey -e ""
- IO::Socket::SSL
perl -mIO::Socket::SSL -e ""
- File::Spec
perl -mFile::Spec -e ""
- File::Path
perl -mFile::Path -e ""
- Perl Digest::HMAC_MD5 module
Good for non plain text password over network.
perl -mDigest::HMAC_MD5 -e ""
- Perl Authen::NTLM
perl -mAuthen::NTLM -e ""
- Perl Time::HiRes
perl -mTime::HiRes -e ""
- Perl Data::Uniqid
perl -mData::Uniqid -e ""
- Perl URI::Escape
perl -mURI::Escape -e ""
Test everything in one command:
perl -mMail::IMAPClient -mDigest::MD5 -mTerm::ReadKey -mIO::Socket::SSL \
-mFile::Spec -mDigest::HMAC_MD5 -mAuthen::NTLM -e -mTime::HiRes \
-mData::Uniqid -mURI::Escape ""
You can install easily those Perl modules in latest release via the
following commands (with root permissions)
perl -MCPAN -e "install Mail::IMAPClient"
perl -MCPAN -e "install Digest::MD5"
perl -MCPAN -e "install Term::ReadKey"
perl -MCPAN -e "install IO::Socket::SSL"
perl -MCPAN -e "install File::Spec"
perl -MCPAN -e "install Digest::HMAC_MD5"
perl -MCPAN -e "install Authen::NTLM"
perl -MCPAN -e "install Time::HiRes"
perl -MCPAN -e "install Data::Uniqid"
perl -MCPAN -e "install URI::Escape"
You can install them easily too by using the standard install
command on your system if the packages have been made on it
(so you may not have the latest but most of the time it
doesn't care).
Modules Authen::NTLM, Data::Uniqid may be not available
this way since I haven't find them on Ubuntu.
Example on Debian/Ubuntu:
aptitude install libmail-imapclient-perl # Mail::IMAPClient
aptitude install libdigest-md5-file-perl # Digest::MD5
aptitude install libterm-readkey-perl # Term::ReadKey
aptitude install libio-socket-ssl-perl # IO::Socket::SSL
aptitude install libfile-spec-perl # File::Spec
aptitude install libdigest-hmac-perl # Digest::HMAC_MD5 Digest::HMAC_SHA1
INSTALLING on Unix
------------------
To see what will be done, just run:
make -n install
To install imapsync in /usr/bin/ and man page in /usr/share/man/man1/, just run (as root):
make install
or copy the file imapsync where you want it to be.
TESTING on Unix
---------------
Tests will break as they are home specific.
You need a running imap server on localhost with several accounts
toto with a password located in the file $HOME/var/pass/secret.toto
titi with a password located in the file $HOME/var/pass/secret.titi
tata with a password located in the file $HOME/var/pass/secret.tata
Of course, you can change the file tests.sh and run tests with:
sh -x tests.sh
Specific tests can be run by using them as argument to tests.sh:
sh -x tests.sh ll_ssl_justconnect ll_bad_host ...
The tests.sh script break on first failure ("set -e" directive).
Since I added more tests you also need a remote imap server see the
file tests.sh and help yourself. No time to make a universal test
file.

1
INSTALL Symbolic link
View File

@ -0,0 +1 @@
INSTALL.d/INSTALL.ANY.txt

4
INSTALL.d/.dockerignore Normal file
View File

@ -0,0 +1,4 @@
#
memo
RCS

113
INSTALL.d/Dockerfile Normal file
View File

@ -0,0 +1,113 @@
## Dockerfile for building a docker imapsync image
# $Id: Dockerfile,v 1.43 2022/05/17 14:29:36 gilles Exp gilles $
# I use the following command to build the image:
#
# docker build -t gilleslamiral/imapsync .
#
# where this Dockerfile is in the current directory
#
# This Dockerfile build an image with two imapsync
# 1) One comes from https://imapsync.lamiral.info/imapsync
# and goes to /usr/bin/imapsync in the Docker image
# It is used with the command:
#
# docker run gilleslamiral/imapsync imapsync ...
#
# 2) One comes from the local file ./imapsync
# and goes to /imapsync in the Docker image
# It is used with the command:
#
# docker run gilleslamiral/imapsync /imapsync ...
# I thank you very much
# I like thanks
# I like stars
# I like money to keep on doing this stuff
# Number of imapsync images pulled so far (2022_04_04): 673551
# Command used:
# curl -s https://hub.docker.com/v2/repositories/gilleslamiral/imapsync/ | jq '.pull_count'
# Debian Bullseye is Debian 11
FROM debian:bullseye
LABEL maintainer="Gilles LAMIRAL <gilles@lamiral.info>" \
description="Imapsync" \
documentation="https://imapsync.lamiral.info/#doc"
# I put a copy of the Dockerfile in the image itself
# It can help maintenance, isn't it?
# Also put optionally my local and usually more recent imapsync on /, for testing purpose
COPY Dockerfile imapsyn[c] prerequisites_imapsyn[c] /
RUN set -xe && \
apt-get update \
&& apt-get install -y \
libauthen-ntlm-perl \
libcgi-pm-perl \
libcrypt-openssl-rsa-perl \
libcrypt-openssl-pkcs12-perl \
libdata-uniqid-perl \
libencode-imaputf7-perl \
libfile-copy-recursive-perl \
libfile-tail-perl \
libio-compress-perl \
libio-socket-ssl-perl \
libio-socket-inet6-perl \
libio-tee-perl \
libhtml-parser-perl \
libjson-webtoken-perl \
libmail-imapclient-perl \
libparse-recdescent-perl \
libmodule-scandeps-perl \
libpar-packer-perl \
libproc-processtable-perl \
libreadonly-perl \
libregexp-common-perl \
libsys-meminfo-perl \
libterm-readkey-perl \
libtest-mockobject-perl \
libtest-pod-perl \
libunicode-string-perl \
liburi-perl \
libwww-perl \
procps \
wget \
make \
cpanminus \
lsof \
ncat \
openssl \
ca-certificates \
&& rm -rf /var/lib/apt/lists/* \
&& cpanm IO::Socket::SSL
# I added the IO::Socket::SSL update to avoid the annoying, confusing and useless warning
# DEBUG: .../IO/Socket/SSL.pm:1177: global error: Undefined SSL object
RUN set -xe \
&& cd /usr/bin/ \
&& pwd \
&& wget -N --no-check-certificate https://imapsync.lamiral.info/imapsync \
https://imapsync.lamiral.info/prerequisites_imapsync \
https://raw.githubusercontent.com/google/gmail-oauth2-tools/master/python/oauth2.py \
&& chmod +x imapsync oauth2.py \
&& /usr/bin/imapsync --testslive && /usr/bin/imapsync --tests # just_a_comment_to_force_update 2022_04_04_21_16_50
USER nobody:nogroup
ENV HOME /var/tmp/
WORKDIR /var/tmp/
STOPSIGNAL SIGINT
CMD ["/usr/bin/imapsync"]
#
# End of imapsync Dockerfile

209
INSTALL.d/INSTALL.ANY.txt Normal file
View File

@ -0,0 +1,209 @@
# $Id: INSTALL.ANY.txt,v 1.62 2020/06/03 07:11:10 gilles Exp gilles $
#
# This is the main INSTALL file for imapsync.
# imapsync : IMAP sync and migrate tool.
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.ANY.txt
=======================================================================
INTRODUCTION
=======================================================================
imapsync works fine under any operating system with Perl and Perl modules (listed below).
imapsync.exe works fine standalone under Windows XP, Vista, Seven, 20XX, either 32 or 64bit.
imapsync_64bit.exe works fine standalone under Seven, 20XX, but only 64bit.
=======================================================================
Installing imapsync on WINDOWS
=======================================================================
Read the file README_Windows.txt
Also available at
https://imapsync.lamiral.info/README_Windows.txt
=======================================================================
Installing imapsync on Unix
=======================================================================
There are specific INSTALL files in the imapsync directory INSTALL.d/
Those documents are also available at https://imapsync.lamiral.info/INSTALL.d/
- Mac OS X
- FreeBSD
- CentOS
- CPanel
- Debian
- Ubuntu
- AWS EC2
- Arch Linux
- Docker
- Online UI like /X
If you are not familiar with any of these systems but you have to choose
one to install imapsync, imapsync online, or the latest imapsync then
Centos 7 (not 8 for now) is the easiest system to consider (June 2020).
If you are not on one of these systems then read the last section
below called "Installing imapsync on other Unixes".
=======================================================================
Installing imapsync on Mac OS X
=======================================================================
Easy.
Read the file INSTALL.d/INSTALL.Darwin.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Darwin.txt
=======================================================================
Installing imapsync on FreeBSD
=======================================================================
Easy.
Read the file INSTALL.d/INSTALL.FreeBSD.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.FreeBSD.txt
=======================================================================
Installing imapsync on CentOS
=======================================================================
Easy.
Read the file INSTALL.d/INSTALL.Centos.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Centos.txt
=======================================================================
Installing imapsync on CPanel
=======================================================================
Easy.
Read the file INSTALL.d/INSTALL.CPanel.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.CPanel.txt
=======================================================================
Installing imapsync on Debian
=======================================================================
Not so easy.
See the file INSTALL.d/INSTALL.Debian.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Debian.txt
=======================================================================
Installing imapsync on Ubuntu
=======================================================================
Not so easy.
See the file INSTALL.d/INSTALL.Ubuntu.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Ubuntu.txt
=======================================================================
Installing imapsync on AWS
=======================================================================
Not so easy.
See the file INSTALL.d/INSTALL.AWS_EC2.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.AWS_EC2.txt
=======================================================================
Installing imapsync on ArchLinux
=======================================================================
Not so easy.
See the file INSTALL.d/INSTALL.ArchLinux.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.ArchLinux.txt
=======================================================================
Installing imapsync online like /X
=======================================================================
Not easy.
Follow the document INSTALL.d/INSTALL.OnlineUI.txt
This document is also available at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.OnlineUI.txt
=======================================================================
Installing imapsync on other Unixes
=======================================================================
Not easy.
Purchase imapsync at
https://imapsync.lamiral.info/
or get it anywhere.
You have access to a compressed tarball called imapsync-1.xxx.tgz
where 1.xxx is the version number. Untar the tarball where
you want:
cd
tar xzvf imapsync-1.xxx.tgz
Go into the directory imapsync-1.xxx
cd imapsync-1.xxx
You can easily detect any missing Perl modules via the
script prerequisites_imapsync located in the INSTALL.d directory:
sh INSTALL.d/prerequisites_imapsync
or
cd INSTALL.d/
sh prerequisites_imapsync
You don't need to be root to run the previous command. You have
to be root if you want the Perl modules to be available for the
whole system, for all users.
You may be in one of following cases:
- you are not root.
- you are in a environment where modifying system Perl
modules can break other things or where you're not allowed to
change the whole system.
The "make" command is a prerequisite to build some Perl modules.
Install make in case it is not already installed.
Run the "cpanm" command with the missing Perl modules as arguments.
For example it can be:
cpanm Authen::NTLM Data::Uniqid File::Copy::Recursive IO::Tee \
Mail::IMAPClient Unicode::String Encode::IMAPUTF7
Once you've run the "cpanm" command, you can rerun "sh prerequisites_imapsync"
to verify everything is ok:
sh prerequisites_imapsync
When everything is ok the script execution ends with this sentence
"All needed modules are already installed"
Now imapsync should work on your system. To check it run:
./imapsync --testslive
=======================================================================
=======================================================================

View File

@ -0,0 +1,117 @@
#!/bin/cat
# $Id: INSTALL.AWS_EC2.txt,v 1.9 2022/05/17 14:32:47 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.AWS_EC2.txt
=======================================================================
Installing imapsync on AWS EC2
=======================================================================
AWS: Amazon Web Service
A) cat /etc/system-release
Amazon Linux release 2 (Karoo)
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install imapsync
imapsync --version
1.727
To install a more recent imapsync
sudo yum install perl-App-cpanminus \
perl-Dist-CheckConflicts \
perl-HTML-Parser \
perl-libwww-perl \
perl-Module-Implementation \
perl-Module-ScanDeps \
perl-Package-Stash \
perl-Package-Stash-XS \
perl-PAR-Packer \
perl-Regexp-Common \
perl-Sys-MemInfo \
perl-Test-Fatal \
perl-Test-Mock-Guard \
perl-Test-Requires \
perl-Test-Deep \
perl-File-Tail \
perl-Unicode-String \
perl-Test-NoWarnings \
perl-Test-Simple \
perl-Test-Warn \
perl-Sub-Uplevel \
perl-Proc-ProcessTable
sudo cpanm Encode::IMAPUTF7
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
./imapsync --testslive
./imapsync --version # command typed on 14 April 2020
1.983
sudo cpanm CGI
B) Old Amazon Linux AMI release 2017.03
cat /etc/system-release
Amazon Linux AMI release 2017.03
sudo yum install \
perl-Class-Load \
perl-IO-Compress \
perl-Crypt-OpenSSL-RSA \
perl-Data-Dumper \
perl-Dist-CheckConflicts \
perl-ExtUtils-Embed \
perl-File-Copy-Recursive \
perl-File-Tail \
perl-IO-Socket-INET6 \
perl-IO-Socket-SSL \
perl-JSON \
perl-HTML-Parser \
perl-libwww-perl \
perl-Mail-IMAPClient \
perl-Module-Implementation \
perl-Module-Runtime \
perl-Module-ScanDeps \
perl-Net-SSLeay \
perl-Package-Stash \
perl-Package-Stash-XS \
perl-Parse-RecDescent \
perl-Readonly \
perl-Regexp-Common \
perl-Sys-MemInfo \
perl-TermReadKey \
perl-Test-Fatal \
perl-Test-MockObject \
perl-Test-Simple \
perl-Test-Pod \
perl-Test-Requires \
perl-Try-Tiny \
perl-Unicode-String \
perl-URI \
perl-Proc-ProcessTable \
cpanminus
sudo cpanm Authen::NTLM Data::Uniqid IO::Tee \
JSON::WebToken JSON::WebToken::Crypt::RSA \
Test::Mock::Guard \
Encode::IMAPUTF7
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
./imapsync
./imapsync --testslive
=======================================================================
=======================================================================

View File

@ -0,0 +1,95 @@
#!/bin/cat
# $Id: INSTALL.ArchLinux.txt,v 1.6 2022/05/17 14:31:44 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.ArchLinux.txt
==========================================
=== Installing imapsync on ArchLinux ===
==========================================
Thanks to Aldo Villagra!
---------------------------------
With yaourt and AUR repositories
---------------------------------
pacman -S --needed base-devel git
git clone https://aur.archlinux.org/package-query.git
cd package-query
makepkg -si
cd ..
git clone https://aur.archlinux.org/yaourt.git
cd yaourt
makepkg -si
cd ..
After you have installed Yaourt, you can install imapsync:
yaourt -S --needed imapsync
That's all folks!
-----------------------------------------------
With the "pacman" and the standard repositories
community/
extra/
-----------------------------------------------
Commands to run:
pacman -S --needed make lsb-release cpanminus wget
pacman -S --needed community/perl-cgi \
extra/perl-crypt-openssl-rsa \
extra/perl-data-uniqid \
extra/perl-digest-hmac \
community/perl-dist-checkconflicts \
extra/perl-file-copy-recursive \
extra/perl-io-socket-inet6 \
extra/perl-io-socket-ssl \
community/perl-io-tee \
community/perl-json \
extra/perl-html-parser \
extra/perl-libwww \
community/perl-module-implementation \
community/perl-module-runtime \
community/perl-module-scandeps \
extra/perl-net-ssleay \
community/perl-package-stash \
community/perl-package-stash-xs \
community/perl-parse-recdescent \
community/perl-readonly \
community/perl-regexp-common \
extra/perl-term-readkey \
community/perl-test-fatal \
community/perl-test-mockobject \
extra/perl-test-pod \
community/perl-test-requires \
community/perl-test-nowarnings \
community/perl-test-deep \
extra/perl-try-tiny \
extra/perl-uri \
community/perl-file-tail
Mandatory Perl modules via cpanm:
cpanm Mail::IMAPClient \
Unicode::String \
Sys::MemInfo \
Encode::IMAPUTF7
Other Perl modules, needed sometimes:
cpanm Authen::NTLM \
JSON::WebToken \
JSON::WebToken::Crypt::RSA \
Test::Mock::Guard \
Test::Warn \
PAR::Packer \
Proc::ProcessTable

View File

@ -0,0 +1,37 @@
#!/bin/cat
# $Id: INSTALL.CPanel.txt,v 1.7 2019/11/28 14:38:18 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.CPanel.txt
=======================================================================
Installing imapsync on CPanel
=======================================================================
This procedure has been tested on cPanel 11.36 (64 bits).
It should be the same for any Cpanel distribution based on Centos.
First install access to the Epel repository via yum:
yum install epel-release
Then install imapsync and its dependencies:
yum install imapsync
After installing imapsync, it should be able to work on your system.
A good test that shows also the basic example:
imapsync
If you want to update to the latest imapsync, then look at
the Centos installation document called INSTALL.Centos.txt
and also INSTALL.ANY.txt
You're on your own, I don't have access to a CPanel system.
=======================================================================
=======================================================================

View File

@ -0,0 +1,233 @@
#!/bin/cat
# $Id: INSTALL.Centos.txt,v 1.32 2022/05/17 14:32:47 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Centos.txt
=======================================================================
Installing imapsync on CentOS
=======================================================================
There is one section for AlmaLinux and Centos 8,
one for Centos 7,
and one for Centos 6.
=======================================================================
AlmaLinux and Centos 8
=======================================================================
How to install imapsync on AlmaLinux and Centos 8:
dnf install --enablerepo=powertools imapsync
imapsync --testslive
For Centos 8 End Of Life fix:
https://www.centos.org/centos-linux-eol/
If you encounter the error
"Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist"
Do this:
sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux-*
sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux-*
dnf update
To upgrade to CentOS stream:
dnf install centos-release-stream
dnf swap centos-{linux,stream}-repos
dnf distro-sync
Taken from
https://stackoverflow.com/questions/70926799/centos-through-vm-no-urls-in-mirrorlist
and tested successfully on one of my hosts, an ex-Centos8/CentOS-stream now.
=======================================================================
AlmaLinux and Centos 8 and latest imapsync
=======================================================================
At the time of this writing (December 2021), the powertools repository
contains imapsync release 1.977, which is not the latest available
imapsync release. (1.977 date is 2019/12/23 while 2.174 is 2021/12/14)
In order to install the latest imapsync I suggest the following process:
Install the powertools imapsync 1.977 release via dnf like mentioned above.
It will install the needed packages, Perl itself and Perl modules
for release 1.977.
The imapsync 2.206 and next ones needs the Perl module:
dnf install perl-Proc-ProcessTable
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
./imapsync --testslive
./imapsync --tests
You may get an even more recent imapsync release this way.
=======================================================================
Centos 7
=======================================================================
First install access to the Epel repository via yum:
yum install epel-release
Then install imapsync and its dependencies:
yum install imapsync
After installing imapsync, it should be able to work on your system.
A good test that shows also the basic example:
imapsync
A live test:
imapsync --testslive --tls1 --ssl2
Unit tests:
imapsync --tests
=======================================================================
Centos 7 and latest imapsync
=======================================================================
At the time of this writing (March 2020), the epel7 repository still
contains imapsync release 1.727, which is not the latest available
imapsync release. (1.727 date is 2016/08/19 while 2.174 is 2021/12/14)
In order to install the latest imapsync I suggest the following process:
Install the epel imapsync 1.727 release via yum like mentioned above.
It will install many needed packages, Perl itself and Perl modules
for release 1.727. The latest imapsync needs more Perl modules.
Then:
yum install perl-App-cpanminus \
perl-Dist-CheckConflicts \
perl-HTML-Parser \
perl-libwww-perl \
perl-Module-Implementation \
perl-Module-ScanDeps \
perl-Package-Stash \
perl-Package-Stash-XS \
perl-PAR-Packer \
perl-Regexp-Common \
perl-Sys-MemInfo \
perl-Test-Fatal \
perl-Test-Mock-Guard \
perl-Test-Requires \
perl-Test-Deep \
perl-File-Tail \
perl-Unicode-String \
perl-Test-NoWarnings \
perl-Test-Simple \
perl-Test-Warn \
perl-Sub-Uplevel \
perl-Proc-ProcessTable \
ca-certificates
Also, run the following commands:
cpanm Encode::IMAPUTF7
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
./imapsync --testslive
Now you should have the latest imapsync locally working on Centos 7.
If you want to install it and replace the old one:
mv /usr/bin/imapsync /usr/bin/imapsync_old
cp ./imapsync /usr/bin/imapsync
imapsync --version
should give the latest release number instead of 1.727
(It is 1.979 at the time of this writing)
If you want to install an online imapsync service
like /X then you also need a recent decent CGI.pm,
add it with cpanm:
cpanm CGI
=======================================================================
Centos 6
=======================================================================
This section has been tested with imapsync release 1.670
First, install access to the Epel repository
wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
Then install imapsync and its dependencies:
yum install imapsync
After installing imapsync, it should be able to work on your system.
A good test that shows also the basic example:
imapsync
A live test:
imapsync --testslive
Unit tests:
imapsync --tests
=======================================================================
Centos 6 and latest imapsync
=======================================================================
Install the epel imapsync 1.670 release via yum like mentioned above.
It will install many needed packages, Perl itself and Perl modules
for release 1.670. The latest imapsync needs more Perl modules.
Then:
yum install perl-App-cpanminus \
perl-Dist-CheckConflicts \
perl-HTML-Parser \
perl-libwww-perl \
perl-Module-Implementation \
perl-Module-ScanDeps \
perl-Package-Stash \
perl-Package-Stash-XS \
perl-PAR-Packer \
perl-Regexp-Common \
perl-Sys-MemInfo \
perl-Test-Fatal \
perl-Test-Mock-Guard \
perl-Test-Requires \
perl-Test-Deep \
perl-File-Tail \
perl-Unicode-String \
perl-Proc-ProcessTable
wget https://imapsync.lamiral.info/imapsync
chmod +x imapsync
./imapsync --testslive
Now you have the latest imapsync on Centos 6,
do you?
=======================================================================
=======================================================================

View File

@ -0,0 +1,251 @@
#!/bin/cat
# $Id: INSTALL.Darwin.txt,v 1.39 2022/05/17 14:31:44 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Darwin.txt
Usual users should follow the A) section only, with the exception
of reading B) before if you use a Catalina system.
A) Installing imapsync binary on Darwin / Mac OS X
B) Installing imapsync on Catalina
C) Installing imapsync script on Darwin / Mac OS X with brew
D) Installing imapsync script on Darwin / Mac OS X the way I do
E) Building imapsync binary on Darwin / Mac OS X
=======================================================================
A) Installing imapsync binary on Darwin / Mac OS X
=======================================================================
There is a standalone imapsync binary for Mac OS X
called "imapsync_bin_Darwin" (without the quotes),
available in the compressed tarball called imapsync-2.200.tgz
where 2.200 is the imapsync version number.
In case you haven't bought it yet or forgot where the download link is,
check https://imapsync.lamiral.info/dist/
Download the tarball imapsync-2.200.tgz, not the binary imapsync_bin_Darwin
because the binary is also in the tarball and there are other useful files
in the tarball. The binary present in /dist/ is just there for a fast upgrade
when you need one.
Suppose this tarball imapsync-2.200.tgz is downloaded under your $HOME directory,
lets say /Users/gilles/, but you can put it anywhere. (But obviously your $HOME
directory is not /Users/gilles/, it maybe /Users/john/ or /Users/zoey/ where John
or Zoey is your login name.)
Open a terminal: /Applications/Utilities/Terminal, double-click on Terminal.
Untar the tarball:
cd
pwd
tar xzvf imapsync-2.200.tgz
In case the previous command fails, it means the tarball file called imapsync-2.200.tgz
is not in your $HOME directory, you may have downloaded it elsewhere on the file system.
A way to find it is the command:
find / | grep imapsync
Now that the tarball is extracted, it created a directory called imapsync-2.200/
Go into the directory imapsync-2.200 with the command:
cd imapsync-2.200
First let's have a simple run to see if imapsync_bin_Darwin works.
You should see some help about options and an example at the end
of this run:
./imapsync_bin_Darwin
To go further, perform a complete test with two
real IMAP server accounts:
./imapsync_bin_Darwin --testslive
If this sync works fine then imapsync_bin_Darwin is ready for any
imap account synchronization.
When reading the documentation with imapsync command lines examples,
you have to replace the command "imapsync" by "imapsync_bin_Darwin"
For example, instead of the command:
./imapsync \
--host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
you have to use:
./imapsync_bin_Darwin \
--host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
The script examples/imapsync_example_darwin.sh is ready to use,
it is a copy of examples/imapsync_example.sh adapted to Mac users.
Be careful the way you edit the script files, use a text editor.
Do not use a word processor because word processors add
or use special formating characters that will break the shell
scripts. If you use TextEdit, use the text mode.
In the terminal, try:
sh examples/imapsync_example_darwin.sh
or copy it and run your copy instead:
cp examples/imapsync_example_darwin.sh mysync.sh
sh mysync.sh
Now read on the tutorial
https://imapsync.lamiral.info/doc/TUTORIAL_Unix.html
in order to complete your formation on imapsync.
If you need to sync or migrate many accounts,
the script examples/sync_loop_darwin.sh is also ready to use.
=======================================================================
B) Installing imapsync on Catalina
=======================================================================
Caveat Catalina: The binary imapsync_bin_Darwin is detected as malware
or similar on the latest Mac OS X named Catalina.
It's Catalina new security policy. Imapsync is not a malware or similar.
All other Mac OS X releases are ok.
In order to pass away this detection, follow these steps:
Run imapsync_bin_Darwin
After this first launch, the security asks to authorize the unknown
developer program. Set up the root of the imapsync_bin_Darwin file
and then run it again or any of the .sh files, it should run ok
without any notice any more.
Sources of this tip:
https://www.quora.com/What-does-Can-t-be-opened-because-it-s-integrity-cannot-be-verified-mean-on-OSX-Catalina
https://github.com/fastlane/fastlane/issues/15186#issuecomment-532047545
https://github.com/neovim/neovim/issues/11011#issuecomment-531369505
Other way, on the binary: Control + Right Click -> Open
See also the last section of https://support.apple.com/en-us/HT202491
"How to open an app that hasn't been notarized or is from an unidentified developer"
I don't own a Catalina computer so I can't experiment on this nor make
imapsync directly ok with Catalina. You're richer than me :-)
=======================================================================
C) Installing imapsync script on Darwin / Mac OS X with brew
=======================================================================
In case you have brew installed on your system ( see https://brew.sh/ ),
install imapsync with the command:
brew install imapsync
Do a real test with imap server test.lamiral.info:
imapsync --testslive
If you encouter an error like "String.c: loadable library and perl binaries
are mismatched (got handshake key 0xc500080, needed 0xc400080)",
then install imapsync from source with the command:
brew install -s imapsync
Thanks to Alessandro for this tip!
=======================================================================
D) Installing imapsync script on Darwin / Mac OS X the way I do
=======================================================================
This part is only for advanced Unix users, or brave users.
The "make" command is a prerequisite to build some Perl modules.
Install the "make" command in case it is not already installed.
First let us install cpanminus locally in ~/perl5
curl -L https://cpanmin.us | perl - -l ~/perl5 App::cpanminus local::lib
Then take this install into account in the current environment
perl -I ~/perl5/lib/perl5 -Mlocal::lib # just to see the variables
eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib` # doing the variables assignments
If you want to have always this setting in your environment then run the commands
echo 'eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`' >> ~/.profile
echo 'export MANPATH=$HOME/perl5/man:$MANPATH' >> ~/.profile
cat ~/.profile
. ~/.profile
Now let's update the standard CPAN Perl module
cpanm CPAN
The specific install part for imapsync begins, the script "prerequisites_imapsync"
helps to verify what is needed to install on your system
curl -L http://imapsync.lamiral.info/INSTALL.d/prerequisites_imapsync > prerequisites_imapsync
sh prerequisites_imapsync
cpanm Authen::NTLM
cpanm IO::Tee
cpanm Mail::IMAPClient # File::Copy::Recursive?
cpanm Unicode::String
cpanm Sys::MemInfo
cpanm File::Tail
cpanm Proc::ProcessTable
cpanm Test::MockObject
cpanm Readonly
cpanm Data::Uniqid
cpanm JSON::WebToken
We're ready to install and test the latest imapsync
curl -L http://imapsync.lamiral.info/imapsync > imapsync
chmod +x imapsync
./imapsync
./imapsync --testslive
./imapsync --tests
You can rerun "sh prerequisites_imapsync"
to verify everything is ok:
sh prerequisites_imapsync
When everything is ok the script execution ends with this sentence
"All needed modules are already installed"
=======================================================================
E) Building imapsync binary on Darwin / Mac OS X
=======================================================================
cpanm Module::ScanDeps
cpanm PAR::Packer
pp -x -u -o imapsync.bin imapsync
./imapsync.bin
./imapsync.bin --testslive
./imapsync.bin --tests
./imapsync.bin --module
=======================================================================
=======================================================================

View File

@ -0,0 +1,297 @@
#!/bin/cat
# $Id: INSTALL.Debian.txt,v 1.38 2022/05/17 14:29:36 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Debian.txt
There are four install sections in this document,
1) one for Debian 9 Stretch, Debian 10 Buster, Debian 11 Bullseye, Debian 12 Bookworm
2) one for all the Debian, called "After installing the dependencies".
3) one for Debian 8 Jessie (obsolete)
4) one for Debian 7 Wheezy (even more obsolete)
After reading and following 1) or 3) or 4), you'll be invited to read and follow
the section "2) After installing the dependencies"
What is your Debian release?
Type on a terminal:
cat /etc/debian_version
12.x => Bookworm
11.x => Bullseye
10.x => Buster
9.x => Stretch
8.x => Jessie
7.x => Wheezy
where x is any number between 0 and 9, x is minor and to be forgotten.
First, a call to Debian packagers
Why imapsync is not in Debian as a package?
Imapsync used to be in Debian from 2005-04-25 (release 1.125) to 2011-01-25 (release 1.315).
It could be in Debian nowadays, as always, my last words on this discussion were
"Do what you want, I promise I won't complain anymore about the fact imapsync is in Debian or not."
https://lists.debian.org/debian-legal/2011/01/msg00058.html
The Imapsync license is "No limits to do anything with this work and this license".
So feel free!
See also a more detailed story at
https://lists.debian.org/debian-user/2016/11/msg00849.html
=======================================================================
1) Installing imapsync on Debian 9 Stretch, 10 Buster, 11 Bullseye
=======================================================================
How to install imapsync dependencies on a Debian system?
The repositories are the classical ones, ie, a
/etc/apt/sources.list file with:
# Debian 12 Bookworm
deb http://deb.debian.org/debian bookworm main contrib
or
# Debian 11 Bullseye
deb http://deb.debian.org/debian bullseye main contrib
or
# Debian 10 Buster
deb http://deb.debian.org/debian buster main contrib
or
# Debian 9 Stretch
deb http://deb.debian.org/debian stretch main contrib
You need root privilege to run it, be root or use sudo.
apt install -y \
libauthen-ntlm-perl \
libcgi-pm-perl \
libcrypt-openssl-rsa-perl \
libdata-uniqid-perl \
libencode-imaputf7-perl \
libfile-copy-recursive-perl \
libfile-tail-perl \
libio-socket-inet6-perl \
libio-socket-ssl-perl \
libio-tee-perl \
libhtml-parser-perl \
libjson-webtoken-perl \
libmail-imapclient-perl \
libparse-recdescent-perl \
libproc-processtable-perl \
libmodule-scandeps-perl \
libreadonly-perl \
libregexp-common-perl \
libsys-meminfo-perl \
libterm-readkey-perl \
libtest-mockobject-perl \
libtest-pod-perl \
libunicode-string-perl \
liburi-perl \
libwww-perl \
libtest-nowarnings-perl \
libtest-deep-perl \
libtest-warn-perl \
make \
time \
cpanminus
After installing the dependencies, imapsync should be working,
go to the section "2) After installing the dependencies" below.
=======================================================================
2) After installing the dependencies
=======================================================================
You don't have to be root to test and use imapsync.
Take imapsync either on github:
wget -N https://raw.githubusercontent.com/imapsync/imapsync/master/imapsync
or be on the cutting edge with the upstream site:
wget -N https://imapsync.lamiral.info/imapsync
Add execution permission to the downloaded script:
chmod +x imapsync
Check the dependencies and print also the basic example:
./imapsync
Perform a live test showing imapsync job:
./imapsync --testslive
Now install imapsync on the system (need root privileges again):
cp imapsync /usr/bin/
That's finished for the installation part.
You can now use imapsync without knowing where it is located
on the system:
imapsync
Now go read the menu http://imapsync.lamiral.info/#doc
and start with the tutorial
https://imapsync.lamiral.info/doc/TUTORIAL_Unix.html
Feedback is welcome!
=======================================================================
3) Installing imapsync on Debian 8 Jessie
=======================================================================
How to install imapsync dependencies on a Debian system?
You need root privilege.
The first command installs standard Debian packages:
apt install \
libauthen-ntlm-perl \
libcgi-pm-perl \
libcrypt-openssl-rsa-perl \
libdata-uniqid-perl \
libencode-imaputf7-perl \
libfile-copy-recursive-perl \
libfile-tail-perl \
libio-socket-inet6-perl \
libio-socket-ssl-perl \
libio-tee-perl \
libhtml-parser-perl \
libmail-imapclient-perl \
libparse-recdescent-perl \
libmodule-scandeps-perl \
libreadonly-perl \
libterm-readkey-perl \
libtest-mockobject-perl \
libtest-pod-perl \
libunicode-string-perl \
liburi-perl \
libwww-perl \
libregexp-common-perl \
libtest-nowarnings-perl \
libtest-deep-perl \
libtest-warn-perl \
gcc \
make \
cpanminus
The package libdata-uniqid-perl is in the jessie-backports repository
https://packages.debian.org/fr/jessie-backports/libdata-uniqid-perl
Here is how to add jessie-backports repository in /etc/apt/sources.list
https://backports.debian.org/Instructions/
If you can't add the jessie-backports repository in /etc/apt/sources.list you
have to install Perl module Data::Uniqid with cpanm:
cpanm Data::Uniqid
The following second command installs "manually" the Perl module Sys::MemInfo
because Debian 8 Jessie doesn't have this Perl module available via a package
(but Debian 9 Stretch does):
cpanm Sys::MemInfo
The following optional cpanm command updates Perl module Mail::IMAPClient
because it is good to be up to date with that imapsync dependency,
but it is not mandatory since Mail::IMAPClient is installed by
the Debian package libmail-imapclient-perl:
cpanm Mail::IMAPClient
Last, in case you need to use XOAUTH2 authentication you have to install
the module JSON::WebToken with the command:
cpanm JSON::WebToken
After installing the dependencies, imapsync should be working,
go to the section "2) After installing the dependencies" above.
=======================================================================
4) Installing imapsync on Debian 7 Wheezy
=======================================================================
How to install imapsync dependencies on a Debian system?
You need root privilege to run the commands.
The first command installs standard Debian packages:
apt-get install \
libauthen-ntlm-perl \
libclass-load-perl \
libcrypt-openssl-rsa-perl \
libdigest-hmac-perl \
libencode-imaputf7-perl \
libfile-copy-recursive-perl \
libfile-tail-perl \
libio-compress-perl \
libio-socket-inet6-perl \
libio-socket-ssl-perl \
libio-tee-perl \
libmail-imapclient-perl \
libmodule-implementation-perl \
libmodule-runtime-perl \
libmodule-scandeps-perl \
libpackage-stash-perl \
libpackage-stash-xs-perl \
libpar-packer-perl \
libreadonly-perl \
libterm-readkey-perl \
libtest-fatal-perl \
libtest-pod-perl \
libtest-requires-perl \
libtest-simple-perl \
libunicode-string-perl \
liburi-perl \
gcc \
make \
cpanminus
The following second command installs "manually" the Perl modules
Data::Uniqid
Sys::MemInfo
because Debian hasn't made them available via a package yet.
This cpanm command also installs manually Perl module Mail::IMAPClient
because it is good to be up to date with that imapsync dependency,
but it is not mandatory since Mail::IMAPClient is installed by
the Debian package libmail-imapclient-perl:
cpanm Data::Uniqid Sys::MemInfo
In case you want to update the Perl module Mail::IMAPClient,
a major module for imapsync, but an old March 2012 release 3.31 in Wheezy,
the following command updates it "manually":
cpanm Mail::IMAPClient
Last, in case you need to use XOAUTH2 authentication you have to install
the module JSON::WebToken with the command:
cpanm JSON::WebToken
After installing the dependencies, imapsync should be working,
then go to the section "2) After installing the dependencies" above.
=======================================================================

View File

@ -0,0 +1,128 @@
#!/bin/cat
# $Id: INSTALL.Docker_build.txt,v 1.17 2022/05/17 14:29:36 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Docker_build.txt
=======================================================================
Building an imapsync docker image from Debian
=======================================================================
Nearly verbatim copy from https://imapsync.lamiral.info/INSTALL.d/Dockerfile
(verbatim but the Dockerfile rcs tag $ removed because it would conflict)
## Dockerfile for building a docker imapsync image
# Id: Dockerfile,v 1.42 2022/04/04 19:16:50 gilles Exp gilles $
# I use the following command to build the image:
#
# docker build -t gilleslamiral/imapsync .
#
# where this Dockerfile is in the current directory
#
# This Dockerfile builds an image with two imapsync
# 1) One comes from https://imapsync.lamiral.info/imapsync
# and goes to /usr/bin/imapsync in the Docker image
# It is used with the command:
#
# docker run gilleslamiral/imapsync imapsync ...
#
# 2) One comes from the local file ./imapsync
# and goes to /imapsync in the Docker image
# It is used with the command:
#
# docker run gilleslamiral/imapsync /imapsync ...
# I thank you very much
# I like thanks
# I like stars
# I like money to keep on doing this stuff
# Number of imapsync images pulled so far (2022_04_04): 673551
# Command used:
# curl -s https://hub.docker.com/v2/repositories/gilleslamiral/imapsync/ | jq '.pull_count'
# Debian Bullseye is Debian 11
FROM debian:bullseye
LABEL maintainer="Gilles LAMIRAL <gilles@lamiral.info>" \
description="Imapsync" \
documentation="https://imapsync.lamiral.info/#doc"
# I put a copy of the Dockerfile in the image itself
# It can help maintenance, can't it?
# Also put optionally my local and usually more recent imapsync on /, for testing purpose
COPY Dockerfile imapsyn[c] prerequisites_imapsyn[c] /
RUN set -xe && \
apt-get update \
&& apt-get install -y \
libauthen-ntlm-perl \
libcgi-pm-perl \
libcrypt-openssl-rsa-perl \
libdata-uniqid-perl \
libencode-imaputf7-perl \
libfile-copy-recursive-perl \
libfile-tail-perl \
libio-compress-perl \
libio-socket-ssl-perl \
libio-socket-inet6-perl \
libio-tee-perl \
libhtml-parser-perl \
libjson-webtoken-perl \
libmail-imapclient-perl \
libparse-recdescent-perl \
libmodule-scandeps-perl \
libpar-packer-perl \
libproc-processtable-perl \
libreadonly-perl \
libregexp-common-perl \
libsys-meminfo-perl \
libterm-readkey-perl \
libtest-mockobject-perl \
libtest-pod-perl \
libunicode-string-perl \
liburi-perl \
libwww-perl \
procps \
wget \
make \
cpanminus \
lsof \
ncat \
openssl \
ca-certificates \
&& rm -rf /var/lib/apt/lists/* \
&& cpanm IO::Socket::SSL
# I added the IO::Socket::SSL update to avoid the annoying, confusing and useless warning
# DEBUG: .../IO/Socket/SSL.pm:1177: global error: Undefined SSL object
RUN set -xe \
&& cd /usr/bin/ \
&& pwd \
&& wget -N --no-check-certificate https://imapsync.lamiral.info/imapsync \
https://imapsync.lamiral.info/prerequisites_imapsync \
https://raw.githubusercontent.com/google/gmail-oauth2-tools/master/python/oauth2.py \
&& chmod +x imapsync oauth2.py \
&& /usr/bin/imapsync --testslive && /usr/bin/imapsync --tests # just_a_comment_to_force_update 2022_04_04_21_16_50
USER nobody:nogroup
ENV HOME /var/tmp/
WORKDIR /var/tmp/
STOPSIGNAL SIGINT
CMD ["/usr/bin/imapsync"]
#
# End of imapsync Dockerfile
=======================================================================
=======================================================================

View File

@ -0,0 +1,39 @@
#!/bin/cat
# $Id: INSTALL.FreeBSD.txt,v 1.12 2022/05/17 14:31:44 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.FreeBSD.txt
=======================================================================
Installing imapsync on FreeBSD
=======================================================================
Thanks to Kurt Jaeger pi@FreeBSD.org, an imapsync package is available in FreeBSD
http://portsmon.freebsd.org/portoverview.py?category=mail&portname=imapsync
So, to install imapsync you just have to type one command in a terminal:
sudo pkg install imapsync
or if you're already root on the console:
pkg install imapsync
Tested 7 March 2015 on FreeBSD 10.1-RELEASE
Tested 2 July 2018 on FreeBSD 11.2-RELEASE
=======================================================================
=======================================================================
In order to install and be able to run latest imapsync online /X,
install also:
p5-Regexp-Common
p5-Test-MockObject
p5-CGI
p5-Encode-IMAPUTF7
Call to FreeBSD developer: These are also the new Perl module dependencies to add
for imapsync 2.206 and next ones:
Proc::ProcessTable

View File

@ -0,0 +1,602 @@
#!/bin/cat
# $Id: INSTALL.OnlineUI.txt,v 1.56 2022/05/19 08:55:50 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.OnlineUI.txt
=======================================================================
Installing imapsync online
=======================================================================
I'm now confident with /X since the /X service is up and running quite
well since January 2017. Anyway, if you run this service on your own,
online, you take responsibility for it.
=======================================================================
Hardware consideration
RAM used per imapsync process, mean value: 230 MB.
Average_bandwidth_rate: 345 KiB/s ~ 2.8 Mbps.
Load mean: 0.8 on a CPU 4 cores "Intel(R) i5-2320 3.00GHz K8-class"
=======================================================================
Installation
You have to be a little familiar with what a CGI script is and how to
activate a CGI script on the Apache HTTP server, or any other HTTP
server. I have received demands to run it on the Ngnix HTTP server but
I haven't played with it yet. Linux is also a preferred platform (I
run /X service on Linux and FreeBSD).
I have tested this visual interface on Mac. It works. For now, it
demands some skills few Mac users have. Drop me a note in case you
want to do that.
I have tested this visual interface on Windows, it fails on Windows
because of some hardcoded Unix paths. I'm working on it to be Windows
ok but it's not done yet (May 2020).
Some users have successfully installed a /X visual interface on
Windows using a Linux VM machine.
The web visual user interface frontend /X is compounded in four files:
a html5 file, a CSS file, a javascript file, and a logo image:
* https://imapsync.lamiral.info/X/imapsync_form_extra.html
* https://imapsync.lamiral.info/X/imapsync_form.css
* https://imapsync.lamiral.info/X/imapsync_form.js
* https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
You can do a "view source" to see the HTML file as it is written, and
a "save" to get it locally. The three other files can be saved the
same way or with a command named "wget". I strongly suggest using
wget, see below the ready-to-use command lines.
Those four files can be put anywhere on a web server, as long as they
stand in the same directory. If you want to put them in different
directories, just change the content of imapsync_form_extra.html to
reflect the change, ie, change the two lines referencing
imapsync_form.css and imapsync_form.js href="imapsync_form.css" (near
the beginning of imapsync_form_extra.html) src="imapsync_form.js"
(near the end of imapsync_form_extra.html) I let you change the image
logo as an exercise, it's safe if you fail.
The actual imap syncing work is done by imapsync acting as a CGI, the
visual interface is only there to give imapsync the parameters needed
for the sync.
Use at least Perl module CGI.pm release 4.08 (2014-10-18) to avoid the
bug "Undefined subroutine CGI::multi_param". You can use the command
named cpanm to upgrade CGI.pm to its last version, it's the easiest
way.
Print the CGI.pm release with:
perl -MCGI -e 'print "$CGI::VERSION\n"'
If it is under release 4.08 (2014-10-18) then upgrade it with
cpanm CGI
It is a good thing to remove the old one if it was installed by a
distribution package, I let you this part as an exercise too. Ok, here
is a way:
apt remove libcgi-pm-perl # for the Debian family
dnf remove perl-CGI # for the Centos family
To check and fix the Perl modules dependencies, run:
cd
wget -N https://imapsync.lamiral.info/prerequisites_imapsync
sh prerequisites_imapsync
To make imapsync work as a CGI script, there are three conditions.
First, imapsync has to work by itself on the web host. If imapsync
doesn't work by itself, as a command line, then it won't work as a CGI
script.
Second, imapsync has to work by itself on the web host using the Unix
user running the webserver.
Third, the file imapsync has to be considered as a cgi script.
Command lines to provide and verify those three conditions will be
provided further in this document, for the Debian family systems and
for the Centos family systems. You are strongly advised to follow this
commands if need and want help from me because I will first ask you to
run them before searching what you did wrong.
The imapsync_form_extra.html file in action calls the CGI location
/cgi-bin/imapsync
which has to be imapsync itself, the file script (not the directory).
The very latest and relatively stable imapsync is
https://imapsync.lamiral.info/imapsync
This file is the program file used verbatim for the service given at
https://imapsync.lamiral.info/X/
Copy the three files imapsync_form.* on a directory that is exported
by your HTTP server.
Copy the imapsync script on the cgi-bin/ directory
allowing CGIs and you'll have your imapsync visual interface
and service. The cgi-bin/ directory is usually outside the
hierarchy exported to anybody by the HTTP server.
The default Apache 2.4 timeout is 60 seconds, one minute, and 300
secondes for older Apache, 5 minutes. See
https://httpd.apache.org/docs/2.4/mod/core.html#timeout
I use "Timeout 3600", 3600 seconds, an hour. I chose this huge timeout
value because imapsync can spend a long time without talking while
getting the headers of huge folders of 100k messages. If you intend
to offer this service for huge mailboxes or for a long time, I
strongly recommend you to set this "Timeout 3600" in the Apache
configuration right now because you will sure end up with this timeout
issue in a few months. You can search for timeouts in the Apache error
log to see if you have timeout issues.
Now that I have explained the general context for any system, I'll
describe concrete examples on several systems, Debian/Ubuntu and
Centos. Feedbacks show that the Centos process is easier in case you
don't know very much any Linux distribution. But I have to add that if
you don't know very much the Linux distribution you use, then you
shouldn't install this imapsync service at all.
=============================================================================
A) Concrete example on a Debian server with Apache:
First, install Apache on your Debian system:
apt install apache2
Imapsync script place on the server disk will be
/usr/lib/cgi-bin/imapsync
This classical /cgi-bin directory is usually already configured
in the Apache configuration file
/etc/apache2/sites-available/default-ssl
or
/etc/apache2/sites-available/default
This configuration file contains the following section
somewhere, maybe in comments for now, ie, with
some # characters at the beginning to make them ignored.
If you don't find the following section, keep reading,
the solution is below.
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
# Next line "no-gzip 1" is to avoid output buffering,
# clients can then see the log during the sync
SetEnv no-gzip 1
Options +ExecCGI -MultiViews
# Choose either one or the other, depending on your Apache version
# Lines beginning with # are ignored
# For Apache 2.2
#Order allow, deny
#Allow from all
# Apache 2.4
Require all granted
</Directory>
In recent Debian distributions you can activate this cgi
stuff with the following commands:
a2enmod cgi
a2enconf serve-cgi-bin
/etc/init.d/apache2 restart
If the cgi mode and the cgi-bin configuration are not activated then
you may encounter a 404 error when, later, you will run the command
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
That's all for the Apache Debian family configuration side.
Now get, test, and install the latest imapsync:
cd
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
# some basic tests
./imapsync
./imapsync --testslive
cp imapsync /usr/lib/cgi-bin/
Assuming that the Unix account running Apache is www-data, check that
it will work under Apache with this command:
su -s /bin/sh -c 'SERVER_SOFTWARE=foo /usr/lib/cgi-bin/imapsync' www-data
You should end with something like:
Exiting with return value 0 (EX_OK: successful termination)
Test that imapsync is considered a cgi with:
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
The last command should print something like:
Status: 200 OK to sync IMAP boxes. Load on bar is ...
...
Exiting with return value 0 (EX_OK: successful termination)
If you get a 404 or a 5xx here then review the cgi installation and
configuration part.
You can also verify that the webserver is not buffering its output with the
command:
wget -nv -S -O- 'http://localhost/cgi-bin/imapsync?testslive=1&simulong=10'
You should get the output as time goes on. If you don't get the output
as time goes on, ie you see no output then all output at once, it
means the webserver is buffering. Fix it with the "SetEnv no-gzip 1"
described above.
The UI front-end file place on the server disk in this example is
/var/www/html/X/imapsync_form_extra.html
but it can be placed anywhere on the disk, the important thing is that
it has to be served by the webserver.
mkdir /var/www/html/X/
cd /var/www/html/X/
wget -N \
https://imapsync.lamiral.info/X/imapsync_form_extra.html \
https://imapsync.lamiral.info/X/imapsync_form.css \
https://imapsync.lamiral.info/X/imapsync_form.js \
https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
ln -s imapsync_form_extra.html index.html
The imapsync process working directory in cgi mode is
/var/tmp/imapsync_cgi/
it is not configurable unless changing it in imapsync directly, it is
hard-coded in imapsync. In this directory will go the log files and
the pid files.
Check
http://yourhost/X/imapsync_form_extra.html
or the safer
https://yourhost/X/imapsync_form_extra.html
Let's encrypt your site because credentials should never travel in
clear form. Go to https://certbot.eff.org/instructions
See the Troubleshooting section to fix the systemd Apache
PrivateTmp=true issue.
That's all for installing a /X service on a Debian family system.
=============================================================================
B) Here is a concrete example on a Centos 7 server with the Apache
webserver httpd:
First, follow and apply the section "Centos 7 and latest imapsync" at
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Centos.txt
Then:
yum install httpd
systemctl restart httpd
cpanm CGI
mkdir /var/www/html/X/
cd /var/www/html/X/
wget -N \
https://imapsync.lamiral.info/X/imapsync_form_extra.html \
https://imapsync.lamiral.info/X/imapsync_form.css \
https://imapsync.lamiral.info/X/imapsync_form.js \
https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
ln -s imapsync_form_extra.html index.html
cd
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
# some basic tests
./imapsync
./imapsync --testslive
cp imapsync /var/www/cgi-bin/
Assuming that the Unix account running Apache is "apache",
which is the default Apache user on Centos system,
check that it will work under Apache with this command:
# a real synchronization but not in cgi context
cd /tmp
su -s /bin/sh -c '/var/www/cgi-bin/imapsync --testslive' apache
# in cgi context but just the imapsync command with no parameter
cd
su -s /bin/sh -c 'SERVER_SOFTWARE=foo /var/www/cgi-bin/imapsync' apache
# a real synchronization in cgi context
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
The last command should print something like:
Status: 200 OK to sync IMAP boxes. Load on bar is ...
...
You can also verify that the webserver is not buffering its output
with the command:
wget -nv -S -O- 'http://localhost/cgi-bin/imapsync?testslive=1&simulong=10'
You should get the output as time goes on. If you don't, no output
then all output at once, it means the webserver is buffering. Fix it
with the "SetEnv no-gzip 1" described above.
Now check
http://yourhost/X/imapsync_form_extra.html
or the safer
https://yourhost/X/imapsync_form_extra.html
That's all for installing a /X service on Centos 7.
B bis) How about Centos 8?
Follow the procedure for Centos 7. While imapsync is ok on the command
line, you will encounter some permission denied in the CGI
context. Something like:
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
...
Host1 failure: can not open imap connection on host1
[test1.lamiral.info] with user [test1]: Unable to connect to
test1.lamiral.info: Permission denied
The issue might come from SELinux. I haven't dig into SELinux enough
to give you the commands that will allow imapsync online and only it
while maintaining SELinux in enforcing mode.
Quick solution:
getsebool httpd_can_network_connect # should show --> off
setsebool -P httpd_can_network_connect=1
getsebool httpd_can_network_connect # should show --> on
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1 # no more "Permission denied"
The -P option installs the rule permanently, even after a reboot
To go back to the previous state:
getsebool httpd_can_network_connect # should show --> on
setsebool -P httpd_can_network_connect=0
getsebool httpd_can_network_connect # should show --> off
Nota bene
=========
You may also want to avoid being placed by systemd in a directory like
(where xxx are crypto hash characters):
/var/tmp/systemd-private-xxx-httpd.service-xxx/tmp/
In that case, see the Troubleshooting section below.
=======================================================================
=================== Bandwidth statistics ==========================
=======================================================================
If you want the bandwidth statistics like the ones at the bottom of
the page and following the image link, more detailed at
https://imapsync.lamiral.info/vnstat/vnstati.html
Those stats are generated by vnstat
https://humdi.net/vnstat/
Vnstat is already available as a package in most Linux distros.
The images are generated by the following commands, every minute:
vnstati -s -o /var/www/html/vnstat/vnstat_s.png
vnstati -h -o /var/www/html/vnstat/vnstat_h.png
vnstati -hg -o /var/www/html/vnstat/vnstat_hg.png
vnstati -hs -o /var/www/html/vnstat/vnstat_hs.png
vnstati -d -o /var/www/html/vnstat/vnstat_d.png
vnstati -m -o /var/www/html/vnstat/vnstat_m.png
vnstati -y -o /var/www/html/vnstat/vnstat_y.png
vnstati -t -o /var/www/html/vnstat/vnstat_t.png
vnstati -vs -o /var/www/html/vnstat/vnstat_vs.png
vnstati -5 -o /var/www/html/vnstat/vnstat_5.png
=======================================================================
====================== Troubleshooting ============================
=======================================================================
The log says the temporary directory is
/var/tmp/imapsync_cgi/
but this directory is not in the system. What a mystery!
It may be that the apache or httpd service is run by systemd with a
jailed temporary directory.
Solution:
find /etc/systemd/ /usr/lib/systemd/ | xargs grep -s PrivateTmp
If systemd jails Apache, then you'll find a line like:
/etc/systemd/system/multi-user.target.wants/apache2.service:PrivateTmp=true
(Debian/Ubuntu)
or
/usr/lib/systemd/system/httpd.service:PrivateTmp=true
(Centos)
The goal is to override the line
PrivateTmp=true
found in /etc/systemd/system/multi-user.target.wants/apache2.service
or
/usr/lib/systemd/system/httpd.service
with the line:
PrivateTmp=false
The right way to do it is by using the "systemctl edit ..." command
and then reload the systemd daemon and restart the apache2 service.
You can also edit directly the file override.conf if you know where
to do it. If you don't use the override.conf mechanism then your change will
be canceled the next time the apache package is updated.
Debian:
systemctl edit apache2
cat /etc/systemd/system/apache2.service.d/override.conf
[Service]
PrivateTmp=false
systemctl daemon-reload
systemctl restart apache2
systemctl status apache2
Centos:
systemctl edit httpd
cat /etc/systemd/system/httpd.service.d/override.conf
[Service]
PrivateTmp=false
systemctl daemon-reload
systemctl restart httpd
systemctl status httpd
Then retry
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
Look now if /var/tmp/imapsync_cgi/ is there.
=======================================================================
If you encounter this issue:
Failed to find a valid digest in the 'integrity' attribute for resource
'https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js'
with computed SHA-256 integrity 'kZMXypKF3if9/5v2tP9UHBvS/535tSyH7vjszruyCso='.
The resource has been blocked.
It may be because of AdBlock.
Verification:
wget https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js
cat jquery.min.js | openssl dgst -sha384 -binary | openssl base64 -A
gives exactly what is in imapsync_form_extra.html
more imapsync_form_extra.html
...
src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"
integrity="sha384-xBuQ/xzmlsLoJpyjoggmTEz8OWUFM0/RC5BsqQBDX2v5cMvDHcMakNTNrHIW2I5f"
So if your https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js
is not what it should be, your access looks compromised.
Thanks to Dominik Ulrich for this insight!
=======================================================================
=======================================================================
This part is for hackers only.
If you want to use the UI but make it more complicated things than
just run imapsync then use the following files:
imapsync_shell_wrapper instead of imapsync itself
imapsync_form_wrapper.js instead of imapsync_form.js
imapsync_form_wrapper.html instead of imapsync_form.html
How to get those files:
wget -N https://imapsync.lamiral.info/X/imapsync_shell_wrapper \
https://imapsync.lamiral.info/X/imapsync_form_wrapper.js \
https://imapsync.lamiral.info/X/imapsync_form_wrapper.html
Centos:
chmod +x imapsync_shell_wrapper
cp imapsync_shell_wrapper /var/www/cgi-bin/
Debian:
chmod +x imapsync_shell_wrapper
cp imapsync_shell_wrapper /usr/lib/cgi-bin/
Normally, you only have to change the script imapsync_shell_wrapper to
suit your needs.
Have in mind that the abort button will kill only one imapsync so it
is not a working button in case of successive imapsync runs.
=======================================================================
=======================================================================
====== mod_perl failure ======
This part is for mod_perl experts only.
The script imapsync doesn't work under Modperl::Registry nor under
ModPerl::PerlRun. So read on if you think you are better than me.
I tried the standard way, telling how any cgi Perl script can be run
under mod_perl perlrun, but it fails with imapsync. Any hint welcome!
# This is a Debian example
# install mod-perl with
apt-get install libapache2-mod-perl2
# edit the file /etc/apache2/mods-available/perl.conf
# with the following lines
more /etc/apache2/mods-available/perl.conf
<IfModule mod_perl.c>
PerlModule ModPerl::PerlRun
Alias /perl-run/ /usr/lib/cgi-bin/
<Location /perl-run>
SetHandler perl-script
PerlResponseHandler ModPerl::PerlRun
PerlOptions +ParseHeaders
Options +ExecCGI
</Location>
</IfModule>
# Enable the Apache perl module
a2enmod perl
# Verify perl.conf and perl.load are in directory mods-enabled/
ls mods-enabled/perl.*
# Reload Apache
apachectl graceful
# Verify imapsync works under perl-run
curl http://localhost/perl-run/imapsync
=======================================================================
=======================================================================

View File

@ -0,0 +1,97 @@
#!/bin/cat
# $Id: INSTALL.Ubuntu.txt,v 1.21 2022/06/03 16:58:47 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.Ubuntu.txt
=======================================================================
Installing imapsync on Ubuntu 16.04 or higher
=======================================================================
Here is the command to install imapsync dependencies,
you need root privilege to run them.
This command installs standard Ubuntu packages:
sudo apt-get install \
libauthen-ntlm-perl \
libclass-load-perl \
libcrypt-openssl-rsa-perl \
libcrypt-ssleay-perl \
libdata-uniqid-perl \
libdigest-hmac-perl \
libdist-checkconflicts-perl \
libencode-imaputf7-perl \
libfile-copy-recursive-perl \
libfile-tail-perl \
libio-compress-perl \
libio-socket-inet6-perl \
libio-socket-ssl-perl \
libio-tee-perl \
libjson-webtoken-perl \
libmail-imapclient-perl \
libmodule-scandeps-perl \
libnet-dbus-perl \
libnet-ssleay-perl \
libpar-packer-perl \
libproc-processtable-perl \
libreadonly-perl \
libregexp-common-perl \
libsys-meminfo-perl \
libterm-readkey-perl \
libtest-fatal-perl \
libtest-mock-guard-perl \
libtest-mockobject-perl \
libtest-pod-perl \
libtest-requires-perl \
libtest-simple-perl \
libunicode-string-perl \
liburi-perl \
libtest-nowarnings-perl \
libtest-deep-perl \
libtest-warn-perl \
make \
time \
cpanminus
In case you want to update the Perl module
Mail::IMAPClient, a major module for imapsync,
the following command installs it "manually":
sudo cpanm Mail::IMAPClient
After installing the dependencies, imapsync should be working.
You don't have to be root to test and use imapsync.
Take the compressed tarball called imapsync-1.xxx.tgz
where 1.xxx is the version number.
Untar the tarball where you want:
cd
tar xzvf imapsync-1.xxx.tgz
Go into the directory imapsync-1.xxx
cd imapsync-1.xxx
A dependencies test that shows also the basic example:
./imapsync
A live test showing imapsync job:
./imapsync --testslive
Now the install command (need root privileges again):
cp imapsync /usr/bin/
That's finished for the installation part.
You can use imapsync.
Now go to read http://imapsync.lamiral.info/#doc
start with the tutorial.
=======================================================================
=======================================================================

View File

@ -0,0 +1,18 @@
#!/bin/cat
# $Id: INSTALL.webserver.txt,v 1.6 2022/06/15 12:09:39 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.webserver.txt
The script named webserver in the top directory is a small web server
dedicated to offer the visual interface. It is experimental and it is
for now a plain text HTTP server (ie, not https).
Dependencies to run webserver on Debian/Ubuntu distributions:
sudo apt install libnet-server-perl

138
INSTALL.d/memo_docker Normal file
View File

@ -0,0 +1,138 @@
#!/bin/sh
# $Id: memo_docker,v 1.17 2022/04/04 14:46:21 gilles Exp gilles $
echo imapsync_docker_timestamp_dockerfile
imapsync_docker_timestamp_dockerfile()
{
DATE_CURRENT=`date +%Y_%m_%d_%H_%M_%S`
echo $DATE_CURRENT
sed -i -e "/just_a_comment_to_force_update/s/comment_to_force_update.*/comment_to_force_update $DATE_CURRENT/" Dockerfile
ci -l -f -m"Changing timestamp to $DATE_CURRENT with imapsync_docker_timestamp_dockerfile" Dockerfile
}
echo imapsync_docker_build
imapsync_docker_build()
{
docker build -t gilleslamiral/imapsync . \
&& imapsync_docker_add_tag_version && echo Build SUCCEEDED || echo Build FAILED
}
echo imapsync_docker_version
imapsync_docker_version()
{
docker run gilleslamiral/imapsync imapsync --version
}
echo imapsync_local_docker_version
imapsync_local_docker_version()
{
docker run gilleslamiral/imapsync /imapsync --version
}
echo imapsync_docker_add_tag_version
imapsync_docker_add_tag_version()
{
docker images gilleslamiral/imapsync
echo
version=`imapsync_docker_version`
echo $version
docker tag gilleslamiral/imapsync:latest gilleslamiral/imapsync:$version
docker images gilleslamiral/imapsync
}
echo imapsync_docker_testslive
imapsync_docker_testslive()
{
echo docker run gilleslamiral/imapsync imapsync --testslive "$@"
docker run gilleslamiral/imapsync imapsync --testslive "$@"
}
echo imapsync_docker_testslive6
imapsync_docker_testslive6()
{
echo docker run gilleslamiral/imapsync imapsync --testslive6 "$@"
docker run gilleslamiral/imapsync imapsync --testslive6 "$@"
}
echo imapsync_docker_tests
imapsync_docker_tests()
{
echo docker run gilleslamiral/imapsync imapsync --tests
docker run gilleslamiral/imapsync imapsync --tests
}
echo imapsync_docker_envfile
imapsync_docker_envfile()
{
docker run --env-file ./secret.txt gilleslamiral/imapsync imapsync \
--host1 test1.lamiral.info --user1 test1 \
--host2 test2.lamiral.info --user2 test2 --justlogin
}
echo docker_delete_all_images
docker_delete_all_images()
{
docker rm `docker ps -a -q`
docker rmi `docker images -q`
}
echo docker_system_prune
docker_system_prune()
{
docker system prune
}
echo docker_delete_dandling_images
docker_delete_dandling_images()
{
docker images
dandling_images=`docker images -f dangling=true -q`
exited_containers=`docker ps -a -f status=exited -q`
test -n "$exited_containers" && docker rm $exited_containers
test -n "$dandling_images" && docker rmi $dandling_images
docker images
}
echo imapsync_docker_rebuild_from_scratch
imapsync_docker_rebuild_from_scratch()
{
delete_all_images
docker images
imapsync_docker_build
}
echo is_release_number
is_release_number()
{
echo "$1" | perl -pe 'm/^\d\.\d+$/ || exit 1'
}
echo imapsync_docker_upload
imapsync_docker_upload()
{
version=`imapsync_docker_version`
version_local=`imapsync_local_docker_version`
# upload only after successful tests
set -x \
&& is_release_number "$version" \
&& is_release_number "$version_local" \
&& test "$version" = "$version_local" \
&& rcsdiff imapsync Dockerfile memo_docker \
&& imapsync_docker_testslive \
&& imapsync_docker_tests \
&& docker run gilleslamiral/imapsync true \
&& docker run gilleslamiral/imapsync diff /imapsync /usr/bin/imapsync \
&& docker login --username=gilleslamiral --password=`cat $HOME/var/pass/secret.docker` \
&& docker push gilleslamiral/imapsync:latest \
&& docker push gilleslamiral/imapsync:$version \
&& set +x && echo imapsync push SUCCEEDED || echo imapsync push FAILED
set +x
}

290
INSTALL.d/prerequisites_imapsync Executable file
View File

@ -0,0 +1,290 @@
#!/bin/sh
# $Id: prerequisites_imapsync,v 1.36 2022/05/17 13:46:16 gilles Exp gilles $
MODULES_MANDATORY='
App::cpanminus
Authen::NTLM
CGI
Compress::Zlib
Crypt::OpenSSL::RSA
Data::Dumper
Data::Uniqid
Digest::HMAC
Digest::HMAC_MD5
Digest::MD5
Dist::CheckConflicts
Encode
Encode::Byte
Encode::IMAPUTF7
File::Copy::Recursive
File::Tail
IO::Socket::INET
IO::Socket::INET6
IO::Socket::SSL
IO::Tee
JSON
JSON::WebToken
JSON::WebToken::Crypt::RSA
HTML::Entities
LWP::UserAgent
Mail::IMAPClient
MIME::Base64
Module::Implementation
Module::Runtime
Module::ScanDeps
Net::SSLeay
Package::Stash
Package::Stash::XS
PAR::Packer
Parse::RecDescent
Pod::Usage
Proc::ProcessTable
Readonly
Regexp::Common
Sys::MemInfo
Term::ReadKey
Test::Fatal
Test::Mock::Guard
Test::MockObject
Test::More
Test::Pod
Test::Requires
Test::Deep
Text::ParseWords
Try::Tiny
Unicode::String
URI::Escape
'
MODULES_FOR_CODERS='
Smart::Comments
'
test_perl() {
# First we need perl
if perl -v > /dev/null 2>&1 ; then
perl_version=`perl -e 'printf "%vd\n", $^V;'`
echo Ok: Found Perl $perl_version
return 0
else
echo 'Failure: Perl is not here. You have to install Perl first.'
return 1
fi
}
test_make() {
# Second we need make to build some Perl modules
if echo foo: | make -f - foo ; then
make_version=`make -v |head -1`
echo Ok: Found make $make_version
return 0
else
echo 'Failure: make is not here. You have to install the "make" command.'
return 1
fi
}
test_cpanm() {
# Second we need make to build some Perl modules
# redirect "cpanm ... < /dev/null" is there for macos buid via ssh
# no clue why it's necessary
if cpanm -h > /dev/null 2>&1 < /dev/null; then
cpanm_version=`cpanm --version < /dev/null | head -1`
echo Ok: Found cpanm $cpanm_version
return 0
else
echo 'Failure: cpanm is not here. You have to install the "cpanm" command.'
return 1
fi
}
test_module() {
test -n $1 || return
M_tested=$1
shift
if perl -m"$M_tested" -e '' >/dev/null 2>&1 ; then
echo "Ok: Found Perl module $M_tested"
else
echo "Failure: Not found Perl module $M_tested $@"
LIST_TO_INSTALL="$LIST_TO_INSTALL $M_tested"
fi
return
}
test_mandatory_modules() {
for M in $MODULES_MANDATORY
do
test_module $M
done
}
search_modules_any() {
test -n "$*" || {
echo "All needed modules are already installed"
return
}
# Debian, Ubuntu & Co
apt-cache -h > /dev/null 2>&1 && {
search_modules_apt "$@"
return
}
# Centos & Co
yum -h > /dev/null 2>&1 && {
search_modules_yum "$@"
return
}
# ArchLinux & Co
pacman -h > /dev/null 2>&1 && {
search_modules_pacman "$@"
return
}
# FreeBSD
pkg version > /dev/null 2>&1 && {
search_modules_freebsd "$@"
return
}
# no yum, no apt-get, no pacman
{
search_modules_cpanm "$@"
return
}
}
search_modules_cpanm() {
cat <<EOD
Here is a cpanm command to install missing Perl modules:
cpanm $@
EOD
}
search_modules_pacman() {
echo
echo Searching pacman packages names
#echo pacman -Fy
#pacman -Fy
echo pacman -S --noconfirm --needed pkgfile
pacman -S --noconfirm --needed pkgfile
echo pkgfile --update
pkgfile --update
for M in "$@" ; do
echo "==== Searching pacman package name for $M"
F=`echo $M|tr -s ":" "/"`.pm
# Not very good "pacman -Fs"
#echo pacman -Fs "$F"
#echo
#pacman -Fs "$F"
#echo
# Better! pkgfile --reg
echo "pkgfile --reg $F | grep perl-"
echo
pkgfile --reg "$F" | grep perl-
echo
done
}
search_modules_yum() {
echo
echo Searching rpm packages names
for M in "$@" ; do
echo "==== Searching rpm package name for $M"
F=`echo $M|tr -s ":" "/"`.pm
echo yum -q whatprovides "*/$F"
echo
yum -q whatprovides "*/$F"
echo
done
}
search_modules_apt() {
echo
echo Searching deb packages names
for M in "$@" ; do
F=`echo $M|tr -s ":" "/"`.pm
echo "==== Searching deb package name for $M with: apt-file search /$F"
#echo apt-file search /$F
echo
if apt-file -h > /dev/null 2>&1
then
apt-file search /$F
else
echo "apt-file is not installed. Suggestion: apt-get install apt-file; apt-file update"
fi
echo
echo "==== Searching deb package name for $M with: apt-cache search $M"
#apt-cache search "$M"
apt-cache search "$M"
echo
done
}
search_modules_freebsd()
{
echo
echo Searching pkg FreeBSD names
for M in "$@" ; do
F=`echo $M|tr -s ":" "-"`
echo "==== Searching deb package name for $M with: pkg search p5-$F"
echo
pkg search "p5-$F"
echo
done
}
list_to_install() {
test -n "$LIST_TO_INSTALL" || return 0
echo
echo 'What you have to do before using imapsync:'
for M in $LIST_TO_INSTALL ; do
echo "Install Perl module $M"
done
# return false/bad if some modules are missing.
return 1
}
test_unix() {
echo '$SHELL says ' $SHELL
echo '$0 gives ' $0
echo -n "ps -ef gives " ; ps -ef | grep $$ | grep -v grep| grep -v 'ps -ef'
sw_vers > /dev/null 2>&1 && sw_vers # Darwin
lsb_release -dirc > /dev/null 2>&1 && lsb_release -dirc # Linux
uname -a
}
test_unix
#exit
test_perl || exit 1
test_make || exit
test_mandatory_modules
list_to_install
EXIT=$?
# Help the user to install missing modules
search_modules_any $LIST_TO_INSTALL
if test "$1" = "MODULES_MANDATORY"
then
search_modules_any $MODULES_MANDATORY
elif test -n "$1"
then
search_modules_any "$@"
fi
test_cpanm
exit $EXIT

2
INSTALL.d/secret.txt Normal file
View File

@ -0,0 +1,2 @@
IMAPSYNC_PASSWORD1=secret1
IMAPSYNC_PASSWORD2=secret2

View File

@ -1,5 +1,5 @@
NO LIMIT PUBLIC LICENSE
Version 0, June 2012
Version 0/0, June 2012
Gilles LAMIRAL
La Billais
@ -10,6 +10,6 @@ France
Terms and conditions for copying, distribution, modification
or anything else.
0 No limit to do anything with this work and this license.
0 No limits to do anything with this work and this license.
1 GOTO 0

1159
Makefile

File diff suppressed because it is too large Load Diff

View File

@ -1,5 +1,5 @@
NO LIMIT PUBLIC LICENSE
Version 0, June 2012
Version 0/0, June 2012
Gilles LAMIRAL
La Billais
@ -10,6 +10,6 @@ France
Terms and conditions for copying, distribution, modification
or anything else.
0 No limit to do anything with this work and this license.
0 No limits to do anything with this work and this license.
1 GOTO 0

1216
README

File diff suppressed because it is too large Load Diff

1
README.txt Symbolic link
View File

@ -0,0 +1 @@
README

225
README_Windows.txt Normal file
View File

@ -0,0 +1,225 @@
# $Id: README_Windows.txt,v 1.27 2022/04/05 19:45:21 gilles Exp gilles $
#
# This is the README_Windows.txt file for imapsync
# imapsync: IMAP syncing and migration tool.
=====================
Imapsync on Windows
=====================
There are two ways to install and use imapsync on Windows systems: A) or B).
Standard users should take the A) way, the simplest way.
Developers, or power users that want to build their own imapsync.exe
or modify it, have to consider the B) way, the complex and powerful way.
---------------
A) Simplest way
---------------
A.0) Preamble for visual users looking for a visual tool.
First, the visual thing I am talking about here is not the online
visual interface I call /X that you might have seen and used at the
URL https://imapsync.lamiral.info/X/
This /X visual interface is not yet available on Windows as a service
that you can install and use in your system. I plan to make a visual
tool available on Windows but, for now, I encountered technical
issues.
So what visual thing am I talking about?
Imapsync itself is not a visual tool. The visual tool is Notepad or
any text editor. Many pure visual users have succeeded in using
imapsync to transfer their email accounts. As you can guess, only the
ones that tried have succeeded, so don't give up before trying at
least once. Another assumption is that visual users can read.
Let's go for some reading!
A.1) Get imapsync.
Get imapsync at https://imapsync.lamiral.info/
You'll then have access to a zip archive file named imapsync_2.200.zip
where 2.200 is the imapsync release number.
A.2) Extract the zip file in a folder where you will work with imapsync.
You can work on the Desktop since the zip file extraction creates
a unique folder named imapsync_2.200/
Do not unzip the archive in what is called a "system" directory since
you may encounter permission issues.
Two points to have in mind:
* You don't need to be an Administrator to unzip the zip archive.
* You don't need to be an Administrator to run imapsync.
In short, any user on your system can use imapsync.
A.3) Check the folder
In the folder extracted and called imapsync_2.200, you see 7 files
and 2 directories. Those files and directories may be presented in a
different order than the following, the order is not important
anyway. There are only two important files to get your mailbox
transfer job started in a few minutes, the first two files of the
following list:
* README_Windows.txt is the current file you are reading.
* imapsync_example.bat is a simple batch file example that you will copy and edit.
* README.txt is the imapsync general document.
* FAQ.d/* FAQs are a good read when something goes wrong.
* sync_loop_windows.bat is a batch file example for syncing many accounts.
* file.txt is an input file example for syncing many accounts.
* imapsync.exe is the imapsync 64bit binary. You don't have to run it directly.
* imapsync_32bit.exe is the imapsync 32bit binary. You don't have to run it directly.
* Cook/ is the directory to build imapsync.exe from its source,
for the B) way and expert users.
You can copy or rename the file imapsync_example.bat as you wish,
as long as its extension remains ".bat", for example, mysync.bat
On Windows systems, a file name ending with a .bat extension means
"I'm a batch script". A batch script is a file containing commands,
it's a program. Don't be afraid, a program can be very simple to
modify and I hope imapsync_example.bat is one of them.
The batch scripts have to stay in the same directory as imapsync.exe
because of the way they call imapsync.exe. They use the string
".\imapsync.exe", so let them be in the same directory.
You can change the path .\ to whatever you want if you understand
what you are doing (you have to use a pathname from the script point
of view).
For the rest of this documentation, I assume you copied
imapsync_example.bat to a file named mysync.bat
If you don't know how to copy and rename a file then use
imapsync_example.bat itself, it's ok. The original file is still in
the zip file in case you want to restart from scratch.
A.4) Edit the batch file
This section describes how to edit the file mysync.bat and change
it by replacing example values with your values.
To edit mysync.bat, you have to right-click on it and select "modify"
in the list presented in the small window menu.
Notepad or Notepadd++ are very good editor candidates to modify the
script. Notepad is already installed on any Windows system,
Notepadd++ is not usually installed but if you have it, then use it.
Office Word or any powerful text processor is NOT good for that job.
Text processors transform files in a special format that is wrong to
make them stay a good batch file, so don't use them!
The documents FAQ.txt and FAQ.d/* contain many tips. They describe
some special options that are sometimes needed by specific imap
software servers like Exchange, Office365, or Gmail.
Those documents are also available online at
https://imapsync.lamiral.info/FAQ.d/
You don't have to look into them unless you encounter problems.
A.5) Run the batch file
To run imapsync with your values just double-click on the batch file
mysync.bat
There is no need to have administrator privileges to run imapsync.
The run happens in a DOS window; usually, this window is black.
If imapsync.exe returns immediately with the ERRORLEVEL -1, it may be
because you have a Group Policy in place to prevent the execution of
programs in the %TEMP% directory. Temporarily remove this restriction
and imapsync will work as expected. Thanks to Walter H. for this
input!
Technically speaking, imapsync.exe is an embedded Perl script with
also the Perl interpreter and many Perl modules, all glued together
in an archive auto-extracted at run time. So it needs write-access to
the temporary directory. The temporary directory name depends on the
user but its value is in the variable %TEMP%. You can have the value
by running the command ECHO %TEMP% in a DOS window. You can have a
DOS window by launching the command cmd.exe
A.6) Look at the sync running.
You can abort the sync at any time with a quick double ctrl-c, hit
ctrl-c twice within one second.
A single ctrl-c will reconnect to both imap servers.
You can also simply abort the sync by closing the DOS window, using
the cross situated at the up-right corner.
What you see in this DOS terminal is also put in a logfile located
in the subdirectory LOG_imapsync/
A.7) Control what happened.
When the sync is finished you can find the whole log file of the
output in the folder named "LOG_imapsync/".
The logfile name is based on the launching date, hour, minute,
second, milliseconds, plus the user1 and user2 parameters.
For example, a file name can be
LOG_imapsync\2019_11_29_14_49_36_514_tata_titi.txt
There is one log file created for each run. The log file name is
printed both at the beginning and the end of the imapsync run.
IMPORTANT: When there is a problem, the problem is very often
described at the end of the log file. It means you don't have to read
all this bloody ununderstandable verbose logfile, just read the end
first.
A.8) Loop on A.4 through A.7
* A.4) edit the batch file
* A.5) run the batch file
* A.6) look at the run and the log file
* A.7) control what happened.
Loop on the process of editing, running and controlling imapsync
until you solve all issues and the sync is over.
A good sign that the sync went very well is when the nearly last lines are like:
" The sync looks good, all 123456 identified messages in host1 are on host2.
" There is no unidentified message
" Detected 0 errors
Congratulations!
------------
B) Hard way
------------
It is the hard way because it installs all software dependencies.
This is the way for modifying imapsync.exe if needed.
B.1) Install Perl if it isn't already installed.
Strawberry Perl is a very good candidate
http://strawberryperl.com/
I use 5.32.1.1 (released 2021-01-24) but previous and later releases
should work as well (Perl 5.18 to 5.30 do).
B.2) Go into the Cook/ directory
B.3) Double-click build_exe.bat
It should create a binary imapsync.exe in the current Cook/ directory.
B.4) Move imapsync.exe in the upper directory and follow instructions
from A.3) to A.8)

Some files were not shown because too many files have changed in this diff Show More