[Twisted-Python] recommended twisted database coding

[Jesus Cea]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Terry Jones wrote:
> Thanks for the info.  I think I've seen Durus before, but was not aware
> that you had implemented a BSD backend. Does that mean the qualification on
> the Durus home page "Durus is best suited to collections of less than a
> million instances with relatively stable state." does not apply here?

Right. My storage engine allows multigigabytes repositories, with a lot
of writes. Actually, I use it myself with a 933 gygabytes repository
(mailbox storage) and a couple of days ago a guy reported me his
profile: 16 terabytes storage managing medical scanning images.

> After looking around at some Pyhon persistence options, I decided to use
> ZODB. I'd be happy to hear your thoughts on Durus+BSD versus ZODB. I was
> disappointed to see that ZODB had dropped BSD as a backend as BSD is battle
> hardened and I wanted to use its secondary DBs.

ZODB is fine, but its storage engine is ugly. I miss also the BSD
backend. Without it, ZODB behaviour when you have a lot of writes is not
better than stock Durus. The main advantages of ZODB over Durus is the
distributed caching client (ZEO) and the multithreading support. I'm
working with the Durus guys to improve durus in those areas. Progress is
slow but working on it :-p

For my environment, Durus simplicity (you can keep the entiry durus code
in your head) and performance, plus my storage backend goodies, are my
favourite poisons :).

> BTW, I had no reply to
> email sent to greg at electricrain.com, the maintainer of the Python bsddb
> interface. I presume you are using that code?

Yes I use pybddb. In fact, next release (in python 2.5 beta 1) will
include some patches of mine :-). Greg is a bit slow answering email. I
suffer it myself from time to time :). The inminent Python 2.5 beta
could increase his workload, also, I guess. So, forgiveness...

> Anyway, I am using the BTrees
> provided by ZCatalog in Zope. Apart from things like robustness, speed, and
> number of objects I could store, I very much wanted to avoid having
> multiple persistence systems. Using ZODB makes this very easy. It sounds
> like you have that advantage too.

Durus has a BTree class also :). I also prefer to use a single
persistence system. I'm decommisioning SQL, for example, migrating all
my systems to Durus backends. My Durus preference over ZODB started
because I could understand Durus completely, and its "limitations" were
not important to me. Nowadays Durus was improved a lot and I still see
additional potential to grow.

BTW, I see very few people using ZODB outside ZOPE. Ah, my personal
webserver runs Zope :p

> How well supported is Durus? What about your interface?

My only complain about Durus support is that it is not a open source
proyect in the sense that you can't directly contribute code. This is an
issue I had talked to the coder guys several times in the past. I guess
that if durus could reach a critical mass, the issue would be solved by
itself. Meantime, the difficulty is convincing the coder guys that your
suggestions a) do not hurt their performance and b) are valuable enough
for their workload. Of course, their workload is not your workload. In
any case, I must say that they are smart and recognize a good idea when
they see it. The mailing list is quiet and supportive, also.

About my backend... currently I try to do a monthly release, although
current releases are already production ready. I'm very motivated and
Durus + my backend are top priorities now for me, so... I'm planning for
replication/redundancy also, if Durus guys implement some suggestions I
did recently :).

> Are you in Spain? We can discuss that offline I guess.

Yes. I live in Madrid.

We are getting a bit offtopic. If you use jabber/XMPP, you can contact
me online. See details in my signature.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBRI9CoJlgi5GaxT1NAQL/pwP+LQQnzJANoDaAjp2/7nAMl1CMcrCgBIrP
WUapc2h6W8SsO4XYRuWd8d/yZAvhG7ZnsGZOHYnn4qvQ9BPBwNDy1z2vej/ZdQpd
KbGOSwW2hNm8burdmHvbVvRAPuuZGS+0u0/YApOQ+XTsiMOxpPHeFdRPncw+J9Lg
uzcnUVXpZC4=
=nhHW
-----END PGP SIGNATURE-----