[Twisted-Python] yielding from within deferToThread
Naveen Michaud-Agrawal
naveen.michaudagrawal at gmail.com
Thu Dec 3 12:54:12 MST 2015
To answer my own question, it looks like i can just pass sqlalchemy's
ResultProxy into deferToThread to make a blocking call to fetch the next
set of results:
@defer.inlineCallbacks
def stream_results():
engine = sqlalchemy.create_engine(...)
query = "select * from table"
proxy = yield threads.deferToThread(engine.execute, query)
while True:
results = yield threads.deferToThread(proxy.fetchmany, 1000)
if not results:
break
print results
On Thu, Dec 3, 2015 at 2:41 PM, Naveen Michaud-Agrawal <
naveen.michaudagrawal at gmail.com> wrote:
> I'm trying to use sqlalchemy from a twisted application (by running all
> blocking queries using deferToThread). Is it possible to yield from within
> the function running in deferToThread? For example:
>
>
> def threadRunQuery(engine, query):
> conn = engine.connect()
> res = conn.execute(query)
>
> while True:
> results = res.fetchmany(1000)
> if not results:
> break
> yield results
>
>
> @defer.inlineCallbacks
> def stream_results():
> engine = sqlalchemy.create_engine(...)
> query = "select * from table"
> result_iter = yield threads.deferToThread(threadRunQuery, engine,
> query)
> for results in result_iter:
> print results
>
>
> It seems that the thread returns a generator, and so everything within
> threadRunQuery is actually running on the main reactor thread. Is there
> anyway to stream back results from a deferToThread?
>
> Regards
> Naveen Michaud-Agrawal
>
--
-----------------------------------
Naveen Michaud-Agrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20151203/c2e04a5e/attachment-0002.html>
More information about the Twisted-Python
mailing list